AI Engineer - Algorithm Evaluation & Agentic Systems
26 days ago Sunnyvale, CA
About us
Working at Apple means doing more than you ever thought possible and having more impact than you ever imagined.
- Size: 10,000+ employees
- Industry: Technology: Manufacturing
Why you should apply for a job to Apple:
Ranked as one of the Best Technology Companies in 2020
3.7/5 in overall job satisfaction
3.1/5 in supportive management
66% say women are treated fairly and equally to men
65% would recommend this company to other women
66% say the CEO supports gender diversity
Ratings are based on anonymous reviews by Fairygodboss members. See all reviews ›
How do we ensure Apple's next-generation AI products are robust, safe, and truly intelligent? Join the DAQ team to help answer that. We are seeking an AI Engineer specializing in algorithm evaluation and agentic systems design for advanced computer vision and video understanding algorithms.
What We Value
Production mindset: correctness, observability and maintainability
Ability to reason about system-level tradeoffs, not just model performance
Ability to balance experimentation speed with engineering rigor
Comfort working in ambiguous problem spaces and defining metrics from first principles
Clear communication of technical findings to both technical and non-technical audiences
Description
Within the DAQ team, our core mission is to evaluate and elevate advanced visual technologies. As a key member of this group, you will lead the benchmarking and integration of state-of-the-art models for image and video understanding. Rather than focusing on core model training, you will apply your deep CV and ML expertise to rigorously test models in applied settings, uncover edge-case failure modes, and architect advanced agentic systems. If you are passionate about AI safety, robust evaluation, and building autonomous multi-modal workflows that bridge experimentation with production, we'd love to hear from you.","responsibilities":"Algorithm Evaluation & Benchmarking: Design, build, and scale comprehensive evaluation pipelines. You will be responsible for both holistic end-to-end system evaluation and granular component-level testing to rigorously measure model capabilities on complex image and video understanding tasks.
Deep Failure Analysis: Leverage your CV and ML background to dive deep into model outputs, identifying root causes of visual hallucinations, temporal inconsistencies in video, and edge-case failures.
Agentic Architecture: Build, deploy, and evaluate agentic workflows that utilize these vision models to autonomously solve multi-step user problems (e.g., video summarization, visual search). You will heavily utilize component-level evaluation to isolate and triage exactly which parts of the agentic workflow (e.g., tool selection, memory retrieval, visual reasoning) are succeeding or failing.
Golden Data Curation: Lead the strategy for curating high-quality, schematized datasets and ground-truth benchmarks specifically tailored for evaluating multi-modal capabilities.
Cross-Functional Collaboration: Partner closely with the core model training teams. You will provide them with actionable, data-driven insights and metrics to guide the next iteration of model training and fine-tuning.
Preferred Qualifications
Demonstrated ability to lead technical evaluation strategies end-to-end, drive architectural decisions for testing infrastructure, and mentor engineers.
Strong foundation in statistics, including hypothesis testing, confidence intervals, and experimental design
Knowledge of reinforcement learning, planning, or decision-making systems
Experience evaluating multi-modal or multi-agent systems
Prior work on AI reliability, safety, or benchmarking
Minimum Qualifications
MS and a minimum of 3 years relevant industry experience
3+ years of applied experience in Machine Learning, Computer Vision, or AI System Evaluation
Solid ML Foundation: Deep understanding of core Machine Learning principles, including probability, statistics, data distributions, and model bias/variance. You can apply statistical rigor to ensure evaluation metrics are meaningful and reliable.
Computer Vision Expertise: Deep theoretical and practical understanding of Computer Vision (CV) and Vision-Language Models (VLMs). You must understand how Vision Transformers (ViTs), spatial-temporal modeling, and image/video processing work under the hood to effectively evaluate them.
Advanced Evaluation Skills: Proven track record of defining robust metrics/KPIs and designing rigorous evaluation frameworks for generative AI or foundation models. Deep experience with custom benchmark creation, automated regression testing, LLM/VLM-as-a-judge methodologies, and human-in-the-loop evaluation.
Agentic Systems: Experience building and evaluating LLM/VLM-powered agents, including tool use, multi-step reasoning, planning, and memory management workflows.
Failure Analysis: Strong intuition for probing ML models to discover edge cases, hallucinations, and performance bottlenecks in constrained environments. Be able to translate findings into actionable improvement recommendations.
Engineering Excellence: Strong proficiency in Python and experience with deep learning frameworks (PyTorch) for running inference, extracting embeddings, and building scalable evaluation pipelines.
Pay & Benefits
At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $150,400 and $277,600, and your base pay will depend on your skills, qualifications, experience, and location.
Apple employees also have the opportunity to become an Apple shareholder through participation in Apple's discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple's Employee Stock Purchase Plan. You'll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses - including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits
Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.
What We Value
Production mindset: correctness, observability and maintainability
Ability to reason about system-level tradeoffs, not just model performance
Ability to balance experimentation speed with engineering rigor
Comfort working in ambiguous problem spaces and defining metrics from first principles
Clear communication of technical findings to both technical and non-technical audiences
Description
Within the DAQ team, our core mission is to evaluate and elevate advanced visual technologies. As a key member of this group, you will lead the benchmarking and integration of state-of-the-art models for image and video understanding. Rather than focusing on core model training, you will apply your deep CV and ML expertise to rigorously test models in applied settings, uncover edge-case failure modes, and architect advanced agentic systems. If you are passionate about AI safety, robust evaluation, and building autonomous multi-modal workflows that bridge experimentation with production, we'd love to hear from you.","responsibilities":"Algorithm Evaluation & Benchmarking: Design, build, and scale comprehensive evaluation pipelines. You will be responsible for both holistic end-to-end system evaluation and granular component-level testing to rigorously measure model capabilities on complex image and video understanding tasks.
Deep Failure Analysis: Leverage your CV and ML background to dive deep into model outputs, identifying root causes of visual hallucinations, temporal inconsistencies in video, and edge-case failures.
Agentic Architecture: Build, deploy, and evaluate agentic workflows that utilize these vision models to autonomously solve multi-step user problems (e.g., video summarization, visual search). You will heavily utilize component-level evaluation to isolate and triage exactly which parts of the agentic workflow (e.g., tool selection, memory retrieval, visual reasoning) are succeeding or failing.
Golden Data Curation: Lead the strategy for curating high-quality, schematized datasets and ground-truth benchmarks specifically tailored for evaluating multi-modal capabilities.
Cross-Functional Collaboration: Partner closely with the core model training teams. You will provide them with actionable, data-driven insights and metrics to guide the next iteration of model training and fine-tuning.
Preferred Qualifications
Demonstrated ability to lead technical evaluation strategies end-to-end, drive architectural decisions for testing infrastructure, and mentor engineers.
Strong foundation in statistics, including hypothesis testing, confidence intervals, and experimental design
Knowledge of reinforcement learning, planning, or decision-making systems
Experience evaluating multi-modal or multi-agent systems
Prior work on AI reliability, safety, or benchmarking
Minimum Qualifications
MS and a minimum of 3 years relevant industry experience
3+ years of applied experience in Machine Learning, Computer Vision, or AI System Evaluation
Solid ML Foundation: Deep understanding of core Machine Learning principles, including probability, statistics, data distributions, and model bias/variance. You can apply statistical rigor to ensure evaluation metrics are meaningful and reliable.
Computer Vision Expertise: Deep theoretical and practical understanding of Computer Vision (CV) and Vision-Language Models (VLMs). You must understand how Vision Transformers (ViTs), spatial-temporal modeling, and image/video processing work under the hood to effectively evaluate them.
Advanced Evaluation Skills: Proven track record of defining robust metrics/KPIs and designing rigorous evaluation frameworks for generative AI or foundation models. Deep experience with custom benchmark creation, automated regression testing, LLM/VLM-as-a-judge methodologies, and human-in-the-loop evaluation.
Agentic Systems: Experience building and evaluating LLM/VLM-powered agents, including tool use, multi-step reasoning, planning, and memory management workflows.
Failure Analysis: Strong intuition for probing ML models to discover edge cases, hallucinations, and performance bottlenecks in constrained environments. Be able to translate findings into actionable improvement recommendations.
Engineering Excellence: Strong proficiency in Python and experience with deep learning frameworks (PyTorch) for running inference, extracting embeddings, and building scalable evaluation pipelines.
Pay & Benefits
At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $150,400 and $277,600, and your base pay will depend on your skills, qualifications, experience, and location.
Apple employees also have the opportunity to become an Apple shareholder through participation in Apple's discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple's Employee Stock Purchase Plan. You'll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses - including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits
Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.
Additional Information
- Salary: $150,400 – $277,600
- Posted: 26 days ago