Software Engineer - Evaluation Engineering
23 days ago Cupertino, CA
About us
Working at Apple means doing more than you ever thought possible and having more impact than you ever imagined.
- Size: 10,000+ employees
- Industry: Technology: Manufacturing
Why you should apply for a job to Apple:
Ranked as one of the Best Technology Companies in 2020
3.7/5 in overall job satisfaction
3.1/5 in supportive management
66% say women are treated fairly and equally to men
65% would recommend this company to other women
66% say the CEO supports gender diversity
Ratings are based on anonymous reviews by Fairygodboss members. See all reviews ›
Join the team redefining what a deeply personal and integrated assistant can be.
As part of the Siri organization, you will help shape one of the world's most widely used AI assistants, powered by our next-generation of Apple Intelligence, with capabilities like personal context understanding and on-screen awareness, built with privacy from the ground up. Your work will have direct, meaningful impact for users across iOS, iPadOS, macOS, watchOS, and visionOS.
This is a rare opportunity to build at the intersection of cutting-edge AI and human-centered design, shipping technology that is centered around users and their needs.
Description
We are looking for a Software Engineer to help build the foundational on-device execution harness and runtime infrastructure for agentic evaluation. As Apple Intelligence evolves, validating agent performance requires reliable, deterministic execution of multi-step agent trajectories directly on heterogeneous user hardware. Our team focuses on building the core harnesses, environment control layers, and trace collection pipelines that enable downstream teams to evaluate agentic reasoning, tool use, on-screen awareness, and personal context actions across the entire Apple ecosystem.
In this role, you will design the low-level execution harnesses and orchestration tooling that drive dynamic agent simulations on target devices. You will solve complex systems engineering challenges around local device state management, process isolation, determinism, and high-fidelity trace generation across varying hardware constraints (iPhones, Macs, iPads, Apple Watch, and Apple Vision Pro).
","responsibilities":"Architect and build high-reliability execution harnesses to drive dynamic, multi-step agentic trajectories and tool interactions directly on target Apple hardware (iOS, macOS, watchOS, visionOS).
Implement lightweight, and event-capture layers to record high-fidelity execution traces (tool inputs/outputs, state mutations, and runtime events) for downstream scoring and analysis.
Build CLI tools, local automation scripts, and integration hooks that allow software and ML teams to run on-device agent harnesses seamlessly within local development environments and continuous testing pipelines.
Optimize harness performance and resource overhead, ensuring test execution remains stable and lightweight without interfering with on-device AI runtime behavior, thermal limits, or memory profiles.
Preferred Qualifications
Hands-on experience developing on Apple platforms (iOS, macOS, visionOS) or working directly with low-level OS runtimes and frameworks.
Experience designing or operating environment-staging systems, such as state reset mechanisms, device provisioning, or sandboxed test environments.
Experience building telemetry, trace-capture, or event-logging infrastructure for complex, multi-threaded, or asynchronous systems.
Familiarity with on-device AI/LLM runtime environments, function calling/tool execution mechanics, or agentic system architectures.
Minimum Qualifications
Strong programming proficiency in compiled languages (Swift, C++, or Objective-C).
Excellent communication skills and ability to collaborate across teams.
Scripting and automation skills in Python for tooling.
B.S. or M.S. in Computer Science, Machine Learning, or equivalent experience.
Solid computer science and systems fundamentals, including process management, and local runtimes.
Pay & Benefits
At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $150,400 and $225,300, and your base pay will depend on your skills, qualifications, experience, and location.
Apple employees also have the opportunity to become an Apple shareholder through participation in Apple's discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple's Employee Stock Purchase Plan. You'll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses - including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits
Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.
As part of the Siri organization, you will help shape one of the world's most widely used AI assistants, powered by our next-generation of Apple Intelligence, with capabilities like personal context understanding and on-screen awareness, built with privacy from the ground up. Your work will have direct, meaningful impact for users across iOS, iPadOS, macOS, watchOS, and visionOS.
This is a rare opportunity to build at the intersection of cutting-edge AI and human-centered design, shipping technology that is centered around users and their needs.
Description
We are looking for a Software Engineer to help build the foundational on-device execution harness and runtime infrastructure for agentic evaluation. As Apple Intelligence evolves, validating agent performance requires reliable, deterministic execution of multi-step agent trajectories directly on heterogeneous user hardware. Our team focuses on building the core harnesses, environment control layers, and trace collection pipelines that enable downstream teams to evaluate agentic reasoning, tool use, on-screen awareness, and personal context actions across the entire Apple ecosystem.
In this role, you will design the low-level execution harnesses and orchestration tooling that drive dynamic agent simulations on target devices. You will solve complex systems engineering challenges around local device state management, process isolation, determinism, and high-fidelity trace generation across varying hardware constraints (iPhones, Macs, iPads, Apple Watch, and Apple Vision Pro).
","responsibilities":"Architect and build high-reliability execution harnesses to drive dynamic, multi-step agentic trajectories and tool interactions directly on target Apple hardware (iOS, macOS, watchOS, visionOS).
Implement lightweight, and event-capture layers to record high-fidelity execution traces (tool inputs/outputs, state mutations, and runtime events) for downstream scoring and analysis.
Build CLI tools, local automation scripts, and integration hooks that allow software and ML teams to run on-device agent harnesses seamlessly within local development environments and continuous testing pipelines.
Optimize harness performance and resource overhead, ensuring test execution remains stable and lightweight without interfering with on-device AI runtime behavior, thermal limits, or memory profiles.
Preferred Qualifications
Hands-on experience developing on Apple platforms (iOS, macOS, visionOS) or working directly with low-level OS runtimes and frameworks.
Experience designing or operating environment-staging systems, such as state reset mechanisms, device provisioning, or sandboxed test environments.
Experience building telemetry, trace-capture, or event-logging infrastructure for complex, multi-threaded, or asynchronous systems.
Familiarity with on-device AI/LLM runtime environments, function calling/tool execution mechanics, or agentic system architectures.
Minimum Qualifications
Strong programming proficiency in compiled languages (Swift, C++, or Objective-C).
Excellent communication skills and ability to collaborate across teams.
Scripting and automation skills in Python for tooling.
B.S. or M.S. in Computer Science, Machine Learning, or equivalent experience.
Solid computer science and systems fundamentals, including process management, and local runtimes.
Pay & Benefits
At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $150,400 and $225,300, and your base pay will depend on your skills, qualifications, experience, and location.
Apple employees also have the opportunity to become an Apple shareholder through participation in Apple's discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple's Employee Stock Purchase Plan. You'll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses - including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits
Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.
Additional Information
- Salary: $150,400 – $225,300
- Posted: 23 days ago