285 jobs for "Lead Platform Automation Engineer"

Senior Software Development Engineer in Test - LLM Evaluation & Automation, T3E

At Apple - Cupertino

Posted on Oct 16

View job Apply on company site

Apple Services Engineering (ASE) Compute - Software Engineering Manager

At Apple - Cupertino

Posted on Oct 16

View job Apply on company site

Platform & Data Engineer

At Apple - Cupertino

Posted on Oct 16

View job Apply on company site

Manager, Product Management- Self-Service User Experience

At Capital One - McLean · New York

Posted on Jul 25

View job Apply on company site

Director, Product Software Architecture and Eng

At Equinix - Remote / Flexible

Posted on Oct 8

View job Apply on company site

AIOps Engineer SME

At Leidos - Fort Belvoir

Posted on Jul 29

View job Apply on company site

Manager, Product Management - Card Authorizations Platform

At Capital One - McLean · Richmond

Posted on Jul 19

View job Apply on company site

Lead Workday Engineer - Integrations and Extend

At Mastercard - Harrison

Posted on Oct 6

View job Apply on company site

Principal Data Engineer - Data Quality

At Mastercard - Pune

Posted on Oct 6

View job Apply on company site

AIOps Engineer SME

At Leidos - Albuquerque

Posted on Jul 29

View job Apply on company site

Director, Product Software Architecture and Eng

At Equinix - Remote / Flexible

Posted on Oct 8

View job Apply on company site

Director, Product Software Architecture and Eng

At Equinix - Remote / Flexible

Posted on Oct 8

View job Apply on company site

Lead Workday Engineer - Integrations and Extend

At Mastercard - Richmond

Posted on Oct 6

View job Apply on company site

Senior OT Integration Engineer

At CRH - Atlanta

Posted on Sep 23

View job Apply on company site

Vice President, Infrastructure Engineering

At Delta Dental - Rancho Cordova

Posted on Oct 6

View job Apply on company site

Digital Solutions Integration Engineer III

At Mitsubishi Electric Power Products - Remote / Flexible

Posted on Oct 7

View job Apply on company site

Privileged Access Management Lead

At Mizuho Americas - MetroPark

Posted on Aug 22

View job Apply on company site

Data Engineering Lead - Senior Vice President

At iCapital - New York

Posted on Sep 15

View job Apply on company site

Sr. Business Systems Analyst - Digital Platforms

At Esri - Redlands

Posted on Aug 26

View job Apply on company site

Solutions Architect

At Appian - Mexico City

Posted on Oct 9

View job Apply on company site

Apple

Senior Software Development Engineer in Test - LLM Evaluation & Automation, T3E

Cupertino, CA$175,000 – $308,500Posted 24 days ago

Apply on company site View full job

Why you should apply for a job to Apple:

Ranked as one of the Best Technology Companies in 2020
3.7/5 in overall job satisfaction
3.1/5 in supportive management
66% say women are treated fairly and equally to men
65% would recommend this company to other women
66% say the CEO supports gender diversity

Ratings are based on anonymous reviews by Fairygodboss members. See all reviews ›

The Apple Intelligence Platform Experience Validation team builds the tooling and automation that keeps Apple Intelligence features high-quality before they ship. We are looking for a Senior SDET to lead the design and implementation of automated model evaluation: standing up LLM-as-a-judge in existing and new pipelines, and building the infrastructure that catches model regressions before they reach human evaluation or the live on population.

This is a hands-on, senior individual-contributor role. You will own eval automation as a discipline across the team, partnering with modeling, framework, and infrastructure teams to make model quality a first-class, continuously measured signal.

Description

You will build and maintain model level, component or end-to-end evaluation coverage for the generative features our team validates. Your job is to leverage LLM judge scoring output quality in automation, ensuring reliable, repeatable eval jobs that run that produce actionable signal.

The kinds of problems you will work on include:

* Image / visual generation: validating model output and its associated classification metadata, and detecting quality or behavior regressions across model updates.

* Natural-language generation: evaluating whether generated artifacts and responses match user intent, moving at-desk LLM judges into a scalable and repeatable automation environment.

* Correctness beyond string matching: replacing exact-match checks for open-ended or factual responses with an LLM-as-judge stage integrated into the pipeline.

* Generated insights and summaries: assessing whether model-generated content is sensible and good enough to surface to users.

You will decide when a component-level check (an API or CLI that exercises the model against its framework) is sufficient and when a full end-to-end user flow is required, and you will build the tooling for both.","responsibilities":"Design, build, and maintain LLM-as-a-judge evaluation harnesses and integrate them into existing and new CI/automation pipelines.

Author and curate eval sets and rubrics; partner with modeling teams whose own eval sets can run hundreds of examples judged by a separate model.

Run eval jobs at scale, triage results, and distinguish real model regressions from rubric problems or infrastructure noise so the signal stays actionable.

Plan and build test coverage and tooling from low level component tests through to end-to-end tests that exercise models and frameworks powering Generative AI features

Package eval tooling for reuse - reusable libraries and jobs that other engineers on the team and partner teams can adopt.

Define quality gates and reporting so model regressions are caught and communicated before human eval or population rollout.

Collaborate with data scientists and modeling engineers on approaches to spot regressions across large output sets or between model updates.

Preferred Qualifications

Strong practical knowledge of Python, including data-pipeline fluency (JSON/YAML, REST APIs).

Hands-on experience with LLM-as-a-judge evaluation and rubric design, or a strong demonstrated ability to ramp into it quickly.

Strong software engineering fundamentals - able to define atomic, composable components and build maintainable pipelines and tooling, not just scripts.

Strong debugging and triage skills; able to separate genuine regressions from infrastructure or rubric noise.

Strong knowledge of the software development lifecycle, testing methodologies, and QA processes.

Excellent written and verbal communication; able to document clearly and describe quality signal to modeling and leadership audiences.

Ability to lead work across varying priorities and partner multi-functionally with modeling, framework, and infrastructure teams.

Experience building on-device tooling and device/model eval infrastructure.

Experience integrating with CI/CD and job orchestration systems, and comfort deploying tooling as reusable libraries.

Familiarity with generative model behavior - image generation, NLP, or LLM output evaluation.

Experience curating and reasoning about large datasets; comfort manually inspecting data (Jupyter or similar) to build intuition and drive next steps.

Awareness of dataset bias and fairness considerations in evaluation.

Experience with database/query tooling (e.g., SQL) and dashboards/visualization for reporting quality trends.

Experience with Xcode is a bonus.

Minimum Qualifications

BS in Computer Science, Mathematics, or a related field (or equivalent practical experience)

Three years of relevant industry experience in test automation, software development, or related areas.

Pay & Benefits

At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $175,000 and $308,500, and your base pay will depend on your skills, qualifications, experience, and location.

Apple employees also have the opportunity to become an Apple shareholder through participation in Apple's discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple's Employee Stock Purchase Plan. You'll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses - including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits

Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

About Apple

Working at Apple means doing more than you ever thought possible and having more impact than you ever imagined.

Size: 10,000+ employees
Industry: Technology: Manufacturing

View company profile