211 jobs for "Machine Learning Evaluation Engineer"

Product Manager, AI/ML - Apple Ads, Marketplace

At Apple - Cupertino

Posted on Oct 14

View job Apply on company site

ML Data Operations Engineer

At Apple - Cupertino

Posted on Oct 10

View job Apply on company site

Senior Search Engineer - Services Special Projects

At Apple - Cupertino

Posted on Oct 10

View job Apply on company site

Senior Manager, Data Science - Model Risk

At Capital One - Richmond · New York

Posted on Nov 1

View job Apply on company site

Sr. Manager, AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation)

At Capital One - New York · San Francisco

Posted on Oct 10

View job Apply on company site

AIML - Machine Learning Research Lead, RL Agents, MLR

At Apple - Cupertino

Posted on Oct 9

View job Apply on company site

Applied AI Scientist - Multimodal Intelligence

At Apple - Seattle

Posted on Oct 17

View job Apply on company site

Data Scientist - Strategic Data Solutions

At Apple - Austin

Posted on Oct 17

View job Apply on company site

Applied AI Scientist - Multimodal Intelligence

At Apple - Seattle

Posted on Oct 16

View job Apply on company site

ML Data Operations Engineer

At Apple - Cupertino

Posted on Oct 10

View job Apply on company site

ML Data Operations Engineer

At Apple - Cupertino

Posted on Oct 9

View job Apply on company site

Apple

Product Manager, AI/ML - Apple Ads, Marketplace

Cupertino, CA$150,400 – $277,600Posted 24 days ago

Apply on company site View full job

Why you should apply for a job to Apple:

Ranked as one of the Best Technology Companies in 2020
3.7/5 in overall job satisfaction
3.1/5 in supportive management
66% say women are treated fairly and equally to men
65% would recommend this company to other women
66% say the CEO supports gender diversity

Ratings are based on anonymous reviews by Fairygodboss members. See all reviews ›

At Apple, we work every day to create products that enrich people's lives. The App Store and Apple Maps are trusted destinations for millions of users to discover apps, places, products, and services. Our advertising platform connects users with high-utility advertiser offerings while maintaining Apple's uncompromising commitment to user privacy and trust.

The Apple Ads Marketplace team is seeking deeply technical Product Managers to help build the next generation of AI/ML systems that power ad discovery, relevance, quality, and safety across Apple's ecosystem. We are hiring across multiple areas, with current opportunities focused on Ad Matching & Retrieval and Ad Relevance, Quality & Safety.

In these roles, you will define product strategy and roadmaps for sophisticated machine learning systems operating at massive scale. Depending on your background and area of expertise, you may focus on matching user intent to relevant advertiser offerings through modern search and retrieval systems, or on evaluating and improving ad relevance, quality, and safety through advanced AI/ML and LLM-based systems.

You will partner closely with ML research, engineering, data science, and cross-functional teams to translate advances in AI/ML into products that deliver value for users and advertisers while upholding Apple's commitment to privacy and user experience.

Description

As a Product Manager on Apple Ads Marketplace, you will work at the intersection of product, machine learning, and large-scale marketplace systems.

You will develop a deep understanding of user and advertiser needs, identify opportunities within complex ML-driven systems, and translate technical capabilities into clear product strategies, requirements, metrics, and execution plans. You will work closely with engineering and applied research teams throughout the ML lifecycle: from data and model development through evaluation, experimentation, inference, and production deployment.

Our current opportunities span two closely related areas:

- Ad Matching & Retrieval focuses on understanding user intent and retrieving high-utility advertiser offerings across the App Store, Apple Maps, and emerging search and conversational surfaces. Areas of work include semantic and lexical matching, query understanding, candidate generation, embeddings and vector retrieval, keyword generation, auto-targeting, and low-latency retrieval systems.

- Ad Relevance, Quality & Safety focuses on evaluating and improving the relevance, utility, and safety of ads delivered across Apple's ecosystem. Areas of work include AI/ML evaluation, LLM-powered raters, relevance and quality scoring, model distillation, classification, human and synthetic feedback systems, and real-time quality and safety guardrails.

You do not need to bring expertise across both areas. We are interested in technically strong product leaders whose experience aligns deeply with one or more of these problem spaces.","responsibilities":"Define product vision, strategy, roadmaps, and success metrics for large-scale AI/ML systems within Apple Ads Marketplace.

Partner deeply with ML research, engineering, and data science teams to shape model and system requirements and translate technical capabilities into impactful product experiences.

Drive products through the full lifecycle, from problem definition and technical exploration through development, experimentation, launch, measurement, and iteration.

Use data and experimentation to identify marketplace opportunities, diagnose gaps, evaluate model and product performance, and prioritize future investments.

Define metrics that connect ML system performance with user and advertiser outcomes, including relevance, retrieval quality, coverage, user satisfaction, engagement, and advertiser value.

Partner across Apple to responsibly incorporate privacy-preserving signals, platform knowledge, and shared technologies into Marketplace systems.

Communicate complex technical concepts, tradeoffs, and product strategies clearly to engineering, research, business, and executive stakeholders.

Depending on your area of expertise, drive products across areas such as:

* Matching & Retrieval: semantic and lexical matching, query understanding and rewriting, candidate generation, embeddings and vector search, keyword generation, auto-targeting, retrieval quality, and low-latency inference.

* Relevance, Quality & Safety: relevance and quality evaluation, LLM raters, model distillation, classification and safety systems, human-in-the-loop and synthetic feedback, model calibration, and real-time quality guardrails.

Preferred Qualifications

Experience in one or more relevant AI/ML domains, including:

Search, information retrieval, matching, recommendations/ranking, query understanding, semantic search, embeddings, vector retrieval, or related systems; or

AI/ML evaluation, model quality, relevance, LLM evaluation/raters, classification, model distillation, human-in-the-loop systems, trust & safety, or related systems.

Experience building or product-managing large-scale advertising, marketplace, search, recommendation, or content systems.

Experience with modern deep learning or LLM systems, including model training, fine-tuning, evaluation, inference, or production deployment.

Understanding of high-throughput, low-latency ML systems and the tradeoffs involved in deploying models in production environments.

Strong analytical skills, including experience with SQL and large-scale data exploration.

Experience designing and analyzing online experiments and using quantitative results to guide product decisions.

Experience taking technically complex or ambiguous products from 0→1.

Software engineering, data science, ML engineering, research, or other hands-on technical experience is a plus.

Minimum Qualifications

3+ years of technical product management or equivalent experience owning and delivering technically complex products or systems, including machine learning, AI, search, advertising, or related technologies.

Hands-on experience with AI/ML systems, with an emphasis on training, fine-tuning, evaluating, and inferencing large-scale deep learning models and LLMs.

Strong domain knowledge in one or more relevant AI/ML areas, such as search and information retrieval, matching and recommendations, AI/ML evaluation, relevance and model quality, LLM systems, classification, or related technologies.

Experience with high-throughput, low-latency online inference architectures across client and cloud server environments.

Strong technical and analytical foundation, including deep proficiency with SQL and data exploration in large-scale data warehouses.

Outstanding written and verbal communication skills, with proven ability to translate complex AI/ML architectures into crisp PRDs, system diagrams, and executive strategy.

Demonstrated leadership and cross-functional influence, adept at aligning engineering, applied research, business, and design stakeholders without formal authority.

Bachelor's or Master's degree in Computer Science, Electrical Engineering, Machine Learning, Data Science, or equivalent practical experience.

Pay & Benefits

At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $150,400 and $277,600, and your base pay will depend on your skills, qualifications, experience, and location.

Apple employees also have the opportunity to become an Apple shareholder through participation in Apple's discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple's Employee Stock Purchase Plan. You'll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses - including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits

Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

About Apple

Working at Apple means doing more than you ever thought possible and having more impact than you ever imagined.

Size: 10,000+ employees
Industry: Technology: Manufacturing

View company profile