19 jobs for "Evaluation Science Lead"

Applied Researcher 5 (AI Foundations: Evaluation and Agentic Workflows)

At Capital One - McLean · Cambridge

Posted on Nov 8

View job Apply on company site

Manager, Data Scientist - Business Cards & Payments

At Capital One - New York · McLean

Posted on Nov 7

View job Apply on company site

Applied Researcher 5

At Capital One - San Jose · San Francisco

Posted on Nov 6

View job Apply on company site

Applied Researcher 4

At Capital One - San Jose · San Francisco

Posted on Nov 6

View job Apply on company site

Applied Researcher 4 (AI Safety & Multimodal)

At Capital One - New York · McLean

Posted on Nov 6

View job Apply on company site

Sr. Manager, AI Engineer (Gen AI Platform Services: Agentic AI, Guardrails, Evaluation)

At Capital One - New York · San Francisco

Posted on Nov 5

View job Apply on company site

Applied Researcher 5 (AI Foundations)

At Capital One - New York · McLean

Posted on Nov 5

View job Apply on company site

Data Scientist - Retail Banking (ML & Predictive Modeling)

At PNC Financial Services Group - Pittsburgh

Posted on Nov 8

View job Apply on company site

Associate, AI Data Scientist- Private Markets

At BlackRock - Bengaluru

Posted on Nov 4

View job Apply on company site

Senior/Principal Systems Engineer - Technical Security Systems, Onsite

At Sandia National Laboratories

Posted on Nov 3

View job Apply on company site

C++/Python Software Engineer

At Cadence Design Systems - Shanghai

Posted on Nov 2

View job Apply on company site

Data Scientist - Retail Banking (ML & Predictive Modeling)

At PNC Financial Services Group - Tysons Corner

Posted on Nov 8

View job Apply on company site

Data Scientist - Retail Banking (ML & Predictive Modeling)

At PNC Financial Services Group - Cleveland

Posted on Nov 8

View job Apply on company site

Senior Data Scientist

At Aristocrat

Posted on Nov 6

View job Apply on company site

Principal Associate, Data Scientist - SBB Marketing & Deposit Strategy

At Capital One - McLean

Posted on Nov 7

View job Apply on company site

Manager, Data Scientist - SBB Marketing & Deposit Strategy

At Capital One - McLean

Posted on Nov 6

View job Apply on company site

Senior Manager, Data Science - Model Risk

At Capital One - McLean · Richmond

Posted on Nov 5

View job Apply on company site

Senior Manager, Data Science - Model Risk

At Capital One - Richmond · McLean

Posted on Nov 5

View job Apply on company site

Manager, Data Science - Emerging ML

At Capital One - McLean · San Jose

Posted on Nov 5

View job Apply on company site

BlackRock

Associate, AI Data Scientist- Private Markets

Bengaluru, IndiaData SciencePosted 5 days ago

Apply on company site View full job

Why you should apply for a job to BlackRock:

3.4/5 in overall job satisfaction
2.9/5 in supportive management
33% say women are treated fairly and equally to men
44% would recommend this company to other women
67% say the CEO supports gender diversity

Ratings are based on anonymous reviews by Fairygodboss members. See all reviews ›

About this role

Build efficient, trusted agentic AI for private-markets intelligence

Research, build and scale advanced agentic systems and fit-for-purpose language models that turn complex information into dependable data, research and decision intelligence.

Your team

You will join a multidisciplinary AI and data science team focused on applying advanced AI to high-value challenges in private markets. The team brings together data science, engineering, product and domain expertise, and works closely with users to develop practical capabilities that improve how information is acquired, researched and managed.

You will work in an environment where applied research is expected to become dependable production capability. We value intellectual curiosity, rigorous experimentation, strong engineering and responsible AI, with quality, security and appropriate human oversight designed in from the start.
Your role & impact

As an Associate, AI Data Scientist, you will be a hands-on technical contributor responsible for researching, building and scaling enterprise-grade agentic AI capabilities. Working directly with users and senior technical leaders, you will translate complex private-markets workflows into well-evaluated systems that move from experimentation to governed production and sustained adoption.

Your core mandate is to advance multi-agent LLM systems through effective agent design patterns, rigorous evaluation and disciplined model engineering. You will develop workflows that plan, retrieve, reason, use tools, verify and escalate appropriately, while selecting the simplest architecture and model that can meet the required quality, latency, cost and control objectives.

You will also help establish the lifecycle for open-source small language models and LLMs, from model discovery and benchmarking through training, adaptation, fine-tuning, deployment, monitoring and retirement. Your impact will be visible in stronger data quality, faster research, scalable automation and reusable AI capabilities that help BlackRock Preqin remain at the forefront of private-markets data and technology.

Your responsibilities

• Work forward-deployed with researchers, data specialists and product teams to understand workflows, frame ambiguous problems, define measurable outcomes and iterate rapidly from evidence to production.

• Research, design and deliver agentic workflows spanning planning, retrieval, tool use, structured extraction, reasoning, verification, exception handling and human escalation, using single-agent or multi-agent patterns where each is demonstrably appropriate.

• Engineer advanced agent patterns, including bounded iterative loops, reflection and critique, planner-executor and evaluator-optimizer patterns, with explicit stop conditions, budgets and safeguards that prevent unproductive or uncontrolled execution.

• Build agent and model harnesses that standardise tool interfaces, context assembly, state and memory handling, policy enforcement, tracing, replay and failure recovery.

• Determine when an agentic architecture, retrieval workflow, specialised model or deterministic service is the best fit, and validate the decision against representative tasks and simpler baselines.

• Design safe caching and reuse across retrieval, prompts, model responses, tool results and intermediate agent state, with clear cache keys, invalidation, freshness, access-control and provenance requirements.

• Build systematic evaluation harnesses using representative datasets, explicit quality dimensions, adversarial and edge-case testing, regression suites, human feedback and monitored production outcomes.

• Research and benchmark open-source SLMs and LLMs for task fit, considering quality, inference performance, cost, licensing, security, deployment constraints and operational supportability; apply fit-for-purpose routing and batching where they improve efficiency.

• Develop model-adaptation strategies using prompt and context optimisation, supervised fine-tuning, parameter-efficient fine-tuning or other suitable techniques, supported by governed dataset creation, experiment tracking, reproducibility and controlled promotion criteria.

• Operate the lifecycle for trained and fine-tuned open-source models, including data and model versioning, lineage, experiment and artifact management, evaluation gates, registry and release controls, deployment, performance and drift monitoring, rollback, retraining triggers and retirement.

• Build production-quality Python components that integrate models with enterprise data, retrieval services, APIs, workflow orchestration and human-review experiences.

• Improve grounding and trust through attribution, provenance, uncertainty handling, validation checks, access controls and clear separation of source evidence from model-generated conclusions.

• Own delivered reliability across testing, observability, latency, cost and maintainability, and diagnose model, data, prompt and orchestration failures in production.

• Document findings, reusable patterns and trade-offs; contribute to design reviews; and communicate results clearly to research, product, engineering, risk and senior stakeholders.

• Partner with engineering, product, risk, privacy, information security, legal and compliance colleagues to support responsible, secure and accountable use of AI.

Required experience

We are looking for a hands-on applied AI practitioner who combines strong research and modelling skills with production judgment and user-focused problem solving. You should bring:

• Strong experience building and deploying AI or machine-learning systems in an enterprise, SaaS, data-product or similarly complex environment, with evidence of taking work beyond prototypes.

• Practical depth in LLM applications and agentic systems, including retrieval, tool use, planning, orchestration, memory or state management, structured outputs, verification and evaluation of non-deterministic behaviour.

• Experience implementing and comparing bounded loops, reflection or critique, planner-executor and evaluator-optimizer patterns against simpler baselines, with measurable treatment of quality, reliability, latency, token usage, cost and failure modes.

• Experience building agent or model harnesses across tool interfaces, context and state management, orchestration, policy enforcement, testing, tracing, replay, deployment and observability.

• Hands-on experience evaluating, selecting, training or adapting open-source SLMs or LLMs, including benchmarking, governed dataset preparation, supervised or parameter-efficient fine-tuning and inference optimisation.

• Practical ModelOps experience for trained or fine-tuned models, including dataset and model lineage, experiment and artifact tracking, registries, evaluation gates, controlled release, performance and drift monitoring, rollback, retraining and retirement.

• Experience designing model routing, batching and safe caching with appropriate keying, invalidation, freshness, privacy, access-control and provenance safeguards.

• Strong grounding in modern machine learning, experimentation and statistical evaluation, with disciplined handling of train-test separation, leakage, uncertainty, reproducibility and model comparison.

• Ability to write production-quality Python and work effectively across data preparation, model development, APIs, orchestration, testing, deployment and observability.

• Experience with unstructured or multimodal information, semantic retrieval, provenance and data-quality engineering in workflows where domain experts provide ground truth and structured feedback.

• Experience working directly with users or domain specialists to turn ambiguous needs into usable capabilities and iterate based on observed workflow outcomes.

• Sound judgment on model and architecture trade-offs across efficacy, efficiency, security, licensing, maintainability and operational risk.

• Clear communication and collaborative delivery skills, with the ability to explain experiments, limitations and production trade-offs to technical and non-technical audiences.

• Understanding of private markets, alternative investments, financial data or adjacent institutional-investment workflows is desirable; curiosity and the ability to build domain depth quickly are essential.

• Commitment to responsible AI, data provenance, privacy, security and meaningful human control in high-trust enterprise workflows.

Why this role

This is an opportunity to work on consequential AI problems, real user workflows and high standards for trust. You will have meaningful hands-on ownership, direct access to users and the opportunity to turn emerging agent and model techniques into durable capabilities with enterprise reach.

Our benefits

To help you stay energized, engaged and inspired, we offer a wide range of benefits including a strong retirement plan, tuition reimbursement, comprehensive healthcare, support for working parents and Flexible Time Off (FTO) so you can relax, recharge and be there for the people you care about.

Our hybrid work model

BlackRock's hybrid work model is designed to enable a culture of collaboration and apprenticeship that enriches the experience of our employees, while supporting flexibility for all. Employees are currently required to work at least 4 days in the office per week, with the flexibility to work from home 1 day a week. Some business groups may require more time in the office due to their roles and responsibilities. We remain focused on increasing the impactful moments that arise when we work together in person - aligned with our commitment to performance and innovation. As a new joiner, you can count on this hybrid model to accelerate your learning and onboarding experience here at BlackRock.

Guidance on AI use for candidates

At BlackRock, AI has long been part of how we work - enhancing decision-making, improving operations, and helping us deliver better outcomes for clients. We encourage candidates to use AI thoughtfully to learn, prepare, and work more effectively; but during our interview process, we want to focus on getting to know you through your own experiences, thinking, and judgment. To support you, we've provided guidance on when and how to use AI during our hiring process so you can approach each step with confidence and showcase your best self.

About BlackRock

At BlackRock, we are all connected by one mission: to help more and more people experience financial well-being. Our clients, and the people they serve, are saving for retirement, paying for their children's educations, buying homes and starting businesses. Their investments also help to strengthen the global economy: support businesses small and large; finance infrastructure projects that connect and power cities; and facilitate innovations that drive progress.

This mission would not be possible without our smartest investment - the one we make in our employees. It's why we're dedicated to creating an environment where our colleagues feel welcomed, valued and supported with networks, benefits and development opportunities to help them thrive.

To learn more about BlackRock, please visit Careers.BlackRock.com. We also encourage you to get to know us on LinkedIn, Instagram, YouTube, X, and TikTok.

BlackRock is proud to be an Equal Opportunity Employer. We evaluate qualified applicants without regard to age, disability, family status, gender identity, race, religion, sex, sexual orientation and other protected attributes at law.

About BlackRock

BlackRock is a global investment firm, trusted to manage more assets than any other.

Size: 10,000+ employees
Industry: Finance: Asset Management

View company profile