Machine Learning Engineer – LLM Evaluations Job in New York, NY | Yulys
×

Job Title: Machine Learning Engineer – LLM Evaluations

Company Name: Headhuntermorales
Salary: USD 200,000.00
-
USD 300,000.00 Yearly
Posted at: Recently posted
Job Industry: Artificial Intelligence
Job Type: Full time
WorkPlace Type: remote
Location: New York, NY, United States
Required Candidates: 1 Candidates
Skills:
Robotics
Speech Recognition
AI Automation
Job Description:

Job Title: Machine Learning Engineer – LLM Evaluations (Evals)

Location: New York, NY (On-site – 5 days/week)

Compensation: $220,000 – $300,000/year + Competitive Equity

Employment Type: Full-Time

Work Authorization: U.S. Citizens or Green Card holders only (visa sponsorship is not available)

About the Opportunity

We are partnering with a well-funded, research-driven AI startup in New York City building the memory and identity layer for autonomous agents. We are looking for a Machine Learning Engineer (Evals) to own our end-to-end evaluation infrastructure from the ground up.

In a complex multi-agent environment with black-box behavior, standard benchmarks aren't enough. You will define what "better" looks like for dynamic entity representations that evolve over time—without relying on pre-existing ground truth—and build the scalable machinery to measure, test, and improve them.

If you thrive on designing evaluation frameworks from scratch, digging into execution traces to uncover root causes, and shipping fixes directly, this role is built for you.

Key Responsibilities

Design Novel Evaluations: Define scoring metrics and establish what "better" means for evolving entity representations, continuously updating frameworks as models and methods advance.

Build End-to-End Pipelines & Harnesses: Architect the data ingestion, labeling, versioning, rerun, and LLM-as-a-judge infrastructure that enables rapid weekly experimentation.

Analyze Traces & Drive Improvements: Deep-dive into model traces and evaluation outputs to uncover subtle failure modes—not just surface metrics—and propose concrete fixes to improve representation fidelity.

Scale Simulation Agents: Build large-scale simulation environments to model and evaluate thousands of entities simultaneously, directly accelerating team iteration speed.

Full Lifecycle Ownership: Own the entire feedback loop—from formulating the research question and building the harness to running experiments, writing up findings, and shipping improvements.

Required Qualifications

3+ years of hands-on experience designing, building, and running end-to-end evaluation systems for Large Language Models (LLMs).

Proven track record of creating novel evaluations from scratch (beyond running off-the-shelf benchmarks).

Production-grade Python engineering skills (automated testing, clean architecture, maintainable code, regular release cadence).

Strong proficiency with PyTorch, Hugging Face, and Weights & Biases (W&B).

Demonstrated research background, evidenced by first/co-authored publications at top-tier venues (NeurIPS,ICML, ICLR, or equivalent) OR significant, self-published open-source research work.

Ability to work on-site 5 days a week in New York, NY (or willingness to relocate).

Legal authorization to work in the United States without current or future visa sponsorship.

Preferred Qualifications

Prior work in AI memory, identity modeling, personalization, agent evaluation, or agentic optimization.

Experience with data pipeline and orchestration tooling such as Hydra, Kafka, or SQL.

Bachelor’s, Master’s, or PhD in Computer Science, Mathematics, Statistics, or a related quantitative field.

Pay: $200,000.00 - $300,000.00 per year

Work Location: In person

Machine Learning Engineer – LLM Evaluations (Evals)

Location: New York, NY (On-site – 5 days/week)

Compensation: $220,000 – $300,000/year + Competitive Equity

Employment Type: Full-Time

Work Authorization: U.S. Citizens or Green Card holders only (visa sponsorship is not available)

About the Opportunity

We are partnering with a well-funded, research-driven AI startup in New York City building the memory and identity layer for autonomous agents. We are looking for a Machine Learning Engineer (Evals) to own our end-to-end evaluation infrastructure from the ground up.

In a complex multi-agent environment with black-box behavior, standard benchmarks aren't enough. You will define what "better" looks like for dynamic entity representations that evolve over time—without relying on pre-existing ground truth—and build the scalable machinery to measure, test, and improve them.

If you thrive on designing evaluation frameworks from scratch, digging into execution traces to uncover root causes, and shipping fixes directly, this role is built for you.

Key Responsibilities

Design Novel Evaluations: Define scoring metrics and establish what "better" means for evolving entity representations, continuously updating frameworks as models and methods advance.

Build End-to-End Pipelines & Harnesses: Architect the data ingestion, labeling, versioning, rerun, and LLM-as-a-judge infrastructure that enables rapid weekly experimentation.

Analyze Traces & Drive Improvements: Deep-dive into model traces and evaluation outputs to uncover subtle failure modes—not just surface metrics—and propose concrete fixes to improve representation fidelity.

Scale Simulation Agents: Build large-scale simulation environments to model and evaluate thousands of entities simultaneously, directly accelerating team iteration speed.

Full Lifecycle Ownership: Own the entire feedback loop—from formulating the research question and building the harness to running experiments, writing up findings, and shipping improvements.

Required Qualifications

3+ years of hands-on experience designing, building, and running end-to-end evaluation systems for Large Language Models (LLMs).

Proven track record of creating novel evaluations from scratch (beyond running off-the-shelf benchmarks).

Production-grade Python engineering skills (automated testing, clean architecture, maintainable code, regular release cadence).

Strong proficiency with PyTorch, Hugging Face, and Weights & Biases (W&B).

Demonstrated research background, evidenced by first/co-authored publications at top-tier venues (NeurIPS,ICML, ICLR, or equivalent) OR significant, self-published open-source research work.

Ability to work on-site 5 days a week in New York, NY (or willingness to relocate).

Legal authorization to work in the United States without current or future visa sponsorship.

Preferred Qualifications

Prior work in AI memory, identity modeling, personalization, agent evaluation, or agentic optimization.

Experience with data pipeline and orchestration tooling such as Hydra, Kafka, or SQL.

Bachelor’s, Master’s, or PhD in Computer Science, Mathematics, Statistics, or a related quantitative field.

Pay: $200,000.00 - $300,000.00 per year

Work Location: In person


Are you looking for more jobs nearby? Find your favorite jobs now by visiting our online jobs page.

Become a part of our growth newsletter