Job Title: Machine Learning Engineer – LLM Evaluations (Evals)
Location: New York, NY (On-site – 5 days/week)
Compensation: $220,000 – $300,000/year + Competitive Equity
Employment Type: Full-Time
Work Authorization: U.S. Citizens or Green Card holders only (visa sponsorship is not available)
About the Opportunity
We are partnering with a well-funded, research-driven AI startup in New York City building the memory and identity layer for autonomous agents. We are looking for a Machine Learning Engineer (Evals) to own our end-to-end evaluation infrastructure from the ground up.
In a complex multi-agent environment with black-box behavior, standard benchmarks aren't enough. You will define what "better" looks like for dynamic entity representations that evolve over time—without relying on pre-existing ground truth—and build the scalable machinery to measure, test, and improve them.
If you thrive on designing evaluation frameworks from scratch, digging into execution traces to uncover root causes, and shipping fixes directly, this role is built for you.
Key Responsibilities
Design Novel Evaluations: Define scoring metrics and establish what "better" means for evolving entity representations, continuously updating frameworks as models and methods advance.
Build End-to-End Pipelines & Harnesses: Architect the data ingestion, labeling, versioning, rerun, and LLM-as-a-judge infrastructure that enables rapid weekly experimentation.
Analyze Traces & Drive Improvements: Deep-dive into model traces and evaluation outputs to uncover subtle failure modes—not just surface metrics—and propose concrete fixes to improve representation fidelity.
Scale Simulation Agents: Build large-scale simulation environments to model and evaluate thousands of entities simultaneously, directly accelerating team iteration speed.
Full Lifecycle Ownership: Own the entire feedback loop—from formulating the research question and building the harness to running experiments, writing up findings, and shipping improvements.
Required Qualifications
3+ years of hands-on experience designing, building, and running end-to-end evaluation systems for Large Language Models (LLMs).
Proven track record of creating novel evaluations from scratch (beyond running off-the-shelf benchmarks).
Production-grade Python engineering skills (automated testing, clean architecture, maintainable code, regular release cadence).
Strong proficiency with PyTorch, Hugging Face, and Weights & Biases (W&B).
Demonstrated research background, evidenced by first/co-authored publications at top-tier venues (NeurIPS,ICML, ICLR, or equivalent) OR significant, self-published open-source research work.
Ability to work on-site 5 days a week in New York, NY (or willingness to relocate).
Legal authorization to work in the United States without current or future visa sponsorship.
Preferred Qualifications
Prior work in AI memory, identity modeling, personalization, agent evaluation, or agentic optimization.
Experience with data pipeline and orchestration tooling such as Hydra, Kafka, or SQL.
Bachelor’s, Master’s, or PhD in Computer Science, Mathematics, Statistics, or a related quantitative field.
Pay: $200,000.00 - $300,000.00 per year
Work Location: In person
Machine Learning Engineer – LLM Evaluations (Evals)
Location: New York, NY (On-site – 5 days/week)
Compensation: $220,000 – $300,000/year + Competitive Equity
Employment Type: Full-Time
Work Authorization: U.S. Citizens or Green Card holders only (visa sponsorship is not available)
About the Opportunity
We are partnering with a well-funded, research-driven AI startup in New York City building the memory and identity layer for autonomous agents. We are looking for a Machine Learning Engineer (Evals) to own our end-to-end evaluation infrastructure from the ground up.
In a complex multi-agent environment with black-box behavior, standard benchmarks aren't enough. You will define what "better" looks like for dynamic entity representations that evolve over time—without relying on pre-existing ground truth—and build the scalable machinery to measure, test, and improve them.
If you thrive on designing evaluation frameworks from scratch, digging into execution traces to uncover root causes, and shipping fixes directly, this role is built for you.
Key Responsibilities
Design Novel Evaluations: Define scoring metrics and establish what "better" means for evolving entity representations, continuously updating frameworks as models and methods advance.
Build End-to-End Pipelines & Harnesses: Architect the data ingestion, labeling, versioning, rerun, and LLM-as-a-judge infrastructure that enables rapid weekly experimentation.
Analyze Traces & Drive Improvements: Deep-dive into model traces and evaluation outputs to uncover subtle failure modes—not just surface metrics—and propose concrete fixes to improve representation fidelity.
Scale Simulation Agents: Build large-scale simulation environments to model and evaluate thousands of entities simultaneously, directly accelerating team iteration speed.
Full Lifecycle Ownership: Own the entire feedback loop—from formulating the research question and building the harness to running experiments, writing up findings, and shipping improvements.
Required Qualifications
3+ years of hands-on experience designing, building, and running end-to-end evaluation systems for Large Language Models (LLMs).
Proven track record of creating novel evaluations from scratch (beyond running off-the-shelf benchmarks).
Production-grade Python engineering skills (automated testing, clean architecture, maintainable code, regular release cadence).
Strong proficiency with PyTorch, Hugging Face, and Weights & Biases (W&B).
Demonstrated research background, evidenced by first/co-authored publications at top-tier venues (NeurIPS,ICML, ICLR, or equivalent) OR significant, self-published open-source research work.
Ability to work on-site 5 days a week in New York, NY (or willingness to relocate).
Legal authorization to work in the United States without current or future visa sponsorship.
Preferred Qualifications
Prior work in AI memory, identity modeling, personalization, agent evaluation, or agentic optimization.
Experience with data pipeline and orchestration tooling such as Hydra, Kafka, or SQL.
Bachelor’s, Master’s, or PhD in Computer Science, Mathematics, Statistics, or a related quantitative field.
Pay: $200,000.00 - $300,000.00 per year
Work Location: In person