- Pay
- $100–$120 / hr
- Open to
- Worldwide
About this role
Mercor is seeking software engineers to build and refine evaluations. You’ll turn completed pull requests into engineering tasks and use our in-house evaluation framework to multiple frontier coding agents, including Claude Code, Codex and more.
We work alongside an in-house research team and have produced industry leading benchmarks to compare the performance of frontier large language models.
In this role, you will:
- Identify repositories with enough substantive work to support challenging evaluations, and select suitable completed PRs.
- Investigate each problem, its reference solution, and the repository’s architecture, tests, and conventions.
- Write task prompts and build evaluations using automated tests, shell commands, and LLM grading prompts.
- Validate evaluations against reference solutions and deliberately flawed implementations. Find missing checks, incorrect grades, and criteria that unnecessarily constrain how a problem can be solved.