About this role
About Turing Turing is one of the world’s fastest-growing AI companies accelerating the advancement and deployment of powerful AI systems. You’ll be at the forefront of evaluating how LLMs interact with real code, influencing the future of AI-assisted software development. This is a unique opportunity to blend practical software engineering with AI research.
About the Role
We are seeking experienced Software Engineers to support a frontier AI training and evaluation initiative focused on improving the coding capabilities of AI agents.
In this role, you will review and compare AI-generated coding solutions across real-world software engineering tasks. You will analyze complete agent execution workflows, evaluate code correctness, efficiency, safety, and overall quality, and provide structured ratings and evidence-based feedback to help improve AI-assisted software development.
What you'll do
- Review and compare three AI coding agents working on the same real-world developer request.
- Analyze each agent's complete execution, including shell commands, files modified, code changes, test results, and final responses.
- Evaluate each agent's performance across 10 quality dimensions, including instruction following, code correctness, efficiency, safety, honesty, and communication.
- Assign an overall quality score from 1 to 5 for each agent and rank all three solutions based on their performance.
- Assess code functionality, implementation quality, debugging approaches, testing practices, and adherence to software engineering best practices.
- Identify coding errors, incomplete implementations, inefficient approaches, security concerns, and incorrect assumptions.
- Write concise, structured, and evidence-based rationales explaining evaluation scores and comparative rankings.