Team Management Coding Agent Benchmark Engineer

You will design and evaluate realistic software engineering tasks used to measure coding agents. The work uses production-like repositories, tests, configurations, documentation, runtime scenarios, evaluators, reference patches, and scoring rubrics.

The partner title identifies this role as Team Management, so review the duties and qualifications before applying. The work described here focuses on coding-agent benchmark design and evaluation.

• Build benchmark tasks for bug fixes, feature additions, CI repairs, configuration migrations, integration updates, and runtime-state fixes.
• Define utility and safety requirements, then create visible and hidden tests for coding-agent tasks.
• Create safe and intentionally unsafe reference solutions for evaluation scenarios.
• Assess shortcuts such as disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, or bypassing validation.
• Package prompts, metadata, runnable repositories, evaluators, reference patches, rubrics, and calibration notes.
• Analyze coding-agent rollouts for utility and safety violations and improve evaluator reliability and benchmark difficulty with engineering, quality assurance, and security stakeholders.

This is a part-time contractor role. The listing does not provide a pay rate.

• Time requirement: 20 or more hours per week.
• Location: Open worldwide.
• Language: English fluency.
• Experience level: Entry level.
• Experience: At least eight years of hands-on software engineering experience in product companies or technology startups.
• Education: Bachelor's or master's degree in computer science or a related technical field.
• Required programming: Expert Python skills, plus proficiency in Java, Go, or C++.
• Required background: Production software used by real customers, software architecture, debugging, performance optimization, deployment pipelines, monitoring, logging, and distributed tracing.
• Required knowledge: High availability, fault tolerance, scalability, disaster recovery, secure code review, vulnerability remediation, and complex production failure analysis.

Apply on OpenTrain with your resume, then complete the application on the hiring site.

AI training is the human work behind systems that learn from examples, including writing code tasks, testing model outputs, and rating whether responses meet requirements. People with software engineering experience help make these evaluations accurate, useful, and safe. OpenTrain helps people find AI training and data-labeling projects, build a profile, and apply in one place.