The work
As a Python Domain Expert, you will design and build realistic benchmark tasks for coding agents. You will work with production-like repositories, tests, configurations, documentation, and runtime scenarios to evaluate whether an agent completes engineering work safely and follows developer intent.
The work includes reviewing agent attempts for unsafe shortcuts, packaging runnable evaluations, and improving benchmark reliability with engineering, quality assurance, and security stakeholders. This role is listed as a Python Domain Expert position, so review the duties and qualifications before applying.
- Create tasks for bug fixes, feature additions, CI repairs, configuration migrations, integration updates, and runtime-state fixes.
- Define utility and safety requirements, including what an agent must do and which unsafe shortcuts it must avoid.
- Build visible and hidden tests, safe and unsafe reference solutions, scoring rubrics, calibration notes, and runnable evaluators.
- Review agent rollouts for disabled tests, weakened assertions, protected-data deletion, secret leaks, broad permissions, and bypassed validation.
- Analyze vulnerabilities and remediation involving injection, authorization, memory safety, race conditions, deserialization, secrets management, and input validation.
What it pays and takes
Pay details are not provided. This is a fully remote, part-time contractor assignment lasting approximately 4 to 8 weeks.
- Time: At least 20 hours per week, including at least 4 hours per day.
- Availability: At least 4 hours of overlap with Pacific Time.
- Location: Remote and open worldwide.
- Language: Fluent in English.
- Experience: At least 8 years of hands-on software engineering experience in leading product companies or technology startups.
- Education: A bachelor's or master's degree in computer science or a related technical field.
- Technical skills: Strong Python expertise and proficiency in Java, Go, or C++.
- Production background: Experience building and operating software used by real customers at scale, including deployment pipelines, monitoring, logging, observability, distributed tracing, high availability, fault tolerance, scalability, and disaster recovery.
- Helpful background: Experience creating coding-agent benchmarks, evaluating advanced coding-agent rollouts, or designing safety and alignment tests.
How it works
Apply on OpenTrain with your resume, then complete the application on the hiring site.
About AI training work
AI training is the human work behind systems that learn from examples, including code tasks and evaluations of model responses. Experienced engineers help test whether AI systems solve problems correctly, follow requirements, and avoid unsafe behavior.