The work
Assess software engineering code used to train and benchmark large language models. The work combines hands-on coding judgment with structured reviews of AI-generated solutions.
- Curate code examples and create clear solutions for training datasets.
- Correct and refine code in Python, C/C++, Rust, Go, Java, and JavaScript, including ReactJS.
- Review AI-generated code for correctness, efficiency, scalability, reliability, and performance.
- Build agents to verify systems-level and infrastructure code, identify recurring errors, and assess model capabilities across software engineering tasks.
- Design automated checks that validate solutions to software engineering tasks.
Skills and requirements
- Strong systems programming, infrastructure, or backend development expertise, especially in Python, C/C++, Rust, or Go.
- Experience building and deploying scalable, production-grade software.
- A strong understanding of software architecture, design, development, debugging, and code-quality review.
- Ability to write and explain clear, structured evaluation rationales.
Schedule and location
- Time commitment: 20+ hours per week.
- Location: Open to candidates based in the United States.
- Engagement: Flexible contractor work, initially for one month, with possible extensions based on performance and fit.