The work
Create and assess software examples used to train and benchmark large language models. You will apply software engineering judgment to improve AI-generated code and evaluate how well solutions work.
- Curate code examples and write precise solutions for training and benchmarking datasets.
- Correct and refine code in Python, C/C++, Rust, Go, Java, and JavaScript, including ReactJS.
- Evaluate systems, infrastructure, and backend code for correctness, performance, scalability, and reliability.
- Assess model abilities across prototyping, architecture and API design, production implementation, monitoring, and maintenance.
- Build agents that check code quality, find error patterns, and test solutions to software engineering tasks.
- Work with technical teams to improve AI coding solutions against industry performance benchmarks.
Hours, location, and requirements
- Hours: 20+ per week, with flexible participation.
- Location: You must be based in the United States.
- Engagement: Part-time contractor role, starting with a one-month term. Extensions may be possible based on performance and fit.
- Fluent in English.
- Strong skills in systems programming, infrastructure, or backend development, with familiarity in Python, C/C++, Rust, or Go.
- Experience building and deploying scalable, production-ready software, plus a strong understanding of software architecture, design, development, debugging, and code review.
- Ability to write clear, structured explanations for technical evaluations.
- Experience shipping products in fast-moving environments may be helpful. Computer science education is welcome, but demonstrated skills and experience take priority.