The work
You will review Python code from real open-source projects and help assess how well AI models handle software bugs. The work includes setting up repositories, running code locally, checking tests, and identifying difficult issues for LLM evaluation.
- Analyze and triage GitHub issues across open-source Python libraries.
- Set up and configure repositories, including Docker environments and basic development pipelines.
- Review unit test coverage and assess test quality.
- Modify and run codebases locally to evaluate LLM performance on bug-fixing tasks.
- Work with researchers to identify repositories and issues that are challenging for LLMs.
What it pays and takes
This is a remote contract project lasting 2 to 3 weeks, with an expected start next week.
- Pay: $20 per approved task.
- Time: 20+ hours per week.
- Location: Open to candidates in India, Pakistan, Nigeria, Egypt, Ghana, Bangladesh, Türkiye, and Mexico.
- Schedule: Include at least 4 hours of overlap with Pacific Time.
- Language: Fluent English.
- Experience: At least 2 years of overall experience.
- Skills: Strong Python experience, plus Git, Docker, and basic software pipeline setup.
- You must be able to understand complex codebases and run, modify, and test projects locally.
- Helpful background includes open-source contributions or evaluation, LLM research or evaluation projects, and developer tools or automation agents.
How it works
Apply on OpenTrain with your resume, then complete the application on the hiring site.