Intelligence Analyst AI Evaluation Specialist

You will help evaluate artificial intelligence systems through structured intelligence scenarios and expert feedback. The work focuses on how well AI handles sources, reasoning, uncertainty, limitations, and practical usefulness.

You will review analytic cases, test cases, and AI-generated responses. You will also help identify situations where a model refuses legitimate analytic exploration without good reason.

• Map analytic cases to defined taxonomies across several intelligence evaluation areas.
• Write complex test cases that examine source use, analytic rigor, uncertainty, and practical usefulness.
• Assess and grade AI-generated responses against clear analytic criteria.
• Document over-refusal, edge cases, analytic disagreements, and meaningful model limitations.
• Help refine source-evaluation procedures and confidence-level assignments.
• Share expert insights that support consistent analytic standards and improve AI evaluation benchmarks.

This is remote, part-time contractor work. The role is open to candidates in the countries listed below and requires professional fluency in English.

• Pay: $30 to $80 per hour.
• Work type: Part-time contractor engagement.
• Location: Remote, open to candidates in the United Arab Emirates, Argentina, Austria, Bangladesh, Belgium, Bahrain, Brazil, Canada, Switzerland, Chile, Colombia, Germany, Denmark, Algeria, Egypt, Spain, France, the United Kingdom, Ghana, Greece, Indonesia, India, Italy, Jordan, Japan, Kenya, Kyrgyzstan, Kuwait, Kazakhstan, Lebanon, Sri Lanka, Morocco, Mexico, Malaysia, Nigeria, the Netherlands, Oman, Peru, the Philippines, Pakistan, Palestine, Portugal, Qatar, Saudi Arabia, Singapore, Thailand, Tunisia, Taiwan, the United States, Uzbekistan, Vietnam, and South Africa.
• Language: Professional fluency in English.
• Experience: Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline.
• Knowledge: Strong source evaluation, confidence-level assessment, established analytic standards, structured analytic techniques, and intelligence tradecraft.
• Communication: Clear written and verbal communication, including the ability to explain nuanced judgments.
• Leadership: Experience teaching, mentoring, or leading analytic reviews and fostering analytic rigor.
• Judgment: Ability to recognize and explain model limitations and over-refusal scenarios.
• AI background: Previous AI evaluation or model assessment experience is helpful but not required.

Apply on OpenTrain with your resume, then complete the application on the hiring site.

AI training is the human work behind systems that learn from examples, ratings, and expert reviews. People with strong subject-matter judgment help assess model behavior and provide the standards needed to make AI responses more useful and reliable.