AI Agent Evaluation Analyst

Location: Remote

About Mindrift

At Mindrift, innovation meets opportunity. Powered by Toloka, the Mindrift platform connects domain experts with cutting-edge AI projects from leading tech clients. Our mission is to unlock the potential of Generative AI by harnessing real-world expertise from diverse professionals around the globe.

Who We’re Looking For

Are you a curious, intellectually proactive thinker who loves playing devil’s advocate? Do you enjoy tackling complexity and ambiguity? This flexible, remote role is perfect if you want to contribute part-time to advanced AI projects while fitting the work around your schedule.

Ideal candidates include:

• Analysts, researchers, consultants, or students with strong critical thinking skills.
• Individuals comfortable with remote, asynchronous work environments.
• Those who thrive on problem-solving, logical analysis, and system thinking.

No coding background is required just curiosity, intellectual rigor, and the ability to evaluate complex systems.

About the Role

As an AI Agent Evaluation Analyst, you’ll be working on a project focused on validating and improving the logic, policies, and evaluation frameworks for autonomous AI agents. Your key responsibilities will include:

• Reviewing evaluation tasks and scenarios for completeness, logic, and realism.
• Spotting inconsistencies, missing assumptions, or unclear decisions.
• Defining clear expected behaviors (“gold standards”) for AI agents.
• Annotating cause-effect relationships and reasoning paths.
• Thinking holistically about complex systems to ensure robust AI testing.
• Collaborating with QA teams, writers, and developers to improve coverage and edge case handling.

About the Role

As an AI Agent Evaluation Analyst, you’ll be working on a project focused on validating and improving the logic, policies, and evaluation frameworks for autonomous AI agents. Your key responsibilities will include:

• Reviewing evaluation tasks and scenarios for completeness, logic, and realism.
• Spotting inconsistencies, missing assumptions, or unclear decisions.
• Defining clear expected behaviors (“gold standards”) for AI agents.
• Annotating cause-effect relationships and reasoning paths.
• Thinking holistically about complex systems to ensure robust AI testing.
• Collaborating with QA teams, writers, and developers to improve coverage and edge case handling.

What You’ll Need to Succeed

• Analytical Excellence: Ability to reason about complex systems, scenarios, and their logical implications.
• Attention to Detail: Skilled at identifying contradictions, ambiguities, and vague requirements.
• Familiarity with Structured Data: Comfortable reading JSON or YAML formats (writing not required).
• Holistic Assessment: Ability to evaluate scenarios for missing elements or potential failures.
• Communication Skills: Clear and concise writing in English to document findings effectively.

Preferred Experience

• Background in consulting, academia, competitive problem-solving (e.g., Olympiads), or research.
• Exposure to AI concepts such as LLMs, prompt engineering, or AI-generated content.
• Familiarity with QA methodologies, test-case design, and edge case evaluation.
• Understanding of evaluation metrics like precision and coverage in AI testing.
• Experience with logic puzzles, policy evaluation, or structured scenario design is a plus

Education & Experience

• Education: Bachelor’s degree or currently enrolled in a relevant field such as Computer Science, Data Science, Mathematics, Logic, Cognitive Science, or related disciplines. Advanced degrees or certifications are a plus but not mandatory.
• Experience: 1+ years in analytical roles, research, quality assurance, consulting, or academic problem-solving environments. Experience working remotely or in flexible project-based roles is highly valued

Why Join Mindrift?

• Competitive pay up to $55/hour, depending on your expertise.
• Fully remote, flexible schedule work when it suits you best.
• Gain hands-on experience with advanced AI systems and evaluation frameworks.
• Enhance your professional portfolio with cutting-edge AI projects.
• Influence how future AI models understand, reason, and communicate.
• Join a forward-thinking company dedicated to ethical AI development and innova

How to Apply

If you’re interested in this position, please register on our portal and submit your application through the link below:

👉 Register & Apply at TeezJobs.com