SWE-Bench Task Auditor

AI Evaluation · Remote · posted 2mo ago
Pay$70–$90/hrAdvertised by the platform.
WhereRemote · Remote — United States, USA
Open toUnited States
Compared with median+43% vs the $56/hr median
This is a referral link. We may earn a fee if you join; it never changes what Mercor pays you.

About this role

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.

Basic Qualifications
• 3+ years professional software engineering
• Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
• Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
• Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)

Preferred Qualifications
• Familiarity with SWE-Bench (Verified) or similar repository benchmarks
• Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
• Prior code-review or task-grading experience

Details come from Mercor’s public posting and may change. Check the original before applying. Job ID e055977558.