Software Engineers: Create and Validate AI Coding Tasks

Incentives for this project are determined by geography. The following incentive per task applies. Rates are in USD:

• United States: $175
• India: $82
• Canada: $128
• South America: $93
• South Korea: $105
• Philippines: $82
• *Each task requires 4 to 5 hours to complete*

We're hiring software engineers for an ongoing remote project focused on building robust evaluations for AI coding agents. Your expertise ensures these models can handle realistic, complex problems in a terminal environment. This work directly feeds into improving the accuracy and reliability of automated engineering tools.

You will design realistic, self-contained software-engineering problems from scratch. The work involves implementing features, debugging software, configuring Docker environments, and writing automated tests. You will also create reference solutions and verify that each task can be graded reliably by our systems. All work happens remotely via your own machine using standard command-line tools and GitHub.

This opportunity is best suited for experienced software engineers who are comfortable navigating unfamiliar codebases. We welcome backend developers, full-stack engineers, DevOps specialists, and systems engineers. Candidates must be highly proficient with command-line tools and writing clean, deterministic tests.

• Design self-contained software-engineering problems for AI agents to solve.
• Build and configure development environments using Docker.
• Write automated, deterministic tests to reliably grade AI performance.
• Create accurate reference solutions for complex coding challenges.

• Professional experience working as a software engineer.
• Comfortable navigating unfamiliar codebases independently.
• Strong proficiency with GitHub and command-line tools.
• Willingness to complete a one-time skills assessment before starting.

• You will be engaged as an independent contractor.
• This is a fully remote opportunity that can be completed on your own schedule.
• Opportunities can be extended, shortened, or concluded early depending on needs and performance.
• Your participation will not involve access to confidential or proprietary information from any employer, client, or institution.
• Payments are processed weekly based on services rendered.
• We are unable to support H1-B or STEM OPT candidates at this time.