Professionals and Domain Experts: Paid Trial on AI Task Design

We are running a paid study to build a bench of evaluators who excel at designing hard tasks that expose where AI models fall short. This work helps us understand the boundaries of current AI capabilities by stressing them with genuine, complex workflows. Successful participants will help shape how we measure and improve model reasoning on difficult problems.

You will select a complex workflow from your professional or personal experience and turn it into a demanding request for an AI model. Using a screen-recorded session, you will run this prompt in ChatGPT to identify where the model breaks down or makes mistakes. After refining the prompt to ensure it requires real reasoning and produces a concrete file, you will write a detailed grading rubric. The entire process takes about an hour and requires a desktop or laptop computer.

We are looking for professionals and domain experts who possess deep knowledge in a specific area and can easily spot flawed AI outputs. You need to be comfortable using ChatGPT and willing to share your screen and thought process while you work. We welcome specialists from any industry, provided you can design a task that requires real-world lookup, calculation, or advanced reasoning.

• Pick a real-world workflow you know well to test an AI model
• Write a demanding prompt that requires reasoning and generates a finished file
• Test your prompt in ChatGPT to find its breaking points and iterate if it succeeds too easily
• Create a clear grading rubric that a stranger could use to evaluate the AI's attempt
• Record your screen, camera, and microphone while completing the exercise

• Deep knowledge in a specific professional or complex personal domain
• Ability to quickly evaluate the accuracy of complex work in your field
• Access to a desktop or laptop computer
• An active ChatGPT account (the free tier is acceptable)
• Comfortable with screen and audio recording during the task

• You will be engaged as an independent contractor.
• This is a fully remote opportunity that can be completed on your own schedule.
• Opportunities can be extended, shortened, or concluded early depending on needs and performance.
• Your participation will not involve access to confidential or proprietary information from any employer, client, or institution.
• Payment is issued as soon as your work is approved.
• We are unable to support H1-B or STEM OPT candidates at this time.