AI Evaluation Specialists: Building Rubrics for Generated Artifacts

We're running a paid study on the best methods for evaluating complex generated outputs. Establishing clear evaluation rubrics is essential for improving the accuracy and reliability of these artifacts. Your expertise will directly inform how we structure our quality assurance frameworks moving forward.

During this remote session, you will review several examples of generated artifacts and analyze their components. We will ask you to identify key quality dimensions and draft specific grading criteria for each category. You will share your screen to walk us through your thought process as you build the evaluation rubric. Finally, you will test your newly created rubric against a fresh set of artifacts to validate its effectiveness.

We are looking for quality assurance professionals and AI evaluation specialists with experience designing grading frameworks. You should be highly analytical and comfortable breaking down complex outputs into measurable criteria. We welcome QA leads, data annotation specialists, technical editors, and senior evaluators.

• Review examples of generated artifacts to identify common patterns and potential flaws
• Draft measurable grading criteria for evaluating output quality
• Walk us through your rubric design process during a remote session
• Test your proposed evaluation framework against new examples to ensure consistency

• Professional experience in quality assurance, data annotation, or AI evaluation
• Proven ability to design structured grading frameworks or rubrics
• Comfortable analyzing complex technical or creative outputs
• Strong verbal communication skills to articulate your evaluation strategy

• You will be engaged as an independent contractor.
• This is a fully remote opportunity that can be completed on your own schedule.
• Opportunities can be extended, shortened, or concluded early depending on needs and performance.
• Your participation will not involve access to confidential or proprietary information from any employer, client, or institution.
• Payments are processed weekly based on services rendered.
• We are unable to support H1-B or STEM OPT candidates at this time.