Role overview

Cohere is an enterprise AI company building foundation models and products for real-world business applications. As a Data Annotation Specialist, you will play a direct role in improving large language models by providing high-quality feedback, evaluation, and corrections on model outputs across multiple data formats and languages.

Key responsibilities

Evaluate and rank model outputs by assessing responses against project guidelines for accuracy, helpfulness, tone, and safety, providing clear written justifications for your judgments.
Stress-test models by probing them adversarially to identify failure modes, unsafe behaviors, and capability gaps, then document reproducible cases for engineering and research teams.
Create training and evaluation datasets by authoring high-quality prompts, responses, and exemplars according to detailed specifications, and editing machine-written or human-written outputs to standard.
Build and apply rubrics and taxonomies by contributing to the design of grading criteria and then applying them consistently across different task types to produce structured annotations.
Annotate and correct data across text, image, and structured formats such as JSON, CSV/TSV, and Markdown, maintaining high data integrity and accuracy.
Participate in calibration exercises and inter-annotator agreement checks to maintain consistency, and flag ambiguous or uncovered edge cases rather than making unsupported judgments.
Report on model performance trends and communicate quality findings to cross-functional partners, providing clear evidence of where models succeed, fail, and degrade.

Requirements

At least one year of experience in AI data annotation, large language model evaluation, content moderation, research, or a related analytical role with exposure to quality assurance or preference ranking.
Demonstrated ability to apply detailed guidelines to complex and ambiguous content, with strong contextual judgment, sensitivity to nuance and tone, and skill in reasoning through cases without a single correct answer.
Willingness to flag unclear or uncovered edge cases rather than guess, and capability to work on novel experimental tasks whose definitions are still evolving.
Sharp attention to detail and ability to identify inconsistencies, subtle errors, and model failure modes; familiarity with large language model behaviors such as hallucination and instruction-following gaps is advantageous.
Excellent written English command and strong reading comprehension, with the ability to justify evaluations clearly and write well-reasoned prompts and exemplars; fluency in another language is a bonus.
Strong commitment to accuracy and consistency when working on high-volume tasks, combined with comfort using annotation platforms and structured data formats.
Ability to work independently and effectively in a remote environment, including strong time management and comfort learning new tools in a global, asynchronous team.

What makes this role worth considering

This is a part-time contract position offering 16 hours per week at CAD $30 per hour, fully remote within Canada. Your work directly shapes the capabilities of Cohere’s frontier models and contributes to making enterprise AI systems safer and more effective for real-world business use.