Site Reliability Engineer, Inference Infrastructure

Advertisements
IT & Technology 🇨🇦 Toronto 12 Jan 2026
This position has been filled. Explore our other open positions here.

Role overview

Cohere is an enterprise AI company building foundation models and end-to-end products for large-scale business applications. As a Site Reliability Engineer on the Model Serving team in Toronto, you will develop and operate the infrastructure that delivers Cohere’s large language models through API endpoints to production environments. This role bridges infrastructure reliability with machine learning deployment, requiring you to build systems that enable low-latency, high-throughput, and highly available model serving.

Key responsibilities

  • Build self-service automation systems for managing, deploying, and operating services, including custom Kubernetes operators that support language model deployments.
  • Design and implement observability and resilience automation to enable all developers to troubleshoot and resolve infrastructure problems independently.
  • Take ownership of meeting defined Service Level Objectives, including participation in an on-call rotation for production systems.
  • Work with internal developer teams to understand their infrastructure needs and influence the broader Infrastructure team’s roadmap based on collected feedback.
  • Deploy optimized NLP models to production in environments prioritizing low latency and high throughput, while also working with customers on customized deployments to meet their specific requirements.
  • Mentor team members through knowledge sharing and active code review processes.

Requirements

  • At least five years of engineering experience operating production infrastructure at large scale.
  • Demonstrated experience designing large, highly available distributed systems using Kubernetes and managing GPU workloads on those clusters.
  • Practical experience with Kubernetes development, production coding, and operational support.
  • Experience with cloud platforms including GCP, Azure, AWS, or OCI, as well as multi-cloud, on-premises, or hybrid serving architectures.
  • Strong background designing, deploying, supporting, and troubleshooting complex Linux-based computing environments.
  • Familiarity with computational characteristics of accelerators such as GPUs, TPUs, or custom accelerators, and how they affect inference latency and throughput.
  • Working knowledge of distributed systems and proficiency in high-performance server languages such as Golang or C++.

What makes this role worth considering

You will join a global technology company headquartered in Toronto with offices across London, New York City, San Francisco, Montreal, Paris, Berlin, and Seoul. Cohere offers comprehensive benefits including a weekly lunch stipend, full health and dental coverage, education and learning support, six weeks of paid vacation, parental leave top-up, and a home office stipend. The role combines infrastructure reliability engineering with cutting-edge AI deployment challenges, offering the opportunity to shape how enterprise AI systems scale and perform in production.

How to apply

Use the Apply button to view the official job posting and submit your application on Cohere's careers site.

ImmigrationCafe is a job listing directory, not a recruiter. Applications are submitted on the employer's official careers website.

Advertisements

Location