Role overview

TribalScale is seeking an Agentic QA Engineer to ensure AI agents and large language model-powered applications perform reliably in production environments. This role combines software quality engineering expertise with hands-on experience validating AI-driven systems, focusing on testing strategies, automation frameworks, and the unique challenges of working with dynamic, nondeterministic AI outputs.

Key responsibilities

Design and implement comprehensive testing strategies tailored to AI agents, LLM-powered applications, and agentic workflows.
Build automated test frameworks that cover functional, integration, regression, and end-to-end testing scenarios specific to AI systems.
Develop evaluation pipelines that measure agent accuracy, reliability, task completion rates, and consistency across multiple runs.
Test agent tool usage, API integrations, multi-step reasoning workflows, and how agents interact with external systems and services.
Identify, reproduce, and troubleshoot AI-specific failures including hallucinations, incorrect tool calls, unexpected outputs, and workflow breakdowns.
Establish quality metrics, evaluation datasets, test scenarios, and performance benchmarks for AI-driven applications.
Integrate automated testing and quality checks into CI/CD pipelines and collaborate with AI engineers, developers, product teams, and client-facing engineering teams to improve system reliability.
Investigate production issues and use findings to enhance agent behavior, expand test coverage, and improve overall application performance.

Requirements

Professional experience in Software QA, Quality Engineering, SDET, Test Automation, or Software Engineering with a strong emphasis on testing.
Proficiency in Python, JavaScript, TypeScript, or another relevant programming language.
Demonstrated ability to build automated tests and testing frameworks from the ground up.
Strong understanding of API, integration, regression, and end-to-end testing methodologies.
Experience debugging applications and investigating complex technical issues in production or development environments.
Familiarity with Git, CI/CD pipelines, and modern software development practices.
Strong analytical thinking skills and the ability to design test cases for complex, real-world scenarios.

What makes this role worth considering

This position sits at the intersection of software quality and emerging AI technologies, allowing you to shape how AI agents are validated and deployed. You’ll work directly with cross-functional teams at an innovation-focused company that partners closely with clients to bring transformative digital solutions into production, giving your work immediate real-world impact.