Job Description

Job Description

Job Description

Come, Disrupt and Thrive.

AI Agent Developer & Test Engineer

Overview

KēSTA I.T. is seeking an experienced AI Agent Developer & Test Engineer to join a cutting-edge team focused on building, testing, and supporting next-generation AI-powered applications. This role combines hands-on AI development with rigorous testing and quality assurance responsibilities, ensuring intelligent agent workflows perform reliably, securely, and consistently throughout the software development lifecycle.

The ideal candidate will possess strong experience with agent-based architectures, Python development, automated testing frameworks, cloud environments, and modern observability practices. This individual will play a key role in designing, validating, and supporting AI-driven solutions operating in production environments.

Responsibilities

• Design, develop, and maintain AI agent workflows utilizing modern agent orchestration frameworks.
• Build and execute comprehensive testing strategies for AI agent systems, including unit, integration, regression, and end-to-end testing.
• Develop node-level test coverage to validate individual workflow components in isolation.
• Perform full workflow and graph-level testing to validate complete agent execution paths.
• Implement effective LLM mocking and simulation strategies to support deterministic and repeatable testing.
• Create and maintain regression test suites to ensure defects remain resolved over time.
• Design testing methodologies that address non-deterministic AI behaviors through controlled inputs, seeded testing, and assertion strategies.
• Develop and maintain automated end-to-end testing coverage for user-facing applications and workflows.
• Monitor test coverage metrics and continuously identify opportunities to improve system quality and reliability.
• Validate structured outputs, schema compliance, and data integrity across AI workflows.
• Implement production monitoring, logging, tracing, and observability practices for AI systems.
• Support cloud-based application deployments and ongoing operational maintenance.
• Collaborate closely with software engineers, architects, product owners, and QA teams to deliver high-quality AI solutions.
• Participate in troubleshooting, root-cause analysis, and production support activities.

Required Qualifications

• Professional experience developing AI-powered applications using agent frameworks such as LangGraph and/or LangChain.
• Strong Python development experience.
• Experience with automated testing frameworks including pytest and pytest-asyncio.
• Experience designing and executing unit tests for complex workflow-based applications.
• Experience testing distributed or orchestrated agent workflows from end to end.
• Strong understanding of test isolation, mocking strategies, and simulation techniques.
• Experience managing regression testing programs and quality assurance processes.
• Hands-on experience with Playwright or similar end-to-end testing frameworks.
• Experience producing and interpreting test coverage metrics and reporting.
• Familiarity with structured output validation and JSON schema testing.
• Experience implementing monitoring and observability solutions for production applications.
• Cloud application development and support experience within modern cloud platforms (GCP preferred).
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and collaboration abilities.

Preferred Qualifications

• Experience with AI evaluation and observability tools such as LangSmith, RAGAS, or similar platforms.
• Experience managing prompt versioning and prompt lifecycle processes.
• Knowledge of adversarial testing, red-team testing, and AI security validation methodologies.
• Experience monitoring token consumption, model utilization, and AI operational costs.
• Experience working with document extraction, OCR, or intelligent document processing solutions.
• Experience supporting applications within regulated industries such as financial services, healthcare, or government.

Preferred Platform & DevOps Experience

• Google Cloud Platform (GCP)
• Cloud Run
• Terraform
• Azure DevOps
• CI/CD pipeline design and automation
• Infrastructure-as-Code (IaC) best practices

Why Join Us?

This is an opportunity to work on innovative AI initiatives while helping shape the quality, reliability, and operational excellence of intelligent software systems. You'll collaborate with talented technical professionals and contribute directly to the advancement of AI-enabled business solutions.

Available Benefits

• Medical Benefits (Platinum level plans available)
• Work from home / Hybrid / Onsite options
• PTO
• Holiday Pay
• VTO
• 401K
• Charitable Match
• Training reimbursement

About KēSTA I.T.

Our name says it all; KēSTA I.T. (Keys-to-I.T.) AND our people are our keys to our success!

KēSTA I.T. is a premier Utah-based technical staffing and consulting services firm. We specialize in temporary and permanent placement of Software, Hardware, Network, Cloud, CRM/ERP, Data, End-User Support, Web, and Executive/Leadership -based positions on a full-time and consulting basis.

If you're interested in a role where top performance is rewarded, personal time is valued, and excellence is demanded at every level, we want to talk to you today!

Where do you want to go? We've got the keys! ~ KēSTA I.T.

Job Tags

Similar Jobs