AI Senior Software Developer Engineer in Test
AI Senior Software Developer Engineer in Test
Role Purpose
The AI SDET leads quality engineering for AI-enabled products, agent workflows, and the conventional software systems that connect them. The role turns product intent, evaluation taxonomies, and production risks into repeatable tests and evaluation evidence. It builds the automated evaluation and regression capability needed to assess changing prompts, models, data, and agent behavior, while preserving clear human judgment for ambiguous or high-impact outcomes.
Scope and Autonomy
Owns quality strategy and evaluation engineering across agent behavior, APIs, data flows, dashboards, and supporting services. Works with product subject matter experts and software engineers to define what acceptable behavior means before automating checks. Uses high agency to identify missing test cases, data, requirements, and failure patterns; escalates unresolved business definitions, privacy concerns, or release risks to accountable owners.
Role Boundaries
The AI SDET provides evidence and release recommendations; the role does not define business policy or taxonomy without the relevant product or client owner, approve production releases unilaterally, or authorize use of unapproved AI services or data. Evaluation scores are evidence, not a substitute for product judgment. Sensitive production conversations, credentials, personal information, and client data must be handled only through approved controls.
Core Responsibilities
AI Evaluation Strategy and Test Design
Translate agent requirements, user journeys, acceptance criteria, and non-functional expectations into measurable evaluation plans and risk-based test coverage.
Design evaluations for correctness, relevance, grounding, instruction following, consistency, and latency. Build rigorous tests for policy-as-code guardrails, verifying appropriate refusal, safe state recovery, and mandatory human-in-the-loop (HITL) escalations to prevent excessive agency in irreversible tool calls.
Partner with product and domain experts to define expected outcomes, evaluation dimensions, taxonomies, rubrics, edge cases, and examples before they are encoded in tests.
Build representative datasets from approved synthetic, curated, or production-derived examples. Track provenance, coverage, labeling quality, and known limits.
Evaluation Automation and Regression
Develop and maintain automated evaluation suites for agent workflows, retrieval, prompt and model changes, tools, APIs, and end-to-end product behavior.
Create reliable comparison methods for non-deterministic outputs by integrating specific NLP and statistical evaluation metrics (e.g., F1 Score, RMSE, perplexity, BLEU/ROUGE scores), confidence considerations, human review, and explicit treatment of borderline results.
Integrate evaluations into CI/CD and release workflows so changes to prompts, models, code, data, or configuration produce timely, interpretable quality evidence.
Version evaluation datasets, taxonomies, prompts, model settings, expected results, and run metadata so results can be reproduced and compared.
Quality Data, Diagnostics, and Product Feedback
Establish observability patterns to monitor model drift and performance degradation in production environments.
Investigate failures and regressions by utilizing trace-level observability to track full agent trajectories, including multi-agent handoffs, reasoning spans, context construction, and tool invocations. Analyze these traces to debug non-deterministic state transitions and identify where latency, cost, or logic errors compound.
Analyze approved production interactions and user feedback to identify recurring failure patterns and convert them into prioritized test cases and product improvements.
Build clear evaluation reporting and dashboards that show coverage, trends, failures, uncertainty, and release implications without reducing quality to one score.
Verify that evaluators themselves are useful, calibrated, and stable; identify false positives, false negatives, bias, data leakage, and metric gaming.
Regulatory & Ethical Compliance and Quality Leadership
Establish a continuous, iterative risk management process throughout the AI system lifecycle, evaluating both intended use and reasonably foreseeable misuse. Apply formal risk management and AI regulations (e.g., EU AI Act principles) to evaluation design. Conduct bias and fairness auditing across different demographics to ensure equitable performance. Lead quality reviews and communicate evidence, residual risk, and human oversight measures.
Adversarial & System Security Testing
Stress-test AI systems to determine operational limits, latency constraints, and performance regressions. Perform rigorous adversarial testing by crafting malicious prompts and inputs to test for 'jailbreaks,' prompt injections, and to ensure robust security postures against adversarial attacks.
Data Pipeline & Lineage Integrity
Validate the quality, lineage, and bias of training and inference datasets to prevent 'garbage-in, garbage-out' scenarios. Verify that datasets are relevant, representative, and free of errors. Collaborate with engineering to refine feature selection and fine-tuning strategies based on testing feedback.
Accountability and Success Measures
Evaluate impact using trustworthy evidence over time. Evaluation count, pass rate, model score, automation percentage, or one benchmark should not be used alone; interpret results alongside dataset quality, coverage, production behavior, and human review.
- Evaluation validity: Evaluation criteria reflect documented product expectations and are reviewed with the appropriate subject matter experts.
- Regression confidence: Changes to agents, models, prompts, data, and services receive repeatable, interpretable regression feedback.
- Dataset and taxonomy quality: Evaluation data is representative, permissioned, versioned, traceable, and explicit about limitations.
- Failure discovery: High-impact behavior gaps and recurring production failure patterns become actionable tests and engineering improvements.
- Release evidence: Recommendations explain coverage, uncertainty, known issues, residual risk, and required human decisions.
Edify Competencies
- Connection
Builds trust through clear communication, dependable follow-through, and constructive collaboration with engineering, product, operations, and client stakeholders.
- Vision
Connects technical choices to user outcomes, system risk, delivery goals, operational constraints, and long-term maintainability.
- Empowerment
Creates shared context, clear interfaces, useful documentation, and coaching that enable teams to own their work.
- Initiative
Anticipates gaps, acts on ambiguous problems, and escalates decisions when impact exceeds role
Position Requirements
Education and Experience
Technical degree or equivalent demonstrated capability in Computer Science, Software Engineering, Quality Engineering, Data Science, Machine Learning, Applied AI, or a related field.
Typically 5 or more years in quality engineering, test automation, or software engineering, with explicit hands-on experience evaluating machine learning pipelines, LLM-integrated applications, or agentic automation systems; equivalent demonstrated capability is accepted.
Ability to provide tangible evidence of applied AI quality engineering, such as documented case studies of AI evaluation, trace-level bug investigations, custom test data factories, or implemented quality gates for non-deterministic systems.
Technical and conversational English at B2+ or higher under the Common European Framework of Reference.
Required Capability and Tools
Strong programming capability in Python, TypeScript, JavaScript, Java, or a comparable language, including debugging, code review, version control, modular design, and maintainable automation.
Experience designing automated tests for APIs, integrations, data workflows, distributed services, and end-to-end user journeys, with practical CI/CD integration.
Practical understanding of AI/ML foundations, including model architectures (Transformers, CNNs, RNNs), training processes, fine-tuning, and RAG (Retrieval-Augmented Generation) architectures. Strong grasp of prompt crafting techniques (Chain-of-Thought, Few-Shot) and the distinction between black-box AI testing and white-box structural testing.
Ability to design evaluation datasets, rubrics, taxonomies, baselines, regression suites, and meaningful metrics; able to explain the limits of automated judges and scores.
Experience with test data management, environment controls, defect investigation, risk-based testing, requirements traceability, and release readiness assessment.
Strong judgment in privacy, security, safety, and responsible handling of production or client data. Deep understanding of AI-specific security vulnerabilities—such as direct/indirect prompt injection, training data poisoning, insecure output handling, and excessive agency—and the ability to design test cases that mitigate these risks.
Useful Additional Experience
Experience with specific AI evaluation frameworks (e.g., RAGAS, DeepEval, LangSmith), adversarial security tools (e.g., Promptfoo, PyRIT), and data/observability platforms (e.g., MLflow, Weights & Biases). Familiarity with Vector Databases (Pinecone, Milvus) and containerized infrastructure.
Experience in education technology, career coaching, recommendation systems, conversational products, voice systems, or other domain-sensitive applications.
Experience with accessibility, performance, security, bias, robustness, or human-factors testing for AI-enabled products.
Edify is proud to be an equal-opportunity employer committed to diversity and inclusion. We encourage all candidates to apply regardless of their sex, gender identity, sexual orientation, race, color, religion, national identity, disability, age, medical condition, pregnancy, marital status, family status or any other characteristic protected by law.