We are looking for a QA Engineer to own testing across the full platform. In a compliance context, a software defect is not just a technical issue — it carries real financial and regulatory consequences. You will be involved from the earliest stages of development, defining acceptance criteria, building test frameworks, and ensuring that every component of the system meets the accuracy and reliability standards required before it goes near a production compliance workflow.
Key Responsibilities
Test Strategy & Planning
Develop and own the overall test strategy for the platform, covering all layers — data pipelines, ML models, APIs, application services, and frontend interfaces.
Work with the Product Manager to define acceptance criteria for every user story before development begins, not after.
Define accuracy and reliability thresholds for each ML model in scope — what performance levels are required before a model is considered production-ready.
Maintain a living test plan that evolves with the product as new use cases are added.
Functional & Business Logic Testing
Translate compliance rules and business logic into structured, executable test cases.
Work closely with the subject matter experts to identify edge cases and exception scenarios that automated tests must cover.
Validate that AI outputs — channel routing decisions, classification recommendations, and generated responses — are correct against defined compliance ground truth.
Test end-to-end workflows from data ingestion through to user-facing outputs, ensuring correctness at every stage.
ML & AI System Testing
Build regression test suites for all deployed models so that retraining does not break existing correct behaviours.
Test model outputs for consistency, groundedness, and accuracy using both automated evaluation and structured manual review.
Validate fallback mechanisms — ensure the system behaves correctly when model confidence is low or an upstream service is unavailable.
Test feedback loop integrity — verify that validator decisions are correctly captured and routed to the model retraining pipeline.
API & Integration Testing
Build and maintain API test suites for all integration endpoints — covering correctness, error handling, and edge cases.
Perform load and performance testing on real-time inference endpoints to validate behaviour under expected production volumes.
Test all third-party and government portal integrations against documented API contracts, including failure and retry scenarios.
User Acceptance & Production Readiness
Coordinate and facilitate user acceptance testing with the RIL compliance and assurance teams.
Produce a formal production readiness assessment before each deployment, documenting what has been tested, what passed, and any known residual risks.
Monitor for defects in production and manage the defect lifecycle through to resolution.
Required Qualifications
Education
Experience
4+ years of QA engineering experience with at least 2 years testing AI, ML, or data-heavy systems in production.
Experience writing and maintaining automated test frameworks, not just manual testing.
Demonstrated ability to translate complex business rules into structured test cases.
Technical Skills
Test Automation: Pytest, Selenium, Playwright, or equivalent for backend and frontend automation.
API Testing: Postman, REST-assured, or equivalent; experience with contract testing.
Performance Testing: Locust, JMeter, or equivalent for load and stress testing.
ML Testing: Experience validating ML model outputs, building evaluation datasets, and running regression suites against retrained models.
Data Testing: Experience validating data pipeline outputs — schema checks, completeness, and consistency.
Languages: Python for test scripting and automation.
CI Integration: Experience integrating test suites into CI/CD pipelines for automated test execution.
Preferred Qualifications
Prior experience testing compliance, fintech, or legal technology systems where business logic accuracy is critical.
Familiarity with LLM evaluation frameworks — RAGAS, TruLens, or equivalent.
Experience with exploratory testing techniques for AI systems where outputs are probabilistic.
Prior exposure to regulated environments where formal sign-off and test documentation are required.
Core Competencies
Technical Competency
Behavioural Competency
Test Strategy & Planning
Attention to Detail
ML & AI System Testing
Proactive Risk Identification
API & Integration Testing
Cross-Functional Collaboration
Business Logic Validation
Ownership & Accountability
Test Automation
Structured Problem Solving