QA Engineer : 200 systems in production, and they have to hold up
Past a certain number of systems in production, the constraint is no longer building. It is guaranteeing that it holds. That is this role, and it is new. A useful clarification: this is QA for AI systems, not interface testing. You test non-deterministic outputs, not buttons.
What you take on
You write an evaluation suite, and it lies
You build a test set for a client's support agent. It passes at 95%. Two weeks later a user asks a question phrased unlike anything in your test set, and the system answers nonsense. Your job: maintaining a test set that keeps closing in on real traffic, week after week.
You break an n8n workflow for good reasons
And check that nothing else breaks with it. A node is changed to fix a bug; a downstream step that depended on a precise output format fails silently. Your regression tests catch that before the client does.
You are the last gate before delivery
Business test set, the client's edge cases, checking that the outputs are presentable to a leader who will use them to decide. You say yes or no to going live.
You verify a number the company repeats everywhere
“7 hours a week recovered on average” is a central commercial argument, and a verifiable claim. You build the method to audit it project by project, and you say when it does not hold for a given case.
You watch for what degrades without warning
Quality dropping over several weeks with no alert, token costs blowing up. You build the signals that make those drifts visible before the client sees them.
You write a lot, and to be understood immediately
Acceptance reports, non-conformity notes, drift write-ups: readable in thirty seconds by someone who has not followed the project.
Month 1, you audit what exists and tell us where it breaks most often. Month 2, you put the evaluation foundation in place on a live system. Month 3, no system ships to a client without going through your acceptance.
Ce qu'on cherche
- Evaluation techniques
- Golden datasets, scoring rubrics, LLM-as-judge calibrated against human annotation, regression testing
- Tools
- Solid Python, an eval framework (promptfoo, DeepEval or equivalent), reading and testing n8n workflows, CI/CD
- Non-determinism
- Thinks in acceptable quality ranges, tells normal variation from a real regression
- Writing
- Clear reports, readable by a non-technical audience
- Field work
- 3 to 8 years in QA, test engineering or SDET, part of it on AI systems in production, comfortable saying no under pressure
Ce qu'on ne cherche pas
Someone who executes test plans written by others. Here, you write the strategy.
Where it differs from neighbouring roles
| Role | What they do | Who owns the outcome |
|---|---|---|
| Generalist QA or manual tester | Predefined scenarios on deterministic software | Functional coverage owns the outcome |
| MLOps or Platform Engineer | Infrastructure that runs the models | Technical availability owns the outcome |
| Data Scientist or ML Engineer | Designs and trains the models | Upstream model performance owns the outcome |
| QA Engineer, AI Systems, AI Makers | Builds the evaluations, decides go or no-go | You own the reliability of the published numbers |
Remote, in a time zone compatible with Paris and Casablanca (Europe, North Africa or nearby), with no requirement to live in either city, and occasional time in our offices.
Is this role for you?
No cover letter. The form takes five minutes and starts with showing what you have built.
Apply for QA Engineer