Career coaching · London
Every company shipping AI needs people who can prove it works. Testing AI apps and agents is a fast-growing specialism — a natural next step for QA.
A strong QA background, but unsure how testing non-deterministic AI differs from testing classic software
Confusing job titles: AI QA Engineer, LLM Evaluation Engineer, AI Test Engineer, Model Evaluation
No portfolio showing AI testing — only traditional test automation
Not knowing which evaluation tools and methods employers actually expect
Domain expertise (finance, legal, customer support) that is valuable for human evaluation — but no idea how to position it
Worrying that AI will replace testers, rather than seeing the new work it creates
Automated tests for AI: evaluation suites in CI with code-based checks and thresholds that fail the build
Evaluation datasets built from real user tasks, each with a clear definition of done
LLM-as-judge — and calibrating it against human review
Human-in-the-loop: rubrics, annotation workflows, inter-annotator agreement, human approval for risky agent actions
Agent testing: task success, trajectory and tool-call evaluation (including MCP), cost and latency
Agent skills evaluation: does the agent trigger the right skill, with-skill vs without-skill comparisons, consistency across runs
Safety: prompt injection, data leakage, red-teaming (OWASP Top 10 for LLM Applications)
Python, plus tools such as DeepEval, Ragas, Promptfoo, LangSmith or LangFuse
A practical path from QA or from your own domain:
Junior
£45–60k (moving from QA automation)
Middle
£60–85k
Senior
£85–120k+ (lead / evaluation engineer)
Indicative — a new role with few UK benchmarks. The UK Automation QA median is £60k (IT Jobs Watch, Oct 2026); senior evaluation roles track LLM engineer pay, higher in London
I built the evaluation framework and testing strategy for GenAI and agentic AI products that became my company’s standard, and I hire for AI roles — and I coach QA engineers and other professionals moving into AI testing and evaluation. Sessions in English or Russian.