AI Assistant JustAsk Your AI App Industries How it works Case studies Pricing Help Careers Sign in Build my AI assistant free
Dashboard AI Assistants Team assistant Knowledge library Team Profile Help
Theme
AI Engineering

Machine Learning Engineer, Evaluation & Quality

Build the measuring stick for answer quality: test sets, LLM-as-judge pipelines and dashboards that tell us whether an assistant is getting better.

Location
Surry Hills, NSW
Work
Hybrid or work from home (Australia)
Type
Full-time
Level
Mid-level

About the role

We change prompts, models and retrieval often. Your job is to make sure every change is a step forward. You'll build the evaluation harness the whole team runs before shipping: golden question sets per industry, automated grading with human spot checks, regression alerts and the Knowledge Gaps signal that tells businesses what their assistant couldn't answer.

What you'll do

  • Build and maintain evaluation datasets across industries and languages
  • Create grading pipelines (LLM-as-judge with calibration against human ratings)
  • Run model comparisons when new models arrive and recommend what we should switch to
  • Improve how unanswered questions are detected and grouped into Knowledge Gaps
  • Report quality trends the whole company can read

What you'll bring

  • 3+ years in ML engineering, data science or a closely related role
  • Strong Python and data tooling; comfortable with statistics and experiment design
  • Experience evaluating generative models, not only classifiers
  • Healthy scepticism about metrics, and the patience to read a lot of transcripts

Nice to have

  • Annotation tooling or inter-rater agreement experience
  • Background in search quality or information retrieval

How we work

You'll be part of a small team based in our Surry Hills office, L1 457 Elizabeth Street, Surry Hills, NSW 2010, Australia. Come in when it helps and work from home when it doesn't: the role is open to people anywhere in Australia. We agree on outcomes and a few shared hours, then trust you with the rest.

Apply for this role

Machine Learning Engineer, Evaluation & Quality · Surry Hills, NSW · hybrid or work from home

CV (PDF or Word, up to 8 MB)
All open roles

More open roles

See all 14 roles
AI Engineering Hiring now

Senior AI Engineer, Retrieval & RAG

Own the retrieval layer every owns.ai answer stands on: hybrid search, chunking, ranking and the evaluations that prove each change made answers better.

  • Surry Hills, NSW
  • Hybrid · Work from home
  • Full-time
  • Senior
Posted 29 SepView role
AI Engineering

Graduate AI Engineer (2027 intake)

Start your career shipping real AI features to thousands of businesses, with a mentor, a structured first year and code in production from your first month.

  • Surry Hills, NSW
  • Hybrid · Work from home
  • Full-time
  • Graduate
Posted 7 OctView role
AI Engineering

Applied AI Engineer, Conversational Agents & Bookings

Teach assistants to get things done: book appointments, take call-backs and send jobs to Jobber, ServiceM8, Housecall Pro and simPRO without a human in the loop.

  • Surry Hills, NSW
  • Hybrid · Work from home
  • Full-time
  • Mid-level
Posted 6 OctView role