Human evaluation of AI and MT output
Native speakers rate, rank and annotate errors in LLM and machine translation output, in the languages and dialects your model serves. Start with a free pilot batch.
Human evaluation of AI output is the review of what a large language model or a machine translation engine produces by native speakers of the target language, who rate, rank and annotate errors in the output against an agreed rubric.
El Turco runs this work for AI and ML teams that build or adapt their own models, and for the AI data units of language service providers and data vendors, who can use the same native raters under their own brand.
Most models are trained mainly on English and Western European languages. In Turkish, Arabic dialects, Persian and the other regional languages we cover, native judgment shows where the output fails: idioms, register, dialect and answers that read as translated English.
Need training data as well? See AI training data for speech, text annotation, moderation and custom MT and LLM data.
Scope
What we evaluate
- Rating
- Native raters score LLM responses and machine translation output for accuracy, fluency, cultural appropriateness and terminology against an agreed rubric.
- Ranking
- Raters compare and rank model responses, check facts and judge whether an answer suits the culture of the reader. The rankings feed alignment and fine-tuning.
- Error annotation
- Raters mark what is wrong in each output, such as answers that read as translated English or Persian replies that drift into Arabic vocabulary.
- RLHF and preferred answers
- Native linguists write preferred answers for instruction tuning, and domain prompts go to raters with matching subject experience.
- Evaluation sets and MT benchmarking
- Evaluation sets built from your domain content and terminology, used to benchmark engine output in each language and rerun after each model update.
Process
How an evaluation batch runs
Timelines are agreed at scoping and data is delivered in batches, so your team can work with early results while the rest is rated.
Scoping
We review the use case, languages and varieties, volume, output formats and your acceptance criteria.
Rubric and calibration
The rating rubric and guidelines are co-developed with you and versioned per batch. Calibration rounds align raters on hard cases before production starts.
Native rating
Approved native raters matched to the language, variety and domain work in your own platform or annotation tool, or deliver into the schema agreed at the start.
Quality control
Gold tasks are seeded throughout and raters are tracked against them. Disagreements are reviewed and the rubric updated.
Delivery
Ranking and rating exports matched to your pipeline format, by secure transfer, API or your own platform.
Languages and varieties
The target variety is fixed per project, and raters are native speakers of that variety.
- MSA, Gulf, Levantine, Egyptian, MaghrebiArabic
Modern Standard Arabic plus the four main dialect groups. Native speakers of each dialect rate that dialect.
- Native Turkish ratersTurkish
Fluency and factual accuracy, answers that read as translated English and domain feedback on business, legal and medical content.
- Iranian PersianPersian (Farsi)
Raters grade how natural and accurate each answer is and mark replies that sound translated from English or drift into Arabic vocabulary.
- All 19 languagesThe other regional languages
RLHF and evaluation work is available in every language we cover, including the Balkan, Eastern European and Turkic languages.
Quality and security
How rater quality is checked
Work runs under our ISO/IEC 27001 certified information security management system. Every linguist and annotator signs an NDA, access is limited to the team assigned to your project and your data is used only for your project.
- Rating rubric and guidelines co-developed with you, versioned per batch
- Calibration rounds before production starts
- Gold tasks seeded throughout, with raters tracked against them
- Inter-annotator agreement measured and reported per delivery
- A QA report with the free pilot batch
Free pilot batch
Test the rubric and the raters first.
Send us your spec and we will deliver a free pilot batch of up to 100 annotated samples or 30 minutes of collected audio, with a QA report included. You evaluate the output against your own acceptance criteria before committing to production.
FAQ
AI evaluation questions
What is human evaluation of AI output?
Human evaluation of AI output is the review of what a large language model or a machine translation engine produces by native speakers of the target language, who rate, rank and annotate errors in the output against an agreed rubric. The results show an AI team where a model fails in each language and dialect. El Turco runs this work for AI and ML teams and for the AI data units of language service providers.
Which kinds of AI output does El Turco evaluate?
El Turco evaluates LLM responses and machine translation output. Native raters score accuracy, fluency, cultural appropriateness and terminology, rank responses against each other, mark errors and write preferred answers for instruction tuning. Evaluation sets can be built from the client domain content and rerun after each model update, so teams can track progress language by language.
Which languages and dialects can El Turco evaluate?
El Turco evaluates Arabic in Modern Standard Arabic and the Gulf, Levantine, Egyptian and Maghrebi dialects, with native speakers of each dialect, as well as Turkish and Iranian Persian. RLHF and evaluation work is available in all 19 languages El Turco covers, including the Balkan, Eastern European and Turkic languages.
How does El Turco check the quality of its raters?
El Turco co-develops the rating rubric with the client and versions it per batch. Calibration rounds align raters before production starts. Gold tasks are seeded throughout the work and every rater is tracked against them. Inter-annotator agreement is measured and reported with each delivery, and disagreements are reviewed so the rubric can be updated.
Can El Turco raters work in our platform or under an LSP brand?
Yes. El Turco raters can work inside a client's own platform or annotation tool, or deliver into a custom schema agreed at the start. Language service providers and data vendors can use the same native raters on a white-label basis, under the LSP brand and in the LSP workflow, following the end client's guidelines.
How does an evaluation project start and how is it priced?
An evaluation project starts with a free pilot batch of up to 100 annotated samples or 30 minutes of audio, with a QA report included. The AI team checks the pilot against its own acceptance criteria before production. Timelines are agreed at scoping and data is delivered in batches. Per-word, hourly or project-based pricing, quoted after scoping.
Talk to us about evaluating your model.
Share your languages, varieties, output type and volume. We reply with a spec, a timeline and a quote. Pricing is per-word, hourly or project-based, quoted after scoping.
A project manager reads your brief, usually the same working day.