STORM ASR 2.0 benchmark

Our most accurate speech recognition for Nigerian languages yet, tested against the leading Nigerian speech recognition models on the same 22,921 recordings in Yoruba, Hausa, Igbo, Nigerian Pidgin, Nigerian English and code-mixed speech.

15.6words wrong per 100 across all languages, the lowest of every model tested
31%fewer word errors than the next best model (22.6)
6 of 6test groups (five languages and code-mixed) where STORM ASR 2.0 is most accurate
5.6letters wrong per 100, with tone marks counted

Words wrong per 100 words spoken

Lower is better. Same 22,921 recordings for every model.

  • STORM ASR 2.0
  • SBPN Large
  • Meta MMS-1B
English
4.5
9.0
19.3
Hausa
11.9
18.1
28.2
Igbo
22.2
28.7
52.3
Pidgin
14.2
15.5
50.4
Yoruba
19.4
24.5
51.6
Code-mixed
14.8
25.1
69.0
All languages
15.6
22.6
51.1

Word error rate

Words wrong per 100 words spoken. A wrong, missing or extra word each count as one error. Best result in each row highlighted.

LanguageRecordingsSTORM ASR 2.0SBPN LargeMeta MMS-1BNCAIR
Nigerian English1,1884.59.019.312.1
Hausa3,61511.918.128.2n/a
Igbo4,66222.228.752.3n/a
Nigerian Pidgin1,36914.215.550.4n/a
Yoruba2,49919.424.551.644.3
Code-mixed (Yoruba + English)9,58814.825.169.058.0
All languages22,92115.622.651.1n/a

Letter error rate

Letters wrong per 100 letters. A missing or wrong tone mark counts as a wrong letter, so this shows how well each model writes Yoruba, Igbo and Hausa as they are really written.

LanguageRecordingsSTORM ASR 2.0SBPN LargeMeta MMS-1BNCAIR
Nigerian English1,1881.93.56.86.1
Hausa3,6153.24.77.4n/a
Igbo4,6628.010.117.6n/a
Nigerian Pidgin1,3697.47.621.4n/a
Yoruba2,4997.79.818.820.1
Code-mixed (Yoruba + English)9,5885.311.127.436.4
All languages22,9215.68.718.7n/a

Examples

Recordings from the test set, as each model wrote them.

Code-mixed
What was said

Babangida ló hand over power fún interim government

STORM ASR 2.0

Babangida ló hand over power fún interim government

SBPN Large

babangida ló wan overpower fún interim government

Yoruba
What was said

Àlùfáà ìjọ aṣọ ara kan ṣe ìṣirò ìgbà tí Jésù yóò dé.

STORM ASR 2.0

Àlùfáà ìjọ aṣọ ara kan ṣe ìṣirò ìgbà tí Jésù yóò dé.

SBPN Large

àlùfáà ìjọ aṣọ ara kan ṣe ìṣe èrò tí jésù yóò dé

Igbo
What was said

a naghị ekwe ka nwaanyị di ya nwụrụ rie nri e siiri ndị ezinụlọ ya

STORM ASR 2.0

A naghị ekwe ka nwaanyị di ya nwụrụ rie nri e siiri ndị ezinụlọ ya

SBPN Large

anaghị ekwe ka nwaanyị di ya nwụrụ rie nri e siri ndị ezinụlọ ya

Nigerian Pidgin
What was said

Chicken wey run way from Borno go Ibadan go still end up inside pot of soup

STORM ASR 2.0

Chicken wey run way from Borno go Ibadan go still end up inside pot of soup

SBPN Large

chicken wey run away from borno go ibadan go still end up inside port of soup

Nigerian English
What was said

First, Hurricane Rosa's path leads to landfall across the Baja California peninsula of Mexico.

STORM ASR 2.0

First, Hurricane Rosa's path leads to landfall across the Baja California peninsula of Mexico.

SBPN Large

first hurricane rose's path leads to landfall across the baha california peninsula of mexico

How we tested

  • Test set. 22,921 recordings, about 34 hours, in Nigerian English, Hausa, Igbo, Nigerian Pidgin, Yoruba and Yoruba mixed with English. Every model transcribed exactly the same recordings.
  • STORM ASR 2.0 was measured through the live production service, exactly as customers receive it, with no language given: it recognises the language on its own. Meta MMS-1B and the NCAIR models were given the correct language for every recording.
  • Scoring. Capitals and punctuation are ignored; tone marks count, so a missing or wrong tone mark makes a word wrong. Recordings whose reference text contains digits were left out, because models write numbers differently. All counts were made by our own scoring code from each model's raw transcripts.

Models compared

  • SBPN Large (ogunlao/SBPN_multilingual_large), a multilingual model for Nigerian languages, licensed for research use only.
  • Meta MMS-1B (facebook/mms-1b-all) with its Yoruba, Hausa, Igbo, Pidgin and English adapters.
  • NCAIR Yoruba-ASR and NigerianAccentedEnglish, which cover Yoruba and Nigerian English only.

Each was run from its published weights with its default settings.

STORM ASR 2.0 is available now.

Files and real-time streams, no language setting needed, one API key.