Speech is something very familiar to all of us. One advantage of using speech for communication is its quick and effortless responsiveness compared to written language. If we could use this not only for face-to-face conversations but also for operating computers and smartphones, it would be very convenient. Just like typing on a keyboard or tapping with fingers, integrating speech recognition into user interfaces seems beneficial. There are other reasons, but this is one of the motivations behind our development of ReazonSpeech.

Before ReazonSpeech's release, there were no large-scale corpus available for unrestricted commercial use in Japan. The fact that it's a freely usable speech corpus is one of ReazonSpeech's selling points.

Previously, if you wanted freely usable speech data without copyright restrictions, you had to invite people to labs or studios to speak into microphones. ReazonSpeech automated the processing of TV One-Seg broadcast data to create corpus, enabling large-scale corpus expansion.

In Japan, the 2019 copyright law revision allowed using copyrighted content for data analysis, redistributing it, and even using it commercially ahead of other countries. However, despite this progressive legislation, initially, everyone cautiously watched each other. Since no one was utilizing such a beneficial law, we thought why not take the lead and utilize it ourselves. Thus, ReazonSpeech was born from such aspirations.

I joined Reazon four years ago. I had been accumulating data steadily because I wanted to create such a corpus, but for speech recognition, it's necessary to organize not just speech but also subtitle data, or rather, textual data. Many over-the-air broadcasts include subtitle information. Before, we couldn't do this, but the law revision made it possible to use such data under a free license, which was significant.

The challenge was system development. Even if both speech and subtitle data flow during a program, they don't necessarily flow at the same time. Moreover, subtitle data may not accurately convert speech into text. Therefore, in actual development, we started by transcribing speech with existing speech recognition models, comparing the results with subtitles in the program to find matches. We turned these matched speech and subtitle pairs into corpus data. This matching process is called 'alignment,' and writing the program to achieve this took time and effort.

Another issue was that the existing speech recognition models used for alignment had to be open and free. Even if they were highly accurate, if their licenses weren't free, the corpus created with them couldn't be freely distributed. Therefore, initially, we used free speech recognition models for alignment and created small corpora. We repeated this process more than ten times, using these corpora to create speech recognition models, aligning them to create a larger corpus.

Not only has the corpus expanded to 35,000 hours, but in the past year, several open-source speech recognition models have been released. We have incorporated NVIDIA's NeMo open-source toolkit into ReazonSpeech v2. As a result, the speech recognition speed has increased sevenfold compared to version 1. Normally, there's a trade-off between speed and accuracy, but ReazonSpeech v2 manages to maintain both at a high level. This is the selling point of ReazonSpeech v2.

Previously with ReazonSpeech, if you had 60 minutes of recording, transcription would be completed in around 10 minutes. I thought that was already quite practical, but with ReazonSpeech v2, it's now capable of real-time transcription while speaking. This opens up possibilities like displaying speech as text on screen during remote conversations or creating text in real-time. You can also input text by voice instead of using a keyboard when writing longer passages on a computer. Additionally, it allows scenarios such as real-time reading of what a hearing-impaired person is saying during a conversation with a hearing person. Its application extends to various situations where real-time capability is crucial.

In ReazonSpeech v2, we trained the model not to omit punctuation marks from the training data. As a result, punctuation marks now appear naturally.

In version 1, we had a process that removed unnecessary symbols and spaces to ensure accurate transcription. This included cleanly removing punctuation marks. However, we decided to include them this time because we felt commas and periods are necessary. Nevertheless, to ensure these punctuation marks don't disrupt the coherence of the text, we carefully prepared the training data for machine learning.

This has been our philosophy since releasing version 1; we believe continuous releases are crucial. "Language" evolves along with societal changes. New words emerge, and vocabularies rapidly change. If our speech corpus doesn't reflect these shifts, it could gradually recognize only outdated language. Therefore, we plan to update it annually.

One thing I'd really like to do is multi-modal learning, incorporating not only voice and text but also facial expressions of speakers. Humans use information from various senses—auditory, visual, olfactory, gustatory, tactile—to perceive the world. Our current models utilize only a part of this. Television broadcasts, which serve as our base, naturally include visual information, often showing the face of the speaker when their voice is heard. I think everyone can relate to the fact that during conversations, we not only listen to words but also observe the speaker's facial expressions to glean hints and understand their intentions. If our corpus includes such information, the model will become more sophisticated. We plan to achieve this in version 3.

Besides releasing advanced models, we also want to foster a community. ReazonSpeech is an open tool for anyone to use freely, so we want to create a space where users can share their experiences and contribute to corpus development.

In a broader sense, not just limited to ReazonSpeech, I'm interested in those who are intrigued by human interaction and wish to create open-source products that benefit society. If you share these values, we'd love to have you join us.