really nice idea. i like the “read along” concept a lot — being able to replay the same word or sentence multiple times is super helpful when learning a language.
am i right that the narration is generated with AI? sometimes there might be small pronunciation quirks, but overall the quality already sounds pretty good from what i tried.
one thing that would make this even more useful for learning (at least for me) is word-level explanations. for example clicking a word and seeing a simple definition in the same language (like german → german explanations in learner dictionaries), not just translation. that really helps build intuition.
hey HN!
this is my attempt at language learning with audiobooks. it synchronizes text to speech, and shows translations. the audiobooks themselves are all public domain.
got all of the audio alignment, translation, and asset generation working on my gaming computer. pretty happy with the pipeline, except for the sometimes subpar translations.
if anyone is interested in the details I am happy to write them up!
if you are into language learning, I would love to hear if this could be useful to you!
Looks great - Exactly what I’m looking for. Could we get different dialects? For Spanish, I would love to be able to select a country for the audio. At least a Latin American version to start. Thanks for sharing.
The pronunciation is different between countries. Not sure about other differences between dialects, but from what I remember from studying years ago, word pronunciation differences were quite predictable. There were a couple of online dictionaries with audio for Spain, Mexico and perhaps Argentina.
went through quite a few iterations of aligning text to speech.
found that ai transcription was really good most of the time but would hallucinate quite a bit towards the start and end of books. which I think might be related to those models being partially trained on audiobooks, and only having the book text itself, without any of the intro or credits.
in the end I landed on extracting text from ebooks, using rule based and language specific segmentation, and espeak based alignment. pretty basic, but it worked wonders in terms of reliability and accuracy.
if you are looking to generate audio from ebooks this is probably not too helpful. it is something I tried to avoid. something about learning a languages from generated audio didn't sit right with me haha.
really nice idea. i like the “read along” concept a lot — being able to replay the same word or sentence multiple times is super helpful when learning a language.
am i right that the narration is generated with AI? sometimes there might be small pronunciation quirks, but overall the quality already sounds pretty good from what i tried.
one thing that would make this even more useful for learning (at least for me) is word-level explanations. for example clicking a word and seeing a simple definition in the same language (like german → german explanations in learner dictionaries), not just translation. that really helps build intuition.
thanks! none of the audio is AI generated, it's all human narration, mostly sourced from librivox. (credits show when clicking the info icon)
the idea of learning languages from AI didn't quite sit right with me. but that might be something to circle back to.
integrating learner dictionaries does sound like a fantastic idea. will definitely explore that!
For those interested in doing this themselves, there’s some great existing software to create EPUB3 files with the audiobook aligned with text.
https://storyteller-platform.gitlab.io/storyteller/
https://codeberg.org/richwaters/StoryAlign
oh that looks amazing. can't believe I didn't find these during research. thanks for pointing them out!
hey HN! this is my attempt at language learning with audiobooks. it synchronizes text to speech, and shows translations. the audiobooks themselves are all public domain.
got all of the audio alignment, translation, and asset generation working on my gaming computer. pretty happy with the pipeline, except for the sometimes subpar translations.
if anyone is interested in the details I am happy to write them up!
if you are into language learning, I would love to hear if this could be useful to you!
Hi. Interesting idea.
Just a heads up: the text is not displayed on my Firefox (140.8.0esr 64bit, Win11) however. On Edge it is displayed correctly.
thanks for pointing that out. should have checked availability of view animation timelines.
will make sure it falls back to fully visible text.
https://developer.mozilla.org/en-US/docs/Web/CSS/Reference/P...
Looks great - Exactly what I’m looking for. Could we get different dialects? For Spanish, I would love to be able to select a country for the audio. At least a Latin American version to start. Thanks for sharing.
cool idea. haven't really explored dialects yet. gonna see if I can find any latin american recordings. thanks for the suggestion!
The pronunciation is different between countries. Not sure about other differences between dialects, but from what I remember from studying years ago, word pronunciation differences were quite predictable. There were a couple of online dictionaries with audio for Spain, Mexico and perhaps Argentina.
This is great! More beginner level stories would be much appreciated.
thanks, that's a really good point. having some beginner friendly books for each language is definitely a goal.
are you looking for stories in a specific language?
This is fantastic!
I've been meaning to learn Spanish, and this looks super useful.
Would love to learn more about your pipeline [selfishly, I was looking to build (free) ebooks -> audio for my own purposes as a side project]
What were the most challenging aspects? What assumptions failed / held true? Any experiences to share? Thx
glad to hear it!
went through quite a few iterations of aligning text to speech. found that ai transcription was really good most of the time but would hallucinate quite a bit towards the start and end of books. which I think might be related to those models being partially trained on audiobooks, and only having the book text itself, without any of the intro or credits.
in the end I landed on extracting text from ebooks, using rule based and language specific segmentation, and espeak based alignment. pretty basic, but it worked wonders in terms of reliability and accuracy.
if you are looking to generate audio from ebooks this is probably not too helpful. it is something I tried to avoid. something about learning a languages from generated audio didn't sit right with me haha.