Open-source TTS model support with timestamps

[alesstracker21]

First of all, I'd like to say this is a great project! I am looking for ways to integrate it into my project.
I have an open ended question here, It feels to me like this project heavily relies on cloud services, but I am running everything locally because I aim to have then a self contained service I can use myself and I find that no open source model (at least that I know of, i.e. Melo/Parler/Coqui and so on) support timestamps and might require converting the outputted phonemes by the TTS manually into timestamps.
Has anybody managed to make this run with open-source TTS models, while keeping timepoint data available for lipsync?

Thanks for being a part of the conversation!

[met4citizen]

I've been looking for such a TTS project for some time (with no luck), so thank you for starting this thread.

One potential candidate on my radar is Piper. I haven't had time to explore it in depth, but based on the documentation and demos, it has several qualities I like: it uses neural voices, it is fast, it is released under an MIT License, and the project seems active. There are even WASM versions available, so you could run it entirely in a browser. Additionally, related to the TalkingHead project, there appears to be an open PR for generating word-level alignment data.

(Edit: It seems that the referenced PR generates audio twice: once for the entire sentence and then again separately for each word. This approach is not optimal. The duration of each phoneme is already available from the first run, but this information is simply not exposed in the lower-level API. More about this here. It seems, however, that one of the maintainers is currently rewriting Piper due to some licensing concerns and new features, so for now it might be best to wait and see how the project evolves.)

[lberthod]

HI Mike,
I don't know how it's relevant and complex to add t2a like tts in talking head?

[met4citizen]

Hi @lberthod

Based on the API documentation, t2a can't provide word-level timestamps, which are required for accurate lip-syncing. In non-streaming mode, it can provide a link to a subtitles file, but the timestamps there are only at the sentence level.

One way to obtain word-level timestamps would be, of course, to extract them from the t2a audio output using a transcription service like whisper.cpp. However, that would complicate the process and increase latency.

I should point out that t2a is not an open-source TTS model but part of a commercial cloud service, so it isn't what Alessandro is looking for in this thread.

[met4citizen]

There is now a timestamped Kokoro TTS model onnx-community/Kokoro-82M-v1.0-ONNX-timestamped, which can be used to create a local, free text-to-speech service with timestamps.

As a proof of concept, I created a KokoroTTS JavaScript class ./examples/kokorotts.mjs that provides not only word-level timestamps but also visemes. It runs entirely in a browser using transformers.js and a simple phonemizer based on eSpeak NG. I also made a simple example app ./examples/kokoro.html, which you can run here.

Note that the KokoroTTS JavaScript class uses WebGPU, so it only runs on the latest versions of Chrome or Edge desktop browsers. It takes some time to load and consumes a significant amount of memory, but the latency is reasonable on my MacBook Air. Currently, only English voices are supported (British "en", American "en-us").

(Edit: Check my next comment 👇)

[met4citizen]

I just created a new HeadTTS GitHub project, a free JavaScript-based text-to-speech (TTS) solution with support for timestamps and visemes. It's built on the already mentioned Kokoro model and can run entirely in the browser (via WebGPU/WASM), or alternatively on a Node.js WebSocket/RESTful server (CPU-based).

The project is still a work-in-progress, but if you're using a desktop version of Chrome or Edge, check out the in-browser demo.