f5-tts

[kunibald413]

you might have already seen it https://github.com/SWivid/F5-TTS

zero-shot results are quite good, have any take on this? @e-c-k-e-r

[e-c-k-e-r]

I gave it a cursory glance the other day; my initial impressions is that it's decently comparable to this repo's output for the voice sample I grabbed and tested. My only ick right now is that, like some other solutions (Bark, I think some implementation(s) of VALL-E X), it requires transcribing the input audio prompt. While I can't complain if it works, it feels like a limitation, but I suppose it's necessary for "flow matching" models, since those are more primed for speech editing where TTS is a byproduct (at least given Meta's Voicebox paper).

As far as the dataset, I'm a bit puzzled? While I want to say another good job to Emilia for it being a decent dataset, 100K hours of audio is extremely steep when typical models seem to go about the ~50K hours LibriSpeech has. I don't know if the onus is on needing it or it just helped to have that much extra audio, especially since SESD boasts <1K hours, and my VALL-E got by with <20K hours of unique audio (I don't have the numbers of total audio fed over the various training sessions off the top of my head). I suppose the paper might explain.

When I get a chance I'll take a deeper look through both the architecture and outputs. I'm curious about things like:

• parameter size
  • 22+4 layers compared to VALL-E's 12, but half the FFN size (practically twice as deep but partially half as wide).
• audio/waveform representation
  • practically raw mels + Vocos' vocoder, which I suppose should have been a given since it's a DiT (and not some cope with quantized tokens in an arch that strives from continuous signals).
• extracting features (if applicable)
• natural-ness of a speaker (natural pauses, breaths, prosidy, etc.)
• retaining the "acoustics" of the input audio prompt (reverb, etc.)
• adhering to the prompt's prosidy (although I think my VALL-E has regressed in this department ironically)

But for sure I'll compare it against my VALL-E in the demo page whenever I get the chance.

[e-c-k-e-r]

Color me impressed; while it performs to my expectations, it does perform pretty well. I'll do what I can to try and express my thoughts:

• both "acoustics" and audio quality of the prompt are retained pretty well.
  • Kingdom Hearts clips retain the reverb-ish effect, GLaDOS retains her effects (both for normal and the high-pass filtered ones), and Half-Life 1's HEV and train ride retain their respective effects too.
  • not a negative, but one property I did notice, however, is that some aren't retained between utterances (I believe my VALL-E sometimes behaves liket his as well, but I only remember noticing it during a poorly baked LoRA).
• prosidy is mostly retained, the "normal" speakers with "normal" prosidies seem fine.
  • Pauling (TF2) surprised me the most with retaining her quick speech AND voice filter.
  • Half-Life 1's G-Man let me down, where it doesn't really try to capture his mannerisms.
  • Shadow-characters of Persona 4 (tried with Adachi at least) interprets the under-voice as a reverb, sadly.
    • As a test, even Ameno-sagiri (since a voice is under Adachi's clips) also is interpreted pretty poorly, but I didn't expect it to perform well anyways.
  • Igor (Persona 3/4) just sounds like Ikutsuki (Persona 3) (which is funny since they share the same VA).
  • Elizabeth (Persona 3) sounds a bit higher pitched, and doesn't quite have the eccentricities of her prosidy.
  • Aigis (Persona 3) doesn't retain her almost-robotic prosidy either.
  • Sonic Adventure 2 characters sound fine (at least Sonic and Shadow).
  • Sam (Sam and Max) seems to speak a little too fast.
  • SHODAN (System Shock 2) falls apart completely, as expected. It doesn't even try to make speech. Again, as expected.
• outputs seem decently natural, where there's inhales where it seems to be needed, albeit I feel when there are breaths, it's really eager to insert them for batches.
• in terms of adhering to the prompt, it does leave a bit to be desired. There doesn't seem to be much (useful) variance between outputs for different input clips. The only thing I did notice is that if there's an utterance with more silence, the output does follow that and have the output slower / with more dead-air. The tone doesn't seem to follow as strong as I would have liked.
  • I suppose this will always be my personal crux with TTS solutions, since it signals that a model is naively trained without using similar utterances as the input prompt (although I don't think I've seen a paper that explicitly mentions doing so; I know TorToiSe's DLAS does have a mechanism to do so, but nothing seems to signal that it's used for TorToiSe itself).
• inference speed is pretty quick, a batch of ten Harvard sentences takes about 40 seconds for an "average" lengthed input prompt.
• outputs seem almost consistently stable.
  • sometimes the beginning of a clip will have signal noise and sometimes eats up a word or two.
  • some utterances does have a bit of signal noise in some portions of the output, or throughout an output. I imagine this is just a crux of using mels directly, but I'd definitely prefer a bit of signal noise over artifacts/crust from inaccuracies in the higher RVQ levels for EnCodec codes.
• having the model transcribe the input prompts doesn't seem to be too much of a boon since it's pretty fast.
• Having longer / shorter input audio prompts doesn't seem to affect the quality of the output (unlike my VALL-E, where it benefits from longer input prompts (at least within what it was trained against)).

For what it's worth, it's a good model and gets my approval, and I feel a bit better knowing that my VALL-E is comparable to a SOTA model (ignoring the hiccups my VALL-E has sometimes).

• I will say though, that doing comparisons with my VALL-E makes me think that I did have a regression in terms of trying to train for longer input prompt durations; having 3-second prompts makes it fall apart pretty hard, when it seemed fine before me training with that in mind, sadly. I suppose I'll need to do some post-training to undo the damage.

[e-c-k-e-r]

As an addendum from doing a cross-eval on my VALL-E:

• F5's audio quality is consistently fine and clear, naturally (at least as far as 24KHz goes). Raw mels go a long way, and EnCodec's RVQs require a strongly confident NAR.
• There's this property with my VALL-E when it has confidence issues with some speakers where the AR will have "le wacky" prosidy (probably best explained as it sounding like each word being stitched together from different utterances) and the NAR will output a noticeable amount of artifacting/crust, as I noticed with Sonic (SA2). I haven't noticed this with F5, but I imagine this is a mix between training on 100K hours (since I feel like the confidence issue appeared less the more speakers my model has seen) and the "VALL-E Continuous" approach of having the input audio prompt's transcription as part of the input text prompt (since it should better guide the output response this way, rather than my VALL-E putting a ton of work through the input prompt).
• My VALL-E seems to fall apart with multiple sentences per utterance, even short sentences. F5 handles it perfectly fine since it auto-batched my Harvard sentence tests together to have multiple sentences per utterance. I imagine, again, the "VALL-E Continuous" approach of <input text><output text><input audio><output audio> helps a ton with multiple sentences since it naturally does contain at least two sentences. I think cfg.dataset.max_resps > 1 to combine sentences during training will go a long way, rather than relying on longer durations to possibly contain multiple sentences (which I doubt, they're probably just one long sentence).
• A source of my "regressions" seems to be the AR requires a lower temperature now? I've found an AR temp. of 0.85 or 0.9 has much more stable audio, which is strange, since 1.0 seemed fine for a long while. I suppose I need to extensively validate the default settings.

I definitely commend F5 for being a good model without really many quirks. I swear every time I do any eval on my VALL-E, it's some different quirk I have to attest to, and F5 felt quirkless for the most part.

[kunibald413]

Thank you for sharing your thoughts in detail. Appreciate it!

[e-c-k-e-r]

No problem at all, it gives me an opportunity to put my observations somewhere (since I keep putting off making the wiki page documentations), and I've been tending to get some good work done in the process of doing so.

I should have F5-TTS as a comparison in the demo page soon-ish rather than later-ish, but:

• there seems to be some consistency problems when using inference-cli.py for F5-TTS; it feels like the input prompt is leaking into the output, but the beginning isn't in the prompt, so I suppose I need to use the web UI to do so since it gave me great output before.
• I want to get satisfactory outputs from my VALL-E to put in the demo page again, so I'm having to nail down the regressions that are probably in my head (from weird quirks like float16 vs bfloat16, the attention backend I'm using, settings, etc.).

---

Demo page should be available here with F5-TTS samples: https://vall-e-demo.ecker.tech/
It's strange that the web UI seems to have better outputs than with inference-cli.py, but I've definitely had similar problems before in the past.

My VALL-E's outputs are a little more consistent with greedy sampling (+ some cope tricks), but it seems even just naive ar_temp=1.0 has some better outputs, but it seems much better than the last time I tried to generate the main demo page. I'll see about updating the LoRA demo page sooner-ish rather than later with the "fixed" sampler settings (although realistically I think it's just from updating PyTorch + using ROCm flash attention + float16).