Would love to go through the workshop

[JohannesTK]

The notebook is missing the pre-trained models: https://github.com/respeecher/vae_workshop/blob/master/latent_codes.ipynb

Do you plan to upload them because I would love to go through the workshop?

Thanks!

[belevtsoff]

@JohannesTK unfortunately, we can't publish pretrained models, because we've used proprietary data to train them. However, it is really easy to train one yourself - just grab one of the librispeech books (with good quality) and follow the instructions in README.md. I've tried to make sure the process is as painless as possible. Let me know how it went

[JohannesTK]

Thanks for the fast answer!

Tried it out with librispeech dev clean where is a total of 323 mins of data. Split it 90% train, 10% test. Didn't get good results.

The VAE training loss exploded:

Epoch: 264, step: 157462,  train_loss: 2217.78, test_loss: 664.88
Epoch: 264, step: 157463,  train_loss: 23646600.00, test_loss: 664.88
Epoch: 264, step: 157464,  train_loss: 444613920.00, test_loss: 664.88
Epoch: 264, step: 157465,  train_loss: 1917470591747771858944.00, test_loss: 664.88
Epoch: 264, step: 157466,  train_loss: 1347883333031297024.00, test_loss: 664.88
Epoch: 264, step: 157467,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157468,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157469,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157470,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157471,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157472,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157473,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157474,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157475,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157476,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157477,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157478,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157479,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157480,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157481,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157482,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157483,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157484,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157485,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157486,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157487,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157488,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157489,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157490,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157491,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157492,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157493,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157494,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157495,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157496,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157497,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157498,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157499,  train_loss: nan, test_loss: 664.88
Epoch: 264, step: 157500,  train_loss: nan, test_loss: nan
Epoch: 264, step: 157501,  train_loss: nan, test_loss: nan

The mel spectrogram inverter looks better:

test_mel_pred: https://instaud.io/3OsU

Seems like the VAE model needs more training steps? But the loss explodes. What are your thoughts?

[belevtsoff]

Hm, yeah, the loss looks really unstable and it should definitely sound better. The reason might be that we trained the model on a single speaker whereas the dataset you've used contains a whole bunch of them. Let me take a look at it a little bit later. In the meantime I'd suggest trying a single clean speaker (e.g. chopping up one good book from librivox) and look for anomalies in the tensorboard images.