2

I have a large text corpus of about ~7M characters that I am training an LSTM network on. However I am consistently seeing that after about the 5th epoch, instead of the generated sentences improving they become completely junk. I have pasted an example below:

Generating with seed: "n who is in possession of the biggest ro"
n who is in possession of the biggest ro to   tato ton ant an to  o  
ona toon t o o taon  an s to  ao t tan t  tout att tj ton  an o  t an $

I have tried with different temperatures as well. The example pasted above was the most conservative. Here's another generation:

Generating with seed: 'to sin, and most of the known world is n'
to sin, and most of the known world is na ararea t tarre a araa arae 
tor tae a a aaa aaata ater tje aea arare ar araererrt tmaear araae

To debug, I ended up copy pasting the LSTM example from keras and trained it on my corpus. Again, around iteration 5, it begins to generate junk.

Any ideas on how to debug this or what this might be a sign off? It starts off with much more coherent predictions but suddenly falls off.

1 Answer 1

1

Usually, 5 epochs is often too early to jump to conclusions. RNN can generate text that looks less like garbage, though it still hasn't learned anything. So you should start worrying if the sampled text obviously degrades with each iteration, otherwise just let it learn.

When this happens, the first thing you should check is how the probability distribution evolves with time. For example, this sample

a ararea t tarre a araa arae 
tor tae a a aaa aaata ater tje aea arare ar araererrt tmaear araae

... clearly indicates that a probability is too high. I don't think there's a lot of aa repetition in your training data, so the subsequence aaata should be very improbable. To check this, print the max probability score, along with the character it corresponds to (the distribution summary will do as well). If you see this probability raising, that's almost certainly the problem.

There could be various reasons for this, starting with data processing or input pipeline bugs and ending with incorrect network wiring. For instance, one particular bug that I saw was related to index 0, which corresponded to an unknown char or a padding, but was actually seen by LSTM as a valid character. Since there were lots of them, the network learned just this, hence its probability rose.

1
  • thanks for the reply. So I actually let it train for over 21+ hours, ~100 epochs, but there was no improvement in it's output. I will try checking the probability score and see what it yields. This was the output from the last iteration with highest temp: to kill every single rick an olnnorse ouneaoua nesnlnu otr tet oe snosneneti o o ter ouolsoretn ouoaae uu lree
    – shekit
    Dec 22, 2017 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.