Welp, it seemed like the problem was, that the inputs need to be defined as 2-dimensional with the sequence length as the first parameter. I thought one would give the RNN only 1 dimension of latent noise and get the sequence through reiterating it trough the RNN.