https://arxiv.org/abs/2302.01560

Using language for long term planning in Minecraft.

Your argument falls apart when you realize that there are training artifacts. Ever wonder why FID scales inversely with model size?

Just because it can imitate doesn't mean it can do so perfectly.

Taking a leaf out of RL, you can add an additional entropy loss.

Alternatively, clip the logits but apply STE (copy gradients) on backprop

From my personal interactions it just gave off this vibe that it was trained on websites, rather than the GPT-3 (both base and chat) models which felt much more natural. Something to do with having to learn too many languages?

2060. 3 times the tensor cores and 1.5x the bandwidth. 6 and 8gb really ain't much.

There is indeed a non-zero gradient. However, symmetric initialization introduces a plethora of problems:

1. The only way to break the symmetry is through the random biases. A fully symmetric network effectively means that individual layers act as a though they are a single weight(1 input 1 output layer), this means that it cannot learn complex functions until the symmetry is broken. Learning will thus be highly delayed as it has to first break the symmetry before being able to learn a useful function. This can explain the plateau at the start.
2. Similar weights at the start, even if symmetry is broken, will lead to poor performance. It is easy to get trapped in local minima if your outputs are constrained due to your weights not having sufficient variance, there is a reason why weights are typically randomly initalized
3. Random weights also allow for more "learning pathways" to be established, by pure chance alone, a certain combination of weights will be slightly more correct than others. The network can then abuse this to speed up it's learning, by changing it's other weights to support these pathways. Symmetric weights do not possess such an advantage.

>Bad initialization is rarely a problem

What if all weights are the same?

If you are a masochist then yes

I really can't give a solid answer, but 600 is definetly too few (a few thousand is already on the low end).

Try dreambooth. 600 is far too little for training a stylegan from scratch (heavy augmentation could help but I doubt it)

There is currently no easy way of pooling vram. If the model can't fit onto vram I suggest you check out https://huggingface.co/transformers/v4.9.2/parallelism.html#tensor-parallelism.

Just going to put this and this here

^(this is for non nefarious purposes right)

Kinda frustrating how half of the help posts here are requrests for laptops. Like have they not bothered to do even the tiniest bit of research?? At the same price you could get an equivalently speced desktop/server AND an additional laptop, with the added bonus of being able to run long training sessions without needing to disrupt it.