I mean practically speaking it doesn’t seem to achieve much more than that, but I don’t think that’s the point of the paper. The point here is that it’s actually rewriting the source code itself each time, which is potentially useful because it can (theoretically) achieve something more novel than just changing hyper parameters.

It would be more interesting if they showed actually nontrivial code changes for sure, if those are even possible. But I don’t think it’s entirely useless; it’s possible, for example, that we may be able to use something similar to deprecate the transformer eventually, in the not so near future.