Viewing a single comment thread. View all comments

LahmacunBear OP t1_j3l5ub2 wrote

Reply to comment by resented_ape in [R] Learning Learning-Rates: SteDy Optimizer by LahmacunBear

Oh damn, that paper almost does exactly what I do. Huh. Oh well. Slightly different implementation though. I in contrast, use both grads from the same timestep and have an accumulated Ct.

5

SatoshiNotMe t1_j3lhi10 wrote

Are either of these open source and easily usable as a PyTorch optimizer ?

1

LahmacunBear OP t1_j3mrexi wrote

Mine’s in Tensorflow 2.11 — I’m sure writing a PyTorch version wouldn’t be hard. The extra lines of the algorithm are three lines in my paper. I can share my code though?

1

SatoshiNotMe t1_j3n5p3v wrote

Thanks! I was just curious for future reference. I’ll need to first read the papers to see if it can help with my projects.

2