[D] Lion , An Optimizer That Outperforms Adam - Symbolic Discovery of Optimization Algorithms Submitted by ExponentialCookie t3_1138jpp on February 15, 2023 at 9:06 PM in MachineLearning 27 comments 43
CyberDainz t1_j8weqb8 wrote on February 17, 2023 at 12:58 PM so, technically this is binary optimizer that updates the weight to either -1 or +1 multiplied by lr. Should be tested with "Learning Rate Dropout", i.e. 30% chance to update with -1/+1, otherwise no update. Permalink 2
Viewing a single comment thread. View all comments