Comments

You must log in or register to comment.

CodeAllDay1337 t1_j5xyiux wrote

I think this one is a good start:
Differentiation of Blackbox Combinatorial Solvers
https://arxiv.org/abs/1912.02175

3

K3tchM t1_j63l7xu wrote

I don't know which numerical optimization OP is trying to solve, but one major weakness of this paper is that their method requires two solver calls per instance per epoch... Training time might quickly become intractable.

OP should have a look at other methods that aim to solve their problem efficiently, such as https://arxiv.org/abs/2112.03609 or recently https://arxiv.org/abs/2203.16067

1

cruddybanana1102 t1_j601vly wrote

Someone has already mentioned Neural Ordinary Differential Equations, which is also the first thing that came to mind. There are also extensions to it, where one can use PDEs(Neural Hamiltonian Flows) or even stochastic DEs(Score-Based Generative Models) in the model. All of them covering different but overlapping use cases.

There are also techniques which use numerical solvers as blackboxes to perform model-order reduction of a complicated system of equations, or identifying slow modes, timescale decomposition, etc.

3

leviaker t1_j5y6jh3 wrote

I am making a solver in pytorch :p

1

arg_max t1_j60qav1 wrote

Typically, if you're solver is not written in Pytorch/tensorflow itself you can't easily calculate gradients through them as your computational graph doesn't capture the solver. If your soler is also written in the framework and differentiable you might be able to just backpropagate through it though. Otherwise, the Neural ODE paper that was linked here a few times has an adjoint formulation that gives you the gradient wrt to the solver as a solution to another ode, but this is specific to their problem and won't apply to non-differential equations.

1