There are Reversible Neural Networks where this is possible to do, they're used for things such as for Normalizing Flows and even for very large NNs that don't fit in memory, since if the outputs are reversible you don't need to save intermediate activations during the forward pass, you can just recalculate them in the backward pass.