Alright, I'll try to explain ...

My goal is to convert 3d scenes to bas-reliefs. The steps for this are:

1. Compute the height and normal maps of a 3d scene, as seen from a specific point of view.
2. Compress the heights in such a way that the overall height is reduced, while the shapes and details of the original 3d scene remain intact.

The first step is simple. The second step is a very difficult problem. There exists no method to do this in an optimal way. There are many approaches to formulate it as linear or quadratic optimization problems.

I'm currently solving it by formulating an optimization problem (which is basically the loss function L = G(x, y')), reformulating it into a poisson-like equation and then solving it using a multigrid method. This is a very standard way to compute such bas-reliefs from height or normal maps. The problems with this approach are:

a) It is relatively slow to transform a single image (1024x1024 takes 1-2 minutes). Neural networks could have the potential to be much faster. My goal is to be as close to real-time as possible. b) It is too restrictive because I have to formulate my loss function in such a way that it results in a linear system. This negatively affects the quality of the final result.

It is possible to compute how well a converted bas-relief y' = F(x) represents the original height map x, which leads to the loss function. As an example, you could compute the similarity of the normal vectors of x and y' (which will already result in a non-linear optimization problem). Similar normal vectors = low loss; different normal vectors = high loss. This type of non-linear optimization problem is what I want to solve, which is what lead me to the idea to solve this problem using a CNN.