It would help if you explain what exactly you want to transmit, the model, results, gradients,...? Btw, how would pruning not reduce the computational demand?

Yep- see Weight Fixing Networks https://arxiv.org/abs/2210.13554

Oh hey this is is a neat justification for this type of pruning, I did my masters on this topic! - The technique we came up with was called Dropback, basically resetting the weights to their initial values instead of zero, used in combination with a pseudo rng so you can reinit weights on the fly deterministically. You only had to track a few million parameters out of tens of millions to achieve similar accuracy. https://mlsys.org/Conferences/2019/doc/2019/135.pdf

you might have misunderstood the message. OP asked if there is work on reducing the number of weights without compromising network strength as it is not compute that is the bottleneck but transmitting the object.

"not reducing computational overhead" is not the same as not reducing performance

Recurrent Parameter Generators https://arxiv.org/abs/2107.07110 could be interesting for you. The idea is not to prune the architecture, but instead use a limited bank of parameters that generates the networks parameters

For example if you prune using the builtin unstructured pruning methods in pytorch it will not lead to faster inference as by default pytorch does not leverage the sparsity.

One of the most popular/studied methods of reduction of neural network weights is "Knowledge Distillation".

[deleted]

Maybe 'Weight Agnostic Neural Networks' will help

No because it makes more sense to start with a too large network and reduce compute to a reasonable level than keeping compute constant.

Might be a bit overkill, but, instead of weight reduction, have you considered changing the architecture completely? I can’t find it so someone maybe can link it below, but, I read an article on using reinforcement learning to decide on an architecture for a neural network. Theirs was designed to just be accurate, however, you could take a similar approach while also adding a parameter to minimize overall size of the network.

Like I said, not really pruning as much as it is about overall size reduction

This is just describing pruning though, the whole purpose of better pruning methods is reduce size without compromising performance on the intended task.

If you are embedding the weights of a model in an FPGA then the size of the FPGA is your bottleneck, it's unlikely to be your bandwidth talking to the ground because FPGA's just aren't that big relatively speaking.

Yes ground comms is a factor, but realistically A. How often are you going to be flashing new models onto your orbital systems relative to B. How much inference are you going to be doing with any one given model, and C. How big is the amount of data you'll then need to beam back down to collect those inferences.

Is the upload of the model weights really the dominant factor here?

By all means, strive to make the model as small as possible. But there's nothing special about the edge device being in orbit compared to it being on earth but hard to access.

hi, you are aware i am not the OP?

Yes. You stated what OP had meant, and I responded to you to say that what you (both) were describing was just pruning in general.

very cool, what do you mean by pseudo rng? Like picking random weight values for a previously established pool/range?

pseudo rngs produce deterministic results from a given seed, so aren't truly random. But have a statistical distribution matching true randomness.

https://en.wikipedia.org/wiki/Pseudorandom_number_generator

bitsandbytes LLM int8 you can quantize most weights in large models, and keep a small subset in full range, and get equivalent output. You could then also use a lookup table to further compress the weights.

Parameter sharing across layers would achieve just that. In the ALBERT paper the authors show that repeating a layer multiple times actually leads to similar performance than having separate parameter matrices. I havent heard a lot about this technique, but I assume this is because people mostly care about speed, which this does not improve (while it is a good match for your usecase)

As in, you care about compression for the sake of communication and not computation?

It sounds like you just need straight compression. Not neural network compression but like storage of numbers compression. That’s going to be 10x or 100x better than whatever pruning will do for you. There is research in the field of compressing models for storage or transmission.

These are exactly the questions one needs to ask before even starting. I have seen it numerous times that people are working on something that might be interesting, but utterly useless at the end of the day.

If I understood correctly, you want to train a model on the "earth based training system" and transmit the whole model to an FPGA on space for inference (usage).

The main paper I remember reading on this is https://arxiv.org/abs/1510.00149.
(they propose techniques, including quantization to reduce size while maintaining the same accuracy.)

In general, for "embedded applications" (or edge IA, low power, etc) it is very common to quantize ANNs (weight and activation-wise. Going from floating 32bits to 8bits or lower).
(It is so common that there is: https://www.tensorflow.org/lite.)
What particularly interests you is weights quantization (because that's the biggest part you'll transmit). So I'd recommend reading more on quantization.

Another thing to consider is the architecture itself, which determines how many parameters you have in the first place.
Particularly, convolutional layers use a small shared kernel for each convolution (e.g. 3x3 weights) but compute several dot matrix multiplications, whereas dense layers are essentially one dot matrix multiplication but with n_in*n_out weights.
(Keep in mind that compression is just a tangent to my research topic and that my main education was not in CS/ML, so I might be missing relevant topics)