1 Comment

This is really interesting - and reminded me of a recent podcast from the Santa Fe-folks, where Krakauer noted that we have a tendency to gravitate towards one or two dimensional models.

What you are describing, in a way, is the reduction of the dimensionality of a problem to a single dimension, where one selected variable varies with another and can be expressed simply in a visual chart. And then - as you point out - when one of the dimensions reduced changes, well then we are stuck. In many senses, I think of model compression as a general case of image compression, and the loss rate is the challenge. The asymmetry is the cost -- and reconstructing a highly compressed piece of music or an image requires a lot of work if it is at all possible.

Another question is if there is a way for us to compress models and then provide a way to unpack the compression (a bit like file compression) or if all compression to be useful implies the loss of dimensionality.

Expand full comment