Exercise for the Reader Solved

Or rather, my conclusion about the answer to this question on the test (I posted the question to see if anyone else could figure it out) was confirmed by the teacher: there is no way to reduce the storage of any set of arbitrary points below the 96 bits required for the raw coordinates. There was an error in the question; specifically, the teacher forgot to mention one additional constraint about the values of the coordinates (though he hasn't said what it was, yet).

As for possible solutions. If it were known that all points were coplanar, you could store the normal of the plane and first point (or the first three points themselves), then convert all other points to 2D coordinates. You could use any manner of 2D coordinate system (they'll all use the same amount of space to store), but as we know the first three points are not collinear, I'd say the easiest would be barycentric coordinates. This was the first thing that came to mind while I was taking the test, but I asked and he said that I couldn't assume all points were coplanar.

Alternately, if you knew something about the locations of the points with regard to the plane, such as that they formed a regularly-spaced grid, or occurred periodically on a curve, you could reduce each point to a single value - the distance from the plane - with some overhead for describing the pattern the points appear on. Given that the hint the teacher gave me after I had finished the test was "height map", I'm guessing this is what he had in mind.