Viewing a single comment thread. View all comments

BornInMappleSyrop t1_iu21yh2 wrote

How much space on a hard drive would a picture like that take. It must be insane!

19

herrbdog t1_iu28d6e wrote

9.6gb, raw uncompressed, they won't want to compress the original

27

Just_Another_Scott t1_iu2jklj wrote

Their website says about 15 terabytes a night. So take that for what you will.

> Rubin Observatory R&D has led to a new-generation imaging CCD which is highly segmented, low noise, and sensitive from the UV to the near IR. The speed with which Rubin Observatory can cover half the sky will produce about 15 terabytes (TB) per night, leading to a total database over the ten years of operations of order 50 petabytes (PB) for the raw data, and 15 PB for the catalog database. The total data volume after processing will be over 100 PB

So total data requirements exceeds 100 PB over a 10 year period.

https://www.lsst.org/about/dm/technology

25

TheRealMrChips t1_iu2qkvr wrote

Which seems like a lot, but with 30TB HDDs coming in 2023 and even larger capacities in the next couple years, 100PB won't be all that hard to make happen over 10 years' time, even with redundant storage and backups taking it to 300-400PB of "actual" storage footprint.

13

bsloss t1_iu4ddam wrote

Back of the napkin math says it would only take 5 to 10 racks of something like this to cover the storage requirements. Not something that comes cheap, but certainly doable and still below the industrial scale of massive data centers run by most big tech companies.

5

cp5184 t1_iu3y234 wrote

There's lossless compression. For something like this they might want to develop a specialized compression format.

6

herrbdog t1_iu4bs72 wrote

of course there is. PNGs are great! JPGs ate my bane (i'm a graphic artist)

but now i wonder if the data-per-pixel will be ONLY 3 channel (RGB), 1 byte, or if it will have more than 3 channels, and will it have more range per pixel (2 bytes gives 16,384 levels of grey per channel instead of only 256... much more nuance and detail.

i should probably look it up instead of speculating lol

3

rocketsocks t1_iu5ax4l wrote

It's 1 channel per image not RGB, this camera doesn't have a mosaic filter on the sensor itself the way consumer digital cameras do. Like almost all scientific imagers it instead is monochromatic but uses filters that can be rotated into place through an automated mechanism. This provides higher resolution for each color channel while also allowing for adjustments to the exposure timing for each channel depending on how much light it passes (which is much more desirable from a scientific standpoint). This particular camera will have 5 wideband color filters covering the visible through near-infrared bands but it won't have an exact match of red, green, and blue color channels.

So an exposure of a particular patch of the sky in all color channels will actually look like 5 successive exposures (or 6 if there is an unfiltered pass) through each of the filters.

It will have 18 bits per pixel of dynamic range.

2

herrbdog t1_iu5hxtd wrote

well then the files would be smaller... only 18 bits per pixel? well that's not that much actually (only 262,144 possible color values), but is probably sufficient for the data acquired. idk, i'm not an astronomer :(

so 6 channels, but only 3 bits per channel?

1

rocketsocks t1_iu5js7y wrote

Every exposure is 18 bits per pixel, each channel is a separate exposure. So a full image of one patch of the sky would be either 5 or 6 individual exposures for each color channel (plus a clear exposure) which would be equivalent to 90 or 108 bits per pixel.

There isn't a 1:1 match of filter channels to RGB colors but for just the 3 color channels closest to RGB that would be the equivalent of 54 bits per pixel, or 18 quadrillion colors.

2

herrbdog t1_iu5n0l0 wrote

GOT IT, thanks!!! (yeah i realise it doesn't convert directly to RGB then)

intense detail then.

1

bsloss t1_iu4ecs6 wrote

Total hard drive cost is likely around a million bucks. If the specialized compression format can save half of that it’s no small chunk of change, but I can also imagine that high development costs could make even significant storage savings not worth the hassle of doing something custom and worrying about potential bugs.

1