Skip to content

Compression

Compression is designed to support different types of compression algorithms with no or minimal impact on query performance. Compression is lossless.

Supported dataset types

Compression is available for

  • grids with single-value items,
  • meshes with single-value items, and with z-level items in SingleTimestep and MultiTimestep storage,

incl. layered items, in every storage type (SingleTimestep, MultiTimestep, TimeSeries - compressed as the uncompressed layout stores the values), except sigma-layer meshes.

Requesting compression for anything else is rejected; the predicate behind every check is CompressionExtensions.IsCompressionSupported, and a compressed dataset must carry the layout parameter of its kind (TimeSeriesBlockRows, CompressedMeshPageSize - CompressionExtensions.MissingLayoutParameter); its metadata and data are written only with the version of its layout (ItemStorage.LayoutVersion, the legacy grid SingleTimestep excepted - see "Compatibility").

Compression algorithms

All of them support every datatype (float, double, int, ...).

  • DeflateFast: the standard .NET deflate optimized for speed
  • DeflateOptimal: the standard .NET deflate optimized for size
  • LZ4: block compression optimized for speed; from layout version 2 (ItemStorageLayouts.Lz4ElementsOnly) a block encodes the elements of its tile, page or block only. The datasets of the earlier layouts - the LZ4 grids imported since 2020 among them - encode the whole tile buffer, the bytes left in it from the previous tile included (a partial edge tile of a small grid over 100 % of its raw size), and are read and written so for good: a dataset keeps the version it was created with
  • DeflateOptimalAuto: Deflate (Optimal) with an automatic per-tile transform (byte shuffle, 1-D/2-D delta, byte delta, float quantization detection): about a third smaller than DeflateOptimal on smooth integer data such as scaled Int16 satellite/ocean grids (20-48 % per sample), 40-60 % smaller on quantised GRIB floats (CDAS wind, GFS; CDAS ice cover, already at 3 %, unchanged), at most 0.2 percentage points worse elsewhere (sparse precipitation, tiny dfs2 grids); reads at 0.7-1.1x of DeflateOptimal (on par on Int16 and GRIB), writes at 0.6-1.1x of its throughput - the transforms are compared on every 32nd tile only (DeflateAutoCompressor.DefaultProbeStride) and the winner is kept for the tiles in between and for the following slices (AutoTransformState: per item and encode chunk for grid tiles, per item for mesh pages, per item, layer and spatial group for TimeSeries blocks)
  • ZstdAuto (the intended setting for grids, meshes and the TimeSeries storage, see "Default compression"): the same automatic per-tile transform with Zstandard (level 7, RFC 8878 frames) as the backend codec - class averages within 0.4 percentage points of DeflateOptimalAuto on grid tiles (single samples -0.5 to +1.0; better on floats, 0.3 worse on Int16 and dfs2), up to 2 points worse on the production mesh samples (4-5 on tiny meshes of a few hundred elements), and reads 1.0-1.7x faster (most on sparse floats). Level 7 was measured against level 9 on the TimeSeries samples: class averages within 0.2 percentage points, single samples 0 to +0.5 (with the rows-per-block rule below), at 1.2-1.8x the write speed; the level is a write-side choice only, data written at any level reads unchanged. Like DeflateOptimalAuto it is a new block format: every deployment that reads MD data must understand it before a dataset is written with it.
  • ZstdFastAuto: ZstdAuto written at zstd level 3. The block format and the read speed are identical to ZstdAuto - but the name is stored in the dataset metadata, so a build has to know the name to read the dataset (see "Compatibility" below). Measured against level 7: on the TimeSeries samples -0.1 to +1.9 percentage points of ratio for 1.2-5x faster writes (2.5-5x on the production mesh samples), on grid tiles and mesh pages 0 to +2.9 points for 2.3-4x. Meant for ingest-speed-critical conversions. Levels 5 and 6 measured worse than level 3 on some datasets (zstd non-monotonicity, and the probe competition picks a different transform; level 4 was measured on four mesh samples only) - there is nothing useful between 3 and 7.

Default compression

A new dataset gets the default compression of its kind from the environment (MDWriterOptions), each set through an environment variable forwarded to the transfer jobs via TransferOptions:

kind option environment variable
grids stored per timestep (SingleTimestep, MultiTimestep) DefaultGridCompression MD_DEFAULT_GRID_COMPRESSION
meshes stored per timestep DefaultMeshCompression MD_DEFAULT_MESH_COMPRESSION
TimeSeries storage of grids and meshes DefaultTimeSeriesCompression MD_DEFAULT_TIMESERIES_COMPRESSION

The code default of each is None, so a freshly deployed build never starts writing a block format on its own; the intended setting is ZstdAuto for all three, one value everywhere, set per environment in the argo TransferOptions of the metadata service. A separate variable per kind exists so that each kind can be switched on once the build that reads it runs everywhere (see "Compatibility"). An older build ignores a variable it does not know. The default is applied only when the writer parameter Compression is not given; pass "Compression": "None" to store a dataset uncompressed. Compression never changes the storage layout: the writer picks the storage type by size (SingleTimestep when at most one timestep fits the 10 MB blob the writer aims at - above 655 360 elements at the 8 B per element it assumes - MultiTimestep with the computed TimeGroupSize otherwise, at most 49 999 timesteps per group as Azure allows 50 000 blocks per blob and a compressed group spends one on its page index; a TimeGroupSize parameter of 1 or less means SingleTimestep, the StorageType parameter overrides all of it) and compresses the blobs of that layout. Before this change a grid or mesh of 655 361-1 310 720 elements was stored as MultiTimestep with a TimeGroupSize of 1; it is SingleTimestep now; a grid of at most 655 360 elements written with a Compression parameter or the grid default in force was SingleTimestep (compression forced that layout; a mesh could not be compressed) and is MultiTimestep now, and MultiTimestep storage does not support removing timesteps. The default of a kind now applies with an explicit TimeGroupSize too (it used to back off to None); a TimeGroupSize outside 1-49 999 with MultiTimestep and a GridTileWidth outside 16-4096 are refused, with or without compression (a tile of a few cells is mostly codec frame, and a tile of one cell made the query loop forever; a stored grid keeps the width it was written with, the reader accepts any width whose tile an int counts, 1-46 340: the LZ4 grids imported before the bound may be wider), and an explicit MultiTimestep of a grid or mesh above 1 310 720 elements gets a TimeGroupSize of 1 (the computed 0 used to fail the write). Existing datasets keep the settings they were created with (append/update follows the dataset, not the default). DataSizeBytes of a compressed dataset is listed from the storage once after an import and after every append or update, not taken from the writer: a job killed and run again, or a second job of the same append, finds the groups the other run committed stored and would count them as nothing. Tiles of one grid slice are encoded by up to 4 workers in parallel (CompressedGridIndex.MaxEncodeWorkers, one per available CPU); MD_ENCODE_WORKERS (forwarded via TransferOptions.EncodeWorkers) sets the count explicitly up to that maximum, 1 makes the encoding serial. A slice of 16 or more tiles is split into 4 fixed chunks (a smaller one is one chunk) with a probe state per item and chunk whatever the worker count, so the encoded bytes never depend on the CPU count or on timing: a job run again, or twice at once, writes the same blocks for the same slices.

Per-timestep storage (SingleTimestep, MultiTimestep)

A compressed timestep is its pages compressed back to back: for a grid the pages of CompressedGridIndex (GridTileWidth, default 128 x 128 cells, in the stored quad-tree order), for a mesh runs of ItemStorage.CompressedMeshPageSize elements of the spatial index order (16 384, persisted when the dataset is created; a compressed mesh without it is refused). A spatial query decodes only the pages it touches, each once whatever the number of query ranges in it (the pages of one timestep in one ranged request, split where they lie more than 60 KB apart or the request would exceed 8 MB), a whole-slice read all of them. Within a grid page the query walks the quad tree like the uncompressed GridIndex: blocks the geometry misses are skipped, blocks a polygon covers are taken whole (a Polygon only, as in GridIndex: a multi-polygon may be invalid, its parts overlapping, and Covers throws on it), only the cells on the border of the geometry are tested one by one. Blocks inside a page and the blocks above the pages are tested with the envelope of their own cells, so a geometry that only touches the last cell of a block finds the cell on grids whose coordinates are not exact in binary (the uncompressed GridIndex tests the envelope the grid computes for a block, which can end short of it; the whole-page shortcut keeps that envelope, it can only take cells the geometry intersects). An all-no-data page is stored with length 0, except the first page of a timestep that is no-data everywhere: that one is stored encoded, in both storage types and for z-level items, because an Azure block (MultiTimestep) cannot be empty. A z-level item of a mesh is one blob per timestep in both storage types (.dat + .idx): its stored values (per mesh index page the nodes x layers block, ItemStorage.ZLevelPagedNodeCount values in total) cut into the same pages. A z query maps the byte range of every mesh index page it hit onto the compressed pages the range covers, decodes each once and takes the layers out of the rebuilt ranges as the raw read does. SingleTimestep: one .dat per timestep, item and layer with the page lengths next to it as .idx (as before for grids). MultiTimestep: the group blob holds its timesteps back to back - one Azure block per timestep, as the uncompressed layout commits them - followed by one more block, the group's page index: the TimeStepIndex of every block in block-list order (timestep t of the group is slot t % TimeGroupSize, page p of slot s is entry s * pages + p), the same bytes an .idx holds for a SingleTimestep, followed by its length and a magic number (TimestepPages.IndexBlock). A reader needs no block list: the blob properties (length, ETag) and one ranged read of the last 4 KB give it the index (a longer index costs one more ranged read), the pages come with the next request pinned to the same ETag; a whole-group read takes the blob and splits the index off its tail. An index whose lengths do not add up to the bytes before it is refused (a damaged blob, a copy of parts), so is any page index with a negative length or a page length no codec writes (TimeStepIndex, TimestepPages.CheckPages), before it sizes a download. A byte copy of the blob therefore stays readable. The writer takes the stored index from the block list (IBlobStorage.GetBlockListAsync: names and sizes of the committed blocks, the ETag and length of the blob) and refuses a blob that has content but no committed blocks (copied without its block list, e.g. with Put Blob - the relocation of a dataset uses Copy Blob, which keeps them). The block ids carry the timestep and a hash of the block (TS0000000005-<hash>, IDX000000000-<hash>, both 29 characters - Azure takes one id length per blob), so a block list names the exact content it commits. A write stages its blocks, stages the new index block (the stored lengths it keeps come from the stored index block) and commits one block list of the kept blocks, its own and the index: there is nothing between the two commit points an .idx next to the blob used to have, so a job killed anywhere leaves the group as it was - another commit takes one of its staged blocks only under the same id, i.e. with the same bytes - and Azure drops them with the next commit of the blob or after seven days. A re-import of a timestep replaces its block in place (same slot, new id, new length). Everything a write builds on comes from one version of the group: the stored index is read with the version the block list came with (If-Match, RangeStream.Version) and the block list is committed with it too (If-None-Match: * for a new group), so a commit of another job in between fails the write with BlobChangedException (run it again, the group holds the other job's commit) - the same when that commit dropped the staged blocks of this one. A commit whose response was lost and that the Azure SDK sent again meets the version it created itself (412, or 409 for a new group): the writer then reads the block list and takes a list equal to its own as its commit. A query reads the index and the pages with the version of the properties it started from; a commit in between makes it read the group again once from the new version, a second one fails the query with BlobChangedException - a read over many timesteps of a group (one pinned request each) fails while appends commit faster than it reads. Groups written by the builds of this change that kept the index in a separate .idx blob are refused with a message to import the dataset again. SingleTimestep and z-level items keep .dat + .idx: the index describes the whole blob. A query that meets a re-import of the timestep can get one of the two from before the upload and one from after it: it reads the pair again (TimestepPages.ReadPairAsync, after 200 ms and once more after 1 s: the writer stores the .idx a round trip after the .dat) when the lengths do not add up or a page does not decode (InvalidDataException), and when an index longer than the new .dat gets its ranges short or refused by Azure (EndOfStreamException, HTTP 416). A pair that still does not add up (a re-import killed between the two uploads, two re-imports of the timestep at once) is refused (TimestepPages.CheckLength) until the re-import is run again. The pair is matched by its total length only: an index paired with a re-imported .dat of the same length passes; the block header of the auto codecs refuses such a page in most cases, DeflateFast, DeflateOptimal and LZ4 data can decode wrongly without an error, as before. The local FileBlobStorage of the CLI stages blocks as files and keeps the block list in a .blocks sidecar whose first line is the version of the commit; both are written as .commit files and moved into place, the blob first. A sidecar left in .commit next to the moved blob (a commit killed between the moves, or a reader held the sidecar open) is the list as long as its sizes add up to the blob: readers take it without moving anything, the next commit puts it in place before it writes its own; one next to a blob still in .commit is a failed commit and is ignored. A commit while a reader holds the sidecar open succeeds and leaves its list pending; the next commit fails while the reader still holds it (IOException, or UnauthorizedAccessException on Windows) before it changed anything - run it again. A whole blob (.dat, .idx, a TimeSeries group, dataset.json) is written next to the blob as a .write file and moved into place, so a write killed half way leaves the blob as it was, as an upload to Azure does. Staged blocks, sidecars, .commit and .write files are no blobs - listings and folder sizes leave them out.

TimeSeries storage

Datasets stored as TimeSeries (grids with single-value items, meshes except sigma-layer meshes with single-value items, incl. layered items, as the uncompressed layout stores them; a z-level item is refused, the layout keeps one row per mesh element and a z-level item holds a value per paged node) are compressed per blob of one spatial group x one time group: the blob holds blocks of TimeSeriesBlockRows whole series (rows), each block one compression block (with the auto codecs the 2-D predictor runs along time and across neighbouring series), and the block lengths are the blob's header (one int32 per block, TimeSeriesBlocks.HeaderBytes; the reader knows the block count from the row count). The group is one blob, so the rewrite of a fixed-step group on append is one atomic upload - a job killed during it leaves the old group, never a torn one - and a blob that does not hold what its header describes is refused on the read (the blob length comes with the first ranged download) and on the append. The block download carries the ETag of the header download (If-Match, RangeStream.PinVersion): a query that meets a rewrite of the group reads it again once from the new version (BlobChangedException). Uncompressed reads do not pin the version - an append keeps their offsets. A series query does one ranged read of the header and one of the blocks it needs (split where they lie more than 60 KB apart or the request would exceed 8 MB). A block never cuts the time axis, so a series query decodes one block and takes the series straight out of the codec buffer; a query reuses its codecs for all the groups it reads, one per stored type and group length for each group read running at once (a codec holds its block buffers and a decompression context - a point series over a year is hundreds of group reads). Sigma-layer meshes are not compressed: the writer and the reader derive the block boundaries and the header from the row count (SigmaLayersInfo.Element3DCount), which is one short until the sigma TimeSeries fix of WI 147930 is in. TimeSeriesBlockRows is chosen when the dataset is created and persisted: the smallest row count whose raw block (rows x TimeGroupSize x the largest element size of the items, grids and meshes alike) reaches 32 KB, clamped to 4-256 and to the spatial group size. An all-no-data block is stored with length 0. Appending or replacing a fixed-step time group decodes the stored group, patches it and encodes it again - the same whole-blob rewrite the uncompressed layout does, plus the codec: 7-15 s of CPU per GB of raw group data with ZstdAuto, 3-7 s with ZstdFastAuto (production cuts, one thread per spatial group; about half of that while the group is half filled). A dataset appended to often is a case for "Compression": "ZstdFastAuto". A replace of variable-step groups overwrites the group blobs, compressed or not, it removes nothing first: a replace killed half way leaves the stored groups readable and can be run again. The default for new TimeSeries datasets is MD_DEFAULT_TIMESERIES_COMPRESSION (TransferOptions.DefaultTimeSeriesCompression), code default None, intended setting ZstdAuto; the grid default does not apply to this layout, it is a separate setting only so that the TimeSeries compression can be switched on after the build that reads it is rolled out.

Compatibility

The compression is stored by name in the dataset metadata (ItemStorage.Compression) and read by every deployment that loads MD datasets: the multidimensional service, the multidimensional cron jobs (data cleanup) and the transfer jobs (append, update, mesh update, relocation, export). A build that does not know a name loads the dataset with CompressionType.Unknown (CompressionTypeConverter, in the builds since PR 146364, develop 9 September 2026; the builds before it fail to load such a dataset): the dataset stays visible and its metadata is left untouched, only reading or writing its data fails with a message saying the compression is not known to this build. Writing such a dataset's metadata back is refused, so an older build can never replace the name it does not know. A newer build's writer accepts a new name as soon as it is deployed - through the environment default or an explicit writer parameter - so the order per environment is: deploy the build to all readers, then enable the compression.

The second compatibility boundary is the kind of dataset (see "Supported dataset types"): a compressed dataset of a kind this build does not read (a sigma-layer mesh, multi-value grid items - as a newer build may write) loads, but every data read or write is refused by DatasetStorage with a message saying this build cannot read it; without that guard the storage classes for those kinds would misread the compressed blobs. The rollout of such an extension is the same as for a new name: deploy the build to all readers first, then enable the compression for that kind. The builds with this guard but before the per-timestep compression of meshes and MultiTimestep refuse those datasets the same way. The builds before the guard (before PR 146364, which no master build holds yet) do not know the auto codecs and fail to load such datasets at all; a mesh, MultiTimestep or TimeSeries dataset compressed with LZ4 or a Deflate* name they would misread.

Every dataset created since the compressed TimeSeries, MultiTimestep and mesh layouts carries ItemStorage.LayoutVersion (ItemStorageLayouts.Current; append and update keep the stored ItemStorage); datasets created before have none. Version 1 (Compression) brought the compressed TimeSeries, MultiTimestep and mesh layouts, version 2 (Lz4ElementsOnly) changes the LZ4 blocks only (see "Compression algorithms"). The version tells a legacy flag from compressed data: the writer of the time set Compression in its SingleTimestep branch only and honoured it on grids only, so a mesh imported with an explicit compression holds raw blobs under a compression flag. That is the one legacy case: a flag without the version on a mesh stored as SingleTimestep loads as uncompressed (DatasetStorage.LoadAsync, CompressionExtensions.IsLegacyCompressionFlag) - unless the first timestep of the first item has a page index (.idx of layer -1 or 0, one or two existence checks when such a dataset loads): the raw mesh never had one, so that is a compressed mesh whose version an older build dropped, it keeps its flag. A flag without the version on grid SingleTimestep is the layout compressed back then and is honoured. A flag without the version on MultiTimestep or TimeSeries is compressed data whose version a build before the versioning dropped when it re-saved the metadata; the dataset keeps its flag and is refused for the missing layout parameter rather than read raw (a grid MultiTimestep has none and stays readable). DatasetStorage.StoreMetadataAsync, the path every writer saves through, refuses a compressed layout without the version, so a dropped version is never written back as the truth, and the data writes (PrepareWrite, GetDataWriter, RemoveTimestepAsync) and StoreSpatialDomainAsync (a mesh update) refuse it before they touch a blob; the relocation copies the blobs first, the copy left behind by its refused metadata save goes with the cleanup job. DatasetStorage refuses every data read and write, metadata save and spatial domain store of a dataset whose LayoutVersion is newer than ItemStorageLayouts.Current - a build that reads a newer layout as the one it knows would return garbage and append its own layout into it, and a re-save of its metadata would drop what it does not know of the layout. The version lives in ItemStorage, not in SchemaVersion: the latter describes the mesh index and MDMeshUpdateWriter rewrites it on a mesh update without touching the item data, while ItemStorage is copied as it is by every writer of this build that re-saves the metadata.

A compressed TimeSeries dataset also stores ItemStorage.TimeSeriesBlockRows, a compressed mesh stored per timestep ItemStorage.CompressedMeshPageSize. A build that knows the compression name but not the parameters (or the version) - one before PR 146364 for an LZ4 or Deflate* name - loads such a dataset without them and drops them when it writes the metadata back (item update, append, update, dataset copy, mesh update); this build then refuses the dataset's data (except a grid MultiTimestep, which needs no layout parameter) and its metadata. The feature therefore rolls out to every reader and writer (multidimensional service, its cron jobs, the transfer image) at once, and a rollback of one of them must not write MD metadata.

Extending compression to another kind of dataset

  1. Implement the compressed read and write path of the storage class for that kind (as SingleTimestepStorage and CompressedGridIndex do for grids) and relax CompressionExtensions.IsCompressionSupported for it - that single predicate gates the writer validation, DatasetStorage.StoreMetadataAsync and the read-side refusal above. A new layout of a kind already compressed is a new ItemStorageLayouts value and Current: the builds with the layout version guard before it refuse the datasets.
  2. Keep the refusal test (DatasetStorageTests.CompressedDatasetOfAKindThisBuildCannotReadStillLoadsButRefusesItsData) for the kinds that remain unsupported and add a round-trip test for the new one.
  3. The defaults are per kind (grid, mesh, TimeSeries - MDWriter.GetCompression) and apply to every dataset the predicate admits, so relaxing the predicate switches the new kind on in every environment where the default of its kind is set. Deploy the build to every reader before the writer runs it, or keep the new kind out of the default in GetCompression until then.

Adding a compression type

  1. Add the value to CompressionType (ItemStorageOptions.cs) at the end - the name is the persisted format identifier, keep it stable (legacy integer JSON is read by ordinal).
  2. Implement ICompressor (or derive from DeflateAutoCompressor for a new backend codec, as ZstdAutoCompressor does) and add the branch to CompressionExtensions.CreateCompressor. A compression built on DeflateAutoCompressor must also be listed in CompressionExtensions.IsAutoTransform, otherwise grids and meshes do not get the per-item probe state for it (the TimeSeries blocks always get one). CompressionTypeTests.EverySelectableCompressionResolvesToACompressor fails until both are done.
  3. Tests: round trip (CompressorRoundTripTests), the garbage patterns (CompressorStressTests), CreateCompressor resolves the type, and a pinned block format if the type must stay readable by an existing reader.
  4. Describe it above and in conversion.dedicated.writers.import.json (writer parameter description).
  5. Roll out as described under "Compatibility"; do not make it the default in the same release. A write-side variant of an existing format (a different level) is still a new name, i.e. a new compatibility boundary - prefer a separate option (e.g. a level) that older builds ignore over a new enum value.

Enabling compression in conversion

Compression is enabled by adding writer parameter Compression with the name of Compression algorithm (see above). Example: 

 "writerParameters": [
    {
      "name": "Compression",
      "value": "ZstdAuto"
    }
  ]

Specific datasets

HYCOM, COPERNICUS

NetCDF variables packed as scaled Int16 (Copernicus, older HYCOM imports) are stored as Int16 - the production census also holds HYCOM GLBy and Mercator stored as Double; ZstdAuto (byte shuffle + delta on Int16), the intended environment default - the grid default, or the TimeSeries default for datasets stored as TimeSeries - is the right choice; the LZ4 of the older imports stays readable (and selectable) but has no reason to be chosen any more.

NOAA

NOAA is highly (lossy with JPEG2000) compressed format. Storing it as uncompressed doubles is not optimal. Float precision is sufficient. NOAA Reader adds reader parameter DoubleAsFloat, which forces converting and storing double value as float.

"readerParameters": [
    {
      "name": "DoubleAsFloat",
      "value": true
    }
  ]