Release notes
v1.2 (unreleased)
Default output needs PTWM v1.2 or later
The default outputs changed format. Readers before v1.2 reject them
with an error; they do not misread them. The Python Decompressor
reports "not a .ptwm container". The Rust reader and the _rust
container functions report "bad magic" for a \x89PTWC container and
"missing EOF sentinel (file may be truncated)" for a \x89PTWS frame,
although the frame is intact.
| Output | v1.1 | v1.2 default |
|---|---|---|
Compressor.compress (one tensor) | legacy container, \x89PTWM | compact frame, \x89PTWS |
ptwm compress model.safetensors (one blob per tensor in a safetensors file) | legacy container per tensor, \x89PTWM | compact frame per tensor, \x89PTWS |
multi-tensor writers, other ptwm compress outputs | legacy container, \x89PTWM | compact container, \x89PTWC |
The multi-tensor writers are compress_ptwm_blob, write_ptwm_shard,
compress_safetensors_file, the LMDB and WebDataset writers,
_rust.compress_model and the Rust compress_model with
CompressorOptions::default(). The other ptwm compress outputs are
--mode a, --mode b, the LMDB and WebDataset stores,
--file_compression, --delta and .bin inputs. A tensor that does
not fit the compact frame is written as a \x89PTWC container.
v1.2 reads every format: \x89PTWM, \x89PTWX, \x89PTWC and
\x89PTWS.
Writing files for v1.1 readers
Write a full container in the legacy layout:
- Python, one tensor:
CompressionConfig(framing=Framing.CONTAINER, container_layout=ContainerLayout.LEGACY). - Python, many tensors:
container_layout="legacy"(orContainerLayout.LEGACY) on the writers listed above. - CLI:
ptwm compress --container-layout legacy, with any--mode. For a.safetensorsfile without--mode, this also writes every tensor as a full\x89PTWMcontainer instead of a\x89PTWSframe. - Rust:
CompressorOptions { container_layout: ContainerLayout::Legacy, .. }, orContainerWriter::new, which always writes the legacy layout.
A legacy file also records the ids of the built-in codecs and
operations it uses, and each id is derived from a release version. A
v1.1 reader recognises only the ids of version 1.1.0, so legacy
writers stamp the ids of 1.1.0 (LEGACY_BUILTIN_VERSION) whatever
their own crate version. v1.2 readers accept the built-in ids of any
release.
Smaller output
Non-payload bytes (everything except coded data), measured with
python -m benchmarks.container_overhead:
| Case | legacy layout | v1.2 default |
|---|---|---|
one 4 KB fp32 tensor, Compressor.compress | 2,867 B (\x89PTWM) | 50 B (\x89PTWS) |
one 4 KB fp32 tensor, Framing.CONTAINER | 2,867 B (\x89PTWM) | 134 B (\x89PTWC) |
| 64 small fp32 tensors, per tensor | 262 B (\x89PTWM) | 83 B (\x89PTWC) |
Most of the legacy bytes for one tensor are Extension Table entries for all 16 built-in codecs, whether the file uses them or not. Legacy containers written by v1.2 keep that v1.1.0 table: the v1.1 transcoder rebuilds it when it reads an exploded LMDB or WebDataset store, and keeps each plane's codec index from the manifest, so a shorter table would make v1.1 decode those stores with the wrong codecs.
Other changes
- Writers reject duplicate tensor names and more than 32,768 chains, instead of writing files whose extra entries no reader can reach.
- Readers check every count and length before they allocate, size each plane from the chain before decoding it, and reject a chain whose Source declares far more bytes than the tensor holds.
Format.BYTEinput: the chain's Source now declares exactly the raw byte count. v1.1 could declare up to 8 times more for some dtypes; v1.2 still decodes those files.ptwm._rust.ContainerHandleopens a container once for many queries and borrowsbytesand read-onlymmapbuffers without a copy.TensorIndex.names()returns tensors in write order.- The
ptwm._corecodec, delta and hash functions,compress_modelandcompress_compactread their input with the GIL released. They now use the same rule asContainerHandle:bytesand read-onlymmapbuffers are borrowed without a copy, and any other buffer (bytearray, writablememoryview, numpy array) is copied first. In v1.1 another thread could change abytearraywhile it was read. - The transcoder (
explode_ptwm/implode_ptwm) writes a"ptwm-manifest-c"version 2 manifest for compact containers. The legacy"ptwm-manifest"version 1 manifest keeps its v1.1 form and gains two optional fields,extension_table_b64andbuiltin_version, andimplode_ptwmrejects manifests with unknown formats, versions or fields. v1.1 ignores the optional fields, so it rebuilds a manifest correctly only when they are absent. Legacy files that v1.2 writes produce no such fields; legacy files with another Extension Table or built-in version may. ptwm compress X --delta Rnames its outputX.delta.R.ptwmwhenXdoes not end in.bin, andptwm decompressrestoresXfrom it. v1.1 cut the last 4 characters ofXand always restored a.binname, somodel.safetensorscame back asmodel.safeten.binand short names such asa.ptandb.ptwere written to the same file. A.bininput keeps the v1.1 name<X without .bin>_delta_R.ptwm.ptwm decompressreads the input name from a name that holds exactly one.delta.or_delta_, so such v1.1 files decompress to the same names as before. A name with more than one separator (usually because the input or reference name holds one) does not show where the input name stops, and the reference name does not settle it, because a renamed reference can match the end of the name. v1.1 split such a name at its first_delta_, which restored some files to the right name and others to a wrong one. Nowptwm compressprints a note with the input path, andptwm decompressstops with exit status 1 instead of guessing; give the output file with the new--outputoption.ptwm decompressalso checks--outputbefore it decompresses, refuses--outputwith a directory input, and exits with status 1 when the delta input or reference file is missing (v1.1 exited 0 without a message). The decompress overwrite prompt now checks the file it writes.- The compact single-tensor frame (
compress_compact) encodes each plane with the row layout of the chain's Source shape, the layout its reader uses. It took the row length from the tensor shape before, so a caller-supplied chain whose Source shape had another last dimension than the tensor could selectOrder1ScaleACwith the wrong row length and write a frame that decoded to wrong bytes. Frames built with the chains of theptwmwriters do not change. Containers store each plane's layout and were not affected.
The byte layouts are in
docs/architecture/wire-format.md.