PTWM

Release notes

Changes between PTWM releases that affect file compatibility.

v1.2 (unreleased)

Default output needs PTWM v1.2 or later

The default outputs changed format. Readers before v1.2 reject them with an error; they do not misread them. The Python Decompressor reports "not a .ptwm container". The Rust reader and the _rust container functions report "bad magic" for a \x89PTWC container and "missing EOF sentinel (file may be truncated)" for a \x89PTWS frame, although the frame is intact.

Outputv1.1v1.2 default
Compressor.compress (one tensor)legacy container, \x89PTWMcompact frame, \x89PTWS
ptwm compress model.safetensors (one blob per tensor in a safetensors file)legacy container per tensor, \x89PTWMcompact frame per tensor, \x89PTWS
multi-tensor writers, other ptwm compress outputslegacy container, \x89PTWMcompact container, \x89PTWC

The multi-tensor writers are compress_ptwm_blob, write_ptwm_shard, compress_safetensors_file, the LMDB and WebDataset writers, _rust.compress_model and the Rust compress_model with CompressorOptions::default(). The other ptwm compress outputs are --mode a, --mode b, the LMDB and WebDataset stores, --file_compression, --delta and .bin inputs. A tensor that does not fit the compact frame is written as a \x89PTWC container.

v1.2 reads every format: \x89PTWM, \x89PTWX, \x89PTWC and \x89PTWS.

Writing files for v1.1 readers

Write a full container in the legacy layout:

  • Python, one tensor: CompressionConfig(framing=Framing.CONTAINER, container_layout=ContainerLayout.LEGACY).
  • Python, many tensors: container_layout="legacy" (or ContainerLayout.LEGACY) on the writers listed above.
  • CLI: ptwm compress --container-layout legacy, with any --mode. For a .safetensors file without --mode, this also writes every tensor as a full \x89PTWM container instead of a \x89PTWS frame.
  • Rust: CompressorOptions { container_layout: ContainerLayout::Legacy, .. }, or ContainerWriter::new, which always writes the legacy layout.

A legacy file also records the ids of the built-in codecs and operations it uses, and each id is derived from a release version. A v1.1 reader recognises only the ids of version 1.1.0, so legacy writers stamp the ids of 1.1.0 (LEGACY_BUILTIN_VERSION) whatever their own crate version. v1.2 readers accept the built-in ids of any release.

Smaller output

Non-payload bytes (everything except coded data), measured with python -m benchmarks.container_overhead:

Caselegacy layoutv1.2 default
one 4 KB fp32 tensor, Compressor.compress2,867 B (\x89PTWM)50 B (\x89PTWS)
one 4 KB fp32 tensor, Framing.CONTAINER2,867 B (\x89PTWM)134 B (\x89PTWC)
64 small fp32 tensors, per tensor262 B (\x89PTWM)83 B (\x89PTWC)

Most of the legacy bytes for one tensor are Extension Table entries for all 16 built-in codecs, whether the file uses them or not. Legacy containers written by v1.2 keep that v1.1.0 table: the v1.1 transcoder rebuilds it when it reads an exploded LMDB or WebDataset store, and keeps each plane's codec index from the manifest, so a shorter table would make v1.1 decode those stores with the wrong codecs.

Other changes

  • Writers reject duplicate tensor names and more than 32,768 chains, instead of writing files whose extra entries no reader can reach.
  • Readers check every count and length before they allocate, size each plane from the chain before decoding it, and reject a chain whose Source declares far more bytes than the tensor holds.
  • Format.BYTE input: the chain's Source now declares exactly the raw byte count. v1.1 could declare up to 8 times more for some dtypes; v1.2 still decodes those files.
  • ptwm._rust.ContainerHandle opens a container once for many queries and borrows bytes and read-only mmap buffers without a copy. TensorIndex.names() returns tensors in write order.
  • The ptwm._core codec, delta and hash functions, compress_model and compress_compact read their input with the GIL released. They now use the same rule as ContainerHandle: bytes and read-only mmap buffers are borrowed without a copy, and any other buffer (bytearray, writable memoryview, numpy array) is copied first. In v1.1 another thread could change a bytearray while it was read.
  • The transcoder (explode_ptwm / implode_ptwm) writes a "ptwm-manifest-c" version 2 manifest for compact containers. The legacy "ptwm-manifest" version 1 manifest keeps its v1.1 form and gains two optional fields, extension_table_b64 and builtin_version, and implode_ptwm rejects manifests with unknown formats, versions or fields. v1.1 ignores the optional fields, so it rebuilds a manifest correctly only when they are absent. Legacy files that v1.2 writes produce no such fields; legacy files with another Extension Table or built-in version may.
  • ptwm compress X --delta R names its output X.delta.R.ptwm when X does not end in .bin, and ptwm decompress restores X from it. v1.1 cut the last 4 characters of X and always restored a .bin name, so model.safetensors came back as model.safeten.bin and short names such as a.pt and b.pt were written to the same file. A .bin input keeps the v1.1 name <X without .bin>_delta_R.ptwm. ptwm decompress reads the input name from a name that holds exactly one .delta. or _delta_, so such v1.1 files decompress to the same names as before. A name with more than one separator (usually because the input or reference name holds one) does not show where the input name stops, and the reference name does not settle it, because a renamed reference can match the end of the name. v1.1 split such a name at its first _delta_, which restored some files to the right name and others to a wrong one. Now ptwm compress prints a note with the input path, and ptwm decompress stops with exit status 1 instead of guessing; give the output file with the new --output option. ptwm decompress also checks --output before it decompresses, refuses --output with a directory input, and exits with status 1 when the delta input or reference file is missing (v1.1 exited 0 without a message). The decompress overwrite prompt now checks the file it writes.
  • The compact single-tensor frame (compress_compact) encodes each plane with the row layout of the chain's Source shape, the layout its reader uses. It took the row length from the tensor shape before, so a caller-supplied chain whose Source shape had another last dimension than the tensor could select Order1ScaleAC with the wrong row length and write a frame that decoded to wrong bytes. Frames built with the chains of the ptwm writers do not change. Containers store each plane's layout and were not affected.

The byte layouts are in docs/architecture/wire-format.md.