PTWM
Concepts

Container format

Multi-tensor container with random-access lookup, hash-verified payloads, and per-plane codec dispatch.

The .ptwm file is a multi-tensor container. A single file holds many tensors with a shared Extension Table, a chain registry, a tensor index, and per-plane codec dispatch.

PTWM writes the container in one of two layouts:

LayoutMagicDefaultReaders
Compact (PTWC)\x89PTWC\r\n\x1a\nyes, since v1.2v1.2 and later
Legacy (PTWM)\x89PTWM\r\n\x1a\nnov1.2 and later; v1.1 with a caveat

Both hold the same content and decode the same way. The compact layout stores built-in codecs and operations by wire id, uses varints, and keeps tensor names in the index. For 64 small fp32 tensors it costs about 83 bytes per tensor besides the coded data, against 262 bytes for the legacy layout. Choose the legacy layout only for readers older than v1.2 (see Compatibility).

Single-tensor output from Compressor.compress uses a third, smaller format by default, the compact frame (\x89PTWS). See Single-tensor frame.

Layout

graph TD
    subgraph Container [".ptwm container"]
        H[Header<br/>magic, section lengths, checksum]
        E[Extension Table<br/>ops and codecs the file uses]
        P[Prelude<br/>shared codec state]
        C[Chain registry<br/>preprocessing chains]
        subgraph Body [Tensor records, in write order]
            T1[Tensor 1: dtype, shape, hash,<br/>plane records + payloads]
            TN[Tensor N: ...]
        end
        I[Tensor index<br/>name → record]
        F[EOF sentinel]
    end

    H --> E --> P --> C --> Body --> I --> F

Each tensor record holds its chain reference, dtype, shape and payload hash, then one plane record per output of the chain. A plane record names the codec of the plane, its codec state, an optional CRC32 and chunk table, and is followed by the plane's payload bytes.

The byte-level layout of both layouts, and of the single-tensor frame, is in docs/architecture/wire-format.md.

Random access

The tensor index maps each name to its record. Opening a .ptwm and fetching one tensor touches only:

  1. The header, Extension Table, prelude, chain registry and index. For a built-in-only file of 64 tensors these take about 2 KB in the compact layout, most of it the names in the index.
  2. The record of that tensor: about 50 bytes for an fp32 tensor in the compact layout, 190 bytes in the legacy layout, plus its payloads.

Use TensorIndex to do this from Python. It opens either layout and lists tensors in write order.

Hash verification

Each tensor record can carry an xxh64 of the raw tensor bytes (emit_payload_hash, 8 bytes per tensor), and each plane record a CRC32 of the decoded plane (emit_plane_crc, 4 bytes per plane). The decoder checks both before it returns bytes to the caller. A mismatch raises ptwm.Error.

Single-tensor frame

Compressor.compress writes a compact frame (\x89PTWS) by default: one tensor with only the fields its decoder needs, at a fixed cost of about 20 to 70 bytes. A tensor that needs shared codec state or chunked planes falls back to a full container in the configured layout. Framing.CONTAINER always writes a full container, which the random-access, store and transcode tools need.

Compatibility

Readers before v1.2 reject the compact container and the compact frame with a "not a .ptwm container" or "bad magic" error; they do not misread them. To write files that v1.1 opens, select the legacy layout and the full container:

from ptwm import CompressionConfig, ContainerLayout, Framing

config = CompressionConfig(
    framing=Framing.CONTAINER,
    container_layout=ContainerLayout.LEGACY,
)

The multi-tensor writers (compress_ptwm_blob, write_ptwm_shard, compress_safetensors_file, the LMDB and WebDataset writers) take container_layout="legacy", and the CLI takes ptwm compress --container-layout legacy for every output, including the per-tensor .ptwm.safetensors file. Legacy writers record the built-in ids of version 1.1.0, which v1.1 readers recognise; see the release notes for the details.

Modes for safetensors input

The compress_safetensors_file entry point writes the compressed payload in one of two shapes; the mode argument selects which:

ModeOutputUse when
"a" (native)A .ptwm directory with one .ptwm blob per shardYou control the loader and want the cleanest random-access surface.
"b" (shell)A .safetensors shell wrapping a .ptwm blobYou need the result to look like a regular .safetensors file to existing tooling.

Both modes round-trip through PTWM and both take container_layout. Mode a is the default.