PTWM
Guides

Python API

Use the Compressor and Decompressor classes directly from Python.

The public Python API lives in the top-level ptwm package. The main entry points are Compressor, Decompressor, and their config dataclasses.

Compress and decompress raw bytes

from ptwm import (
    Compressor, Decompressor,
    CompressionConfig, DecompressionConfig,
    Method, Format,
)

config = CompressionConfig(
    method=Method.AUTO,       # AUTO | HUFFMAN | MICROSCALE | ZSTD
    bytearray_dtype="bfloat16",   # hint for byte-reordering raw bytes
    threads=8,                    # encode parallelism (default: all cores)
)
compressor = Compressor(config)
compressed: bytes = compressor.compress(data, fmt=Format.BYTE)

decompressor = Decompressor(DecompressionConfig(threads=8))
restored: bytes = decompressor.decompress(compressed, fmt=Format.BYTE)

The AUTO method picks a codec by dtype: HUFFMAN for float / int weights, MICROSCALE for MXFP4 / NVFP4 paired tensors, ZSTD as a generic fallback (plain in-tree Zstd; the byte-split preprocessing op upstream supplies the SHUFFLE-style reordering).

Compress and decompress a torch tensor

import torch
from ptwm import Compressor, Decompressor, CompressionConfig, Format

config = CompressionConfig(input_format=Format.TORCH)
compressor, decompressor = Compressor(config), Decompressor()

tensor = torch.randn(1024, 1024, dtype=torch.bfloat16)
restored = decompressor.decompress(compressor.compress(tensor))
assert torch.equal(tensor, restored)

Supported input/output formats

FormatDescription
Format.BYTERaw bytes in / raw bytes out. dtype hint required for byte-reordering.
Format.TORCHtorch.Tensor in / torch.Tensor out. dtype and shape preserved.
Format.NUMPYnumpy.ndarray equivalent.
Format.FILEFile-path workflow. Input path is read; output path is written.

CompressionConfig knobs

OptionDefaultDescription
methodAUTOAUTO, HUFFMAN, MICROSCALE, or ZSTD
bytearray_dtype"bfloat16"Dtype hint for byte-reordering when input is BYTE
threadsNone (up to 16 cores)Maximum threads for parallel encode
compression_threshold0.95Skip compression if ratio exceeds this threshold
compression_chunk262144Chunk size in bytes (256 KB)
framingFraming.COMPACTCOMPACT writes a compact single-tensor frame (\x89PTWS, about 20 to 70 B of fixed overhead; raw bytes stored when no codec helps) and falls back to a full container for tensors that need shared codec state or chunking. CONTAINER always writes a full .ptwm container, which the random-access, store and transcode tools require.
container_layoutContainerLayout.COMPACTLayout of a full container (CONTAINER, or the COMPACT fallback). COMPACT writes \x89PTWC (about 85 to 120 B for one bf16 tensor, 125 to 140 B for one fp32 tensor). LEGACY writes \x89PTWM (about 2.5 to 2.9 KB for one tensor, most of it the v1.1 table of all 16 built-in codecs), the only layout readers before v1.2 open. Strings "compact" and "legacy" are accepted.
emit_payload_hashTrueStore an xxh64 of the raw bytes (8 B per tensor), checked on decode
emit_plane_crcTrueStore a CRC32 per coded plane (4 B per plane), checked on decode. The compact frame stores plane CRCs only when emit_payload_hash is off, since the hash already covers every plane

Files for readers before v1.2

The default outputs, the compact frame and the compact container, need PTWM v1.2 or later to decode. Older readers reject them with an error; they do not misread them. For files that v1.1 opens, write a full container in the legacy layout:

from ptwm import CompressionConfig, Compressor, ContainerLayout, Framing

legacy = Compressor(CompressionConfig(
    framing=Framing.CONTAINER,
    container_layout=ContainerLayout.LEGACY,
))

The multi-tensor writers take the same choice as a keyword, container_layout="legacy". See the release notes for the built-in id caveat.

Full reference

See the Python API reference for every public symbol in ptwm.* with docstrings, parameters, return types, and source links.