Python API
The public Python API lives in the top-level ptwm package. The main
entry points are Compressor, Decompressor, and their config
dataclasses.
Compress and decompress raw bytes
from ptwm import (
Compressor, Decompressor,
CompressionConfig, DecompressionConfig,
Method, Format,
)
config = CompressionConfig(
method=Method.AUTO, # AUTO | HUFFMAN | MICROSCALE | ZSTD
bytearray_dtype="bfloat16", # hint for byte-reordering raw bytes
threads=8, # encode parallelism (default: all cores)
)
compressor = Compressor(config)
compressed: bytes = compressor.compress(data, fmt=Format.BYTE)
decompressor = Decompressor(DecompressionConfig(threads=8))
restored: bytes = decompressor.decompress(compressed, fmt=Format.BYTE)
The AUTO method picks a codec by dtype: HUFFMAN for float / int
weights, MICROSCALE for MXFP4 / NVFP4 paired tensors, ZSTD as a
generic fallback (plain in-tree Zstd; the byte-split preprocessing op
upstream supplies the SHUFFLE-style reordering).
Compress and decompress a torch tensor
import torch
from ptwm import Compressor, Decompressor, CompressionConfig, Format
config = CompressionConfig(input_format=Format.TORCH)
compressor, decompressor = Compressor(config), Decompressor()
tensor = torch.randn(1024, 1024, dtype=torch.bfloat16)
restored = decompressor.decompress(compressor.compress(tensor))
assert torch.equal(tensor, restored)
Supported input/output formats
| Format | Description |
|---|---|
Format.BYTE | Raw bytes in / raw bytes out. dtype hint required for byte-reordering. |
Format.TORCH | torch.Tensor in / torch.Tensor out. dtype and shape preserved. |
Format.NUMPY | numpy.ndarray equivalent. |
Format.FILE | File-path workflow. Input path is read; output path is written. |
CompressionConfig knobs
| Option | Default | Description |
|---|---|---|
method | AUTO | AUTO, HUFFMAN, MICROSCALE, or ZSTD |
bytearray_dtype | "bfloat16" | Dtype hint for byte-reordering when input is BYTE |
threads | None (up to 16 cores) | Maximum threads for parallel encode |
compression_threshold | 0.95 | Skip compression if ratio exceeds this threshold |
compression_chunk | 262144 | Chunk size in bytes (256 KB) |
framing | Framing.COMPACT | COMPACT writes a compact single-tensor frame (\x89PTWS, about 20 to 70 B of fixed overhead; raw bytes stored when no codec helps) and falls back to a full container for tensors that need shared codec state or chunking. CONTAINER always writes a full .ptwm container, which the random-access, store and transcode tools require. |
container_layout | ContainerLayout.COMPACT | Layout of a full container (CONTAINER, or the COMPACT fallback). COMPACT writes \x89PTWC (about 85 to 120 B for one bf16 tensor, 125 to 140 B for one fp32 tensor). LEGACY writes \x89PTWM (about 2.5 to 2.9 KB for one tensor, most of it the v1.1 table of all 16 built-in codecs), the only layout readers before v1.2 open. Strings "compact" and "legacy" are accepted. |
emit_payload_hash | True | Store an xxh64 of the raw bytes (8 B per tensor), checked on decode |
emit_plane_crc | True | Store a CRC32 per coded plane (4 B per plane), checked on decode. The compact frame stores plane CRCs only when emit_payload_hash is off, since the hash already covers every plane |
Files for readers before v1.2
The default outputs, the compact frame and the compact container, need PTWM v1.2 or later to decode. Older readers reject them with an error; they do not misread them. For files that v1.1 opens, write a full container in the legacy layout:
from ptwm import CompressionConfig, Compressor, ContainerLayout, Framing
legacy = Compressor(CompressionConfig(
framing=Framing.CONTAINER,
container_layout=ContainerLayout.LEGACY,
))
The multi-tensor writers take the same choice as a keyword,
container_layout="legacy". See the
release notes for the built-in id caveat.
Full reference
See the Python API reference for every public symbol
in ptwm.* with docstrings, parameters, return types, and source
links.