PTWM
Python API

ptwm

PTWM — lossless compression for PyTorch model weights.

ptwm

PTWM — lossless compression for PyTorch model weights.

pymodule

ptwm

python/ptwm/__init__.py:1
module ptwm

PTWM — lossless compression for PyTorch model weights.

A typed preprocessing graph (bit reorder, byte/nibble split) feeds two peer entropy coders — Huffman and rANS — alongside a Zstd plane codec and a dedicated microscaling-format codec. The on-disk format is a multi-tensor ``.ptwm`` container with per-tensor random-access lookup. Public entry points: :class:`Compressor`, :class:`Decompressor`, :class:`CompressionConfig`, :class:`DecompressionConfig`, :class:`Method`, :class:`Format`, :class:`Framing`, :class:`ContainerLayout`. Third-party integrations (HuggingFace `transformers`, `safetensors`) live in :mod:`ptwm.integrations` and are imported from there explicitly, not re-exported here.
pyclass

ptwm.CompressionConfig

python/ptwm/_config.py:138
class CompressionConfig:
pymethod

ptwm.CompressionConfig.__init__

python/ptwm/_config.py:1
def __init__(self, method: Method = Method.AUTO, input_format: Format = Format.BYTE, bytearray_dtype: str = 'bfloat16', is_monotonic: int = 0, threads: int | None = None, compression_threshold: float = 0.95, check_th_after_percent: int = 10, reorder_signbit: int = 0, delta_compressed_type: str | None = None, lossy_compressed_type: Lossy = Lossy.NONE, lossy_compressed_factor: int = 27, compression_chunk: int = 256 * 1024, is_streaming: bool = False, streaming_chunk: int = 1024 * 1024, input_file: str | None = None, compressed_file: str | None = None, decompressed_file: str | None = None, zstd_level: int = 3, lz4_compression_level: int = 0, codec_menu: list[CodecId] | None = None, codec: str | None = None, device: int | None = None, framing: Framing = Framing.COMPACT, container_layout: ContainerLayout = ContainerLayout.COMPACT, emit_payload_hash: bool = True, emit_plane_crc: bool = True) -> None
pyattribute

ptwm.CompressionConfig.bytearray_dtype

python/ptwm/_config.py:142
bytearray_dtype: str = 'bfloat16'
pyattribute

ptwm.CompressionConfig.check_th_after_percent

python/ptwm/_config.py:146
check_th_after_percent: int = 10
pyattribute

ptwm.CompressionConfig.codec

python/ptwm/_config.py:167
codec: str | None = None
pyattribute

ptwm.CompressionConfig.codec_menu

python/ptwm/_config.py:159
codec_menu: list[CodecId] | None = None

Restrict per-role trial-encode menus to this set of codecs.

None → use the full role-applicable menu. An empty intersection on a role raises ``InvalidContainer`` — an Identity fallback would silently falsify ablation measurements. Ignored when ``method`` selects a forced- codec path (ZSTD / RANS / IDENTITY).
pyattribute

ptwm.CompressionConfig.compressed_file

python/ptwm/_config.py:155
compressed_file: str | None = None
pyattribute

ptwm.CompressionConfig.compression_chunk

python/ptwm/_config.py:151
compression_chunk: int = 256 * 1024
pyattribute

ptwm.CompressionConfig.compression_threshold

python/ptwm/_config.py:145
compression_threshold: float = 0.95
pyattribute

ptwm.CompressionConfig.container_layout

python/ptwm/_config.py:171
container_layout: ContainerLayout = ContainerLayout.COMPACT

Layout of a full container written with ``Framing.CONTAINER`` (or as the ``Framing.COMPACT`` fallback): ``COMPACT`` (default, ``‰PTWC``) or ``LEGACY`` (``‰PTWM``, readable before v1.2); see :class:`ContainerLayout`. A string such as ``"legacy"`` is coerced on construction.

pyattribute

ptwm.CompressionConfig.decompressed_file

python/ptwm/_config.py:156
decompressed_file: str | None = None
pyattribute

ptwm.CompressionConfig.delta_compressed_type

python/ptwm/_config.py:148
delta_compressed_type: str | None = None
pyattribute

ptwm.CompressionConfig.device

python/ptwm/_config.py:168
device: int | None = None
pyattribute

ptwm.CompressionConfig.emit_payload_hash

python/ptwm/_config.py:177
emit_payload_hash: bool = True

Store an xxh64 of the raw bytes (8 B per tensor), checked on decode.

pyattribute

ptwm.CompressionConfig.emit_plane_crc

python/ptwm/_config.py:179
emit_plane_crc: bool = True

Store a CRC32 of every coded plane (4 B per plane), checked on decode.

A compact frame stores plane CRCs only when ``emit_payload_hash`` is off: the payload hash already covers every plane of its one tensor.
pyattribute

ptwm.CompressionConfig.framing

python/ptwm/_config.py:169
framing: Framing = Framing.COMPACT

Output layout; see :class:`Framing`.

pymethod

ptwm.CompressionConfig.from_resolved_policy

python/ptwm/_config.py:199
def from_resolved_policy(cls, rp: ResolvedPolicy, base: CompressionConfig | None = None) -> CompressionConfig

Build a constrained ``CompressionConfig`` from a ``ResolvedPolicy``.

The returned config's ``codec_menu`` is the intersection of the resolved policy's ``effective_global`` set with the wire-stable built-in ``CodecId`` variants. The resolved policy lists every contribution (built-in or installed third-party) that is currently trusted *and* allowed by the policy file. Restricting the trial-encode menu to that intersection is how an ablation run actually exercises a variant policy: tensors that previously round-tripped through a now-excluded codec will fall through to the next-best allowed coder instead. ``base`` is the starting config (defaults to a default-constructed ``CompressionConfig``); every field other than ``codec_menu`` is preserved.
pyattribute

ptwm.CompressionConfig.input_file

python/ptwm/_config.py:154
input_file: str | None = None
pyattribute

ptwm.CompressionConfig.input_format

python/ptwm/_config.py:141
input_format: Format = Format.BYTE
pyattribute

ptwm.CompressionConfig.is_monotonic

python/ptwm/_config.py:143
is_monotonic: int = 0
pyattribute

ptwm.CompressionConfig.is_streaming

python/ptwm/_config.py:152
is_streaming: bool = False
pyattribute

ptwm.CompressionConfig.lossy_compressed_factor

python/ptwm/_config.py:150
lossy_compressed_factor: int = 27
pyattribute

ptwm.CompressionConfig.lossy_compressed_type

python/ptwm/_config.py:149
lossy_compressed_type: Lossy = Lossy.NONE
pyattribute

ptwm.CompressionConfig.lz4_compression_level

python/ptwm/_config.py:158
lz4_compression_level: int = 0
pyattribute

ptwm.CompressionConfig.method

python/ptwm/_config.py:140
method: Method = Method.AUTO
pyattribute

ptwm.CompressionConfig.reorder_signbit

python/ptwm/_config.py:147
reorder_signbit: int = 0
pyattribute

ptwm.CompressionConfig.streaming_chunk

python/ptwm/_config.py:153
streaming_chunk: int = 1024 * 1024
pyattribute

ptwm.CompressionConfig.threads

python/ptwm/_config.py:144
threads: int | None = None
pyattribute

ptwm.CompressionConfig.zstd_level

python/ptwm/_config.py:157
zstd_level: int = 3
pyclass

ptwm.CompressionMethodNotSupportedError

python/ptwm/_exceptions.py:13
class CompressionMethodNotSupportedError(Error):

Raised when an unsupported compression method is requested.

pyclass

ptwm.Compressor

python/ptwm/core/_compressor.py:57
class Compressor:

Stateless compressor for weights data.

pymethod

ptwm.Compressor.__init__

python/ptwm/core/_compressor.py:60
def __init__(self, config: CompressionConfig) -> None
pymethod

ptwm.Compressor.compress

python/ptwm/core/_compressor.py:64
def compress(self, data: bytes | bytearray | np.ndarray | torch.Tensor, delta_second_data: bytes | bytearray | np.ndarray | torch.Tensor | None = None) -> bytes

Compress the provided data.

Streaming (``config.is_streaming=True``) enables plane-level chunking: each chunk is codec-encoded independently and indexed by ``plane_record.chunk_table``. Delta compression XORs the whole buffer once; the reference's BLAKE3 digest is recorded as a Dependency so the decoder can verify that the caller supplied the matching reference.
pyattribute

ptwm.Compressor.config

python/ptwm/core/_compressor.py:61
config = config
pyclass

ptwm.ContainerLayout

python/ptwm/_config.py:81
class ContainerLayout(Enum):

On-disk layout of a multi-tensor ``.ptwm`` container.

``COMPACT`` (magic ``\x89PTWC``, the default for new writes) stores built-in extensions by id, integers as varints and tensor names only in the index; it needs v1.2 or later to read. ``LEGACY`` (magic ``\x89PTWM``) is the fixed-width layout that v1.1 and later read. Its Extension Table interns all 16 built-in codecs in built-in order, as the v1.1.0 writer does, so a v1.1 transcoder rebuilds exploded LMDB and WebDataset stores of it correctly. Strings coerce case-insensitively: ``ContainerLayout("compact")`` and ``ContainerLayout("LEGACY")`` work, and the bindings take the lower-case :attr:`value`.
pyconstant

ptwm.ContainerLayout.COMPACT

python/ptwm/_config.py:97
COMPACT = 'compact'
pyconstant

ptwm.ContainerLayout.LEGACY

python/ptwm/_config.py:98
LEGACY = 'legacy'
pyclass

ptwm.DecompressionConfig

python/ptwm/_config.py:260
class DecompressionConfig:
pymethod

ptwm.DecompressionConfig.__init__

python/ptwm/_config.py:1
def __init__(self, threads: int | None = None, delta_second_data: bytes | None = None, skip_missing: bool = False, codec: str | None = None, device: int | None = None) -> None
pyattribute

ptwm.DecompressionConfig.codec

python/ptwm/_config.py:273
codec: str | None = None
pyattribute

ptwm.DecompressionConfig.delta_second_data

python/ptwm/_config.py:265
delta_second_data: bytes | None = None
pyattribute

ptwm.DecompressionConfig.device

python/ptwm/_config.py:274
device: int | None = None
pyattribute

ptwm.DecompressionConfig.skip_missing

python/ptwm/_config.py:266
skip_missing: bool = False

If ``True``, open containers that reference non-builtin extensions that are not installed on this host. Tensors needing those extensions will fail when decoded, but the container can still be opened and interrogated (e.g. listing tensor names). When ``False`` (the default), opening such a container raises ``ValueError``.

pyattribute

ptwm.DecompressionConfig.threads

python/ptwm/_config.py:264
threads: int | None = None
pyclass

ptwm.Decompressor

python/ptwm/core/_decompressor.py:31
class Decompressor:

Stateless decompressor for weights data.

pymethod

ptwm.Decompressor.__init__

python/ptwm/core/_decompressor.py:34
def __init__(self, config: DecompressionConfig | None = None) -> None
pyattribute

ptwm.Decompressor.config

python/ptwm/core/_decompressor.py:35
config = config or DecompressionConfig()
pymethod

ptwm.Decompressor.decompress

python/ptwm/core/_decompressor.py:38
def decompress(self, data: bytes | memoryview) -> bytes | np.ndarray | torch.Tensor

Decompress a ``.ptwm`` blob.

Accepts a compact single-tensor frame (magic ``\x89PTWS``) or a full container in either layout (magic ``\x89PTWM``, ``\x89PTWX`` or ``\x89PTWC``). The single-tensor container uses name ``"_"``. Raw decompressed bytes come from one :class:`ptwm._rust.ContainerHandle`; shape/dtype/input_format (when present) come from the CBOR shape-metadata map. Format recovery: - No metadata or ``input_format == BYTE`` → raw ``bytes``. - ``input_format == TORCH`` → :class:`torch.Tensor` reconstructed via ``torch.frombuffer`` then ``reshape(shape)``. - ``input_format == NUMPY`` → :class:`numpy.ndarray` via ``np.frombuffer`` + ``reshape(shape)``. Delta: when the tensor record carries a Dependency with a BLAKE3 hash, ``config.delta_second_data`` is required; its BLAKE3 must match, and the decoded bytes are XOR-ed with it to recover the original tensor before any format reconstruction.
pymethod

ptwm.Decompressor.decompress_container_to_state_dict

python/ptwm/core/_decompressor.py:175
def decompress_container_to_state_dict(self, blob: bytes) -> dict[str, bytes]

Decompress a container (either layout) into ``{name: raw_bytes}``.

Opens the container once and decodes each tensor in write order, so the keys come in the order the tensors were compressed. Result keys preserve the tensor names from the source safetensors archive.
pymethod

ptwm.Decompressor.decompress_file

python/ptwm/core/_decompressor.py:185
def decompress_file(self, path: str) -> bytes | np.ndarray | torch.Tensor

Decompress data from a file.

pyclass

ptwm.Error

python/ptwm/_exceptions.py:9
class Error(Exception):

Base exception for all errors originating from the PTWM library.

pyclass

ptwm.Format

python/ptwm/_config.py:30
class Format(Enum):
pyconstant

ptwm.Format.BYTE

python/ptwm/_config.py:31
BYTE = 1
pyconstant

ptwm.Format.FILE

python/ptwm/_config.py:34
FILE = 4
pyconstant

ptwm.Format.NUMPY

python/ptwm/_config.py:33
NUMPY = 3
pyconstant

ptwm.Format.TORCH

python/ptwm/_config.py:32
TORCH = 2
pyclass

ptwm.Framing

python/ptwm/_config.py:45
class Framing(Enum):

Byte layout of a single-tensor :meth:`Compressor.compress` output.

``COMPACT`` (default) writes a compact single-tensor frame (magic ``\x89PTWS``) with a fixed cost of at most about 70 bytes, and stores the raw bytes when no codec makes them smaller. It falls back to a full container when the tensor cannot use the frame (streaming, or a plane that selects shared codec state). ``CONTAINER`` always writes a full ``.ptwm`` container, which the random-access, store and transcode tools open. A full container (from ``CONTAINER`` or the ``COMPACT`` fallback) has the layout :attr:`CompressionConfig.container_layout` selects: ``\x89PTWC`` for :attr:`ContainerLayout.COMPACT` (the default) and ``\x89PTWM`` for :attr:`ContainerLayout.LEGACY`. So ``Framing.CONTAINER`` with the default layout gives a ``\x89PTWC`` container. Readers before v1.2 read only ``CONTAINER`` output in the :attr:`ContainerLayout.LEGACY` layout (magic ``\x89PTWM``); they reject the ``\x89PTWS`` frame and the ``\x89PTWC`` compact container. Set ``framing=Framing.CONTAINER`` and ``container_layout=ContainerLayout.LEGACY`` for output that older releases read.
pyconstant

ptwm.Framing.COMPACT

python/ptwm/_config.py:69
COMPACT = 1
pyconstant

ptwm.Framing.CONTAINER

python/ptwm/_config.py:70
CONTAINER = 2
pyclass

ptwm.HeaderParseError

python/ptwm/_exceptions.py:21
class HeaderParseError(Error):

Raised when the binary header is invalid or corrupt.

pyclass

ptwm.InvalidDTypeError

python/ptwm/_exceptions.py:17
class InvalidDTypeError(Error):

Raised when a tensor or numpy array has an unsupported dtype.

pyclass

ptwm.Lossy

python/ptwm/_config.py:124
class Lossy(Enum):
pyconstant

ptwm.Lossy.INTEGER

python/ptwm/_config.py:126
INTEGER = 1
pyconstant

ptwm.Lossy.NONE

python/ptwm/_config.py:125
NONE = 0
pyconstant

ptwm.Lossy.UNSIGN

python/ptwm/_config.py:127
UNSIGN = 2
pyfunction

ptwm.materialize_hf_cache

python/ptwm/integrations/_hf.py:137
def materialize_hf_cache(snapshot_dir: str | os.PathLike[str], remove_source: bool = False) -> list[Path]

Decompress every ``.ptwm`` file under ``snapshot_dir``.

Recursively walks the directory, decompressing each ``*.ptwm`` file into its sibling without the suffix (``model.safetensors.ptwm`` → ``model.safetensors``). Skips files whose decompressed form already exists, so repeat runs are cheap.
NameTypeDescription
snapshot_dirstr | os.PathLike[str]Any directory; typically an HF cache snapshot (``~/.cache/huggingface/hub/models--<org>--<name>/snapshots/<rev>``).
remove_source = FalseboolIf ``True``, delete each ``.ptwm`` file once its decompressed sibling is on disk. Defaults to ``False`` (keeps both).
Returns
list of :class:`pathlib.Path` — Paths of the decompressed files, in traversal order.
pyclass

ptwm.Method

python/ptwm/_config.py:13
class Method(Enum):
pyconstant

ptwm.Method.AUTO

python/ptwm/_config.py:14
AUTO = 0
pyconstant

ptwm.Method.HUFFMAN

python/ptwm/_config.py:15
HUFFMAN = 1
pyconstant

ptwm.Method.IDENTITY

python/ptwm/_config.py:17
IDENTITY = 3
pyconstant

ptwm.Method.MICROSCALE

python/ptwm/_config.py:19
MICROSCALE = 5
pyconstant

ptwm.Method.RANS

python/ptwm/_config.py:18
RANS = 4
pyconstant

ptwm.Method.ZSTD

python/ptwm/_config.py:16
ZSTD = 2