ptwm_core
ptwm_core::transcode
PTWM container ⇄ canonical PTWM-JSON manifest transcoder.
ptwm_core::transcode
PTWM container ⇄ canonical PTWM-JSON manifest transcoder.
ptwm_core::transcode
crates/ptwm-core/src/transcode/mod.rs:1mod transcodePTWM container ⇄ canonical PTWM-JSON manifest transcoder.
The binary `.ptwm` container is structurally *(a)* a registry of
codecs/ops/shared-state/chains, *(b)* per-tensor records, and *(c)* per-plane
compressed payloads. This module lifts that into a declarative manifest
(see [`manifest`]) plus loose binary members, and back again.
* [`explode`] parses a container into `(registry_json, per-tensor manifests,
loose members)`.
* [`implode`] reverses it, re-driving the deterministic container writer so
that `implode(explode(blob)) == blob` byte-for-byte for every container the
compression engine produces.
Both container layouts are supported. A legacy (`\x89PTWM`) container gives
a [`RegistryManifest`] (`"ptwm-manifest"`, version 1); `implode` rebuilds
its Extension Table from the chains and codecs and checks the recorded
hash. A compact (`\x89PTWC`) container gives a [`CompactRegistryManifest`]
(`"ptwm-manifest-c"`, version 2) that lists the table entries, the
registry chains (a bound Source has `params_b64` `null`) and the tensors in
body order; `implode` writes them back as listed. Readers that predate the
compact manifest reject it on its format and version. Tensor manifests
have the same form for both layouts. `implode` with a subset of the tensor
manifests writes a container of those tensors with the archive-wide
tables, so each tensor decodes as in the full container.
The encoder may build a container transiently in memory; the stored artifact
(WebDataset shard / LMDB exploded value) never contains one.
ptwm_core::transcode::explode
crates/ptwm-core/src/transcode/mod.rs:324pub fn explode(blob: &[u8]) -> Result<Exploded, PtwmCoreError>Parse a `.ptwm` container (either layout) into a registry manifest, per-tensor manifests in body order, and loose binary members.
ptwm_core::transcode::Exploded
crates/ptwm-core/src/transcode/mod.rs:64pub struct ExplodedResult of exploding a container.
ptwm_core::transcode::ExplodedTensor
crates/ptwm-core/src/transcode/mod.rs:74pub struct ExplodedTensorptwm_core::transcode::implode
crates/ptwm-core/src/transcode/mod.rs:789pub fn implode(registry_json: &str, tensor_jsons: &[String], members: HashMap<String, &[u8]>) -> Result<Vec<u8>, PtwmCoreError>Reconstruct a `.ptwm` container from a registry manifest, per-tensor manifests (in file order), and loose binary members. The registry manifest's `ptwm_format` and `manifest_version` select the layout; any other pair is rejected.