iokit

Python library for serialization and file operations. Unifies multiple format codecs (JSON, YAML, Tar, etc.) with a single State interface that combines data, path, and timestamp.

Installation

Base install covers txt, bin, dat, json, gz, zip, tar:

pip install iokit

For all formats and web download support:

pip install iokit[ultra]

Missing formats will tell you what to install:

from iokit import Yaml
Yaml({"key": "value"}, "file")

ModuleNotFoundError: Missing required packages: PyYAML>=6.0.1. Install with: pip install PyYAML

Quick Start

Work with data as States. Each State carries the data, a path, and a timestamp. Load and save without thinking about formats.

JSON file:

from iokit import Json

state = Json({"key": "value"}, path="config.json")
print(state.path)      # config.json
print(state.size)      # 16
print(state.load())    # {'key': 'value'}

Text file:

from iokit import Txt

state = Txt("Hello, World!", "message")
state.save("/tmp/data", parents=True)

Load any file from disk:

from iokit import file

state = file("/path/to/file.txt")
content = state.load()

Common Operations

Chain transformations:

from iokit import Txt

state = Txt("Secret data", "notes")
encrypted = state.encrypt(password="secret")
compressed = encrypted.gzip()
compressed.save("/tmp")

Load compressed:

loaded = encrypted.load(password="secret").load()

Archives:

from iokit import Tar, Txt

file1 = Txt("First", "a")
file2 = Txt("Second", "b")

archive = Tar([file1, file2], "bundle")
states = list(archive.load())

Find states by pattern:

from iokit import filtrate, first

results = filtrate(archive.load(), "*.txt")
first_match = first(archive.load(), "a*")

Download:

from iokit import web

state = web("https://example.com/data.json", Json)
data = state.load()

A data: URL is decoded where it stands, without reaching the network. It carries no name for its payload, so the state is left with the bare extension of its media type:

state = web("data:application/json;base64,eyJhIjogMX0=")
assert state.path == ".json"
assert state.load() == {"a": 1}

Checksum:

state.digest("sha256").base64
state.digest("xxh128").base64url

Payloads of your own, filed under a format that knows nothing of them:

from dataclasses import dataclass
from typing import Any

from iokit import Json


@dataclass
class Person:
    name: str
    age: int


class PersonJson(Json[Person]):
    def dump(self, data: Person) -> dict[str, Any]:
        return {"name": data.name, "age": data.age}

    def parse(self, data: dict[str, Any]) -> Person:
        return Person(data["name"], data["age"])


state = PersonJson(Person("Joe", 32), "joe")
print(state.path)    # joe.json
print(state.load())  # Person(name='Joe', age=32)

The file stays an ordinary joe.json, and state.load() is a Person both at runtime and for the type checker. Every format takes the same pair: Txt, Csv, Npy, and the rest.

Storage

Store and retrieve records by uid:

from iokit.storage import LocalStorage

storage = LocalStorage("/data")
storage.push("records/data.json", data)

loaded = storage.pull("records/data.json")
storage.remove("records/data.json")

for uid in storage.index(prefix="records/"):
    print(uid)

State-aware storage with automatic encoding:

from iokit.storage import StateStorage

storage = StateStorage(
    LocalStorage("/data"),
    compression=6,
    password="secret"
)

storage.push("file.json", {"data": 123})
result = storage.pull("file.json")

Contributing

Found a bug or have an idea? Open an issue or submit a pull request.