feat: encode the remaining types Pydantic's JSON mode knows

[shcheklein]

A model field holding one of several ordinary types cannot be stored:

class Doc(dc.DataModel):
    source: pathlib.PurePosixPath
    ttl: datetime.timedelta

class Batch(dc.DataModel):
    docs: list[Doc]

dc.read_values(b=[Batch(docs=[Doc(source=..., ttl=...)])]).save("b")
# TypeError: Object of type PurePosixPath is not JSON serializable

Why it depends on where the model sits

Two things can turn a model into storable JSON, and they know different types:

	converts with	knows
flatten	model_dump() — python mode	leaves Python objects for this encoder
warehouse	model_dump(mode="json")	Pydantic converts everything it knows

Which one runs is decided by the container:

list[Model]        flatten     -> python mode -> this encoder
tuple[Model, ...]  warehouse   -> JSON mode   -> Pydantic

So the same field works in one shape and fails in the other. Measured, for a declared field type:

                json mode (warehouse)   python mode + this encoder
timedelta       'PT1M30S'               TypeError
PurePosixPath   '/a/b'                  TypeError
IPv4Address     '1.2.3.4'               TypeError
plain enum      1                       TypeError
set[int]        [1, 2]                  TypeError
datetime/UUID/str enum                  same
ndarray         PydanticSerializationError   [1, 2]

This PR closes five of those rows, plus IPv6Address. Each is written the way Pydantic's JSON mode writes it, and the tests assert that equivalence against model_dump(mode="json") rather than against a fixed string, so they stay honest if Pydantic changes.

Scope

All six raise today, so nothing that currently writes changes. Eight of the nine new cases fail against main.

Decimal is deliberately left out: it already writes, as a number where Pydantic writes a string (1.2 vs "1.20"), so aligning it would change stored data rather than add to it. That belongs in its own change.

This is a prerequisite for making the two converters agree — with the type gap closed, which one runs stops changing what gets written.

[cloudflare-workers-and-pages]

Deploying datachain with    Cloudflare Pages

Latest commit:	44a4199
Status:	✅  Deploy successful!
Preview URL:	https://e45487c1.datachain-2g6.pages.dev
Branch Preview URL:	https://feat-json-encoder-extra-type.datachain-2g6.pages.dev

View logs

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[shcheklein]

Closing — superseded by a different direction.

This taught the encoder the types Pydantic's JSON mode already knows, so that python mode could be used everywhere. We are going the other way instead: use Pydantic's JSON mode everywhere, which covers those types natively and makes this unnecessary.

What that direction needs instead is numpy normalized before the dump, since Pydantic refuses ndarray where this encoder handles it. Measured, that costs about 1 ms on top of the tolist() that has to happen anyway.

Keeping the measurement here for whoever picks it up — declared field types, JSON mode versus python mode plus this encoder:

	json mode	python mode + encoder
timedelta	'PT1M30S'	TypeError
PurePosixPath	'/a/b'	TypeError
IPv4Address	'1.2.3.4'	TypeError
plain enum	1	TypeError
set[int]	[1, 2]	TypeError
Decimal	'1.20'	1.2
ndarray	PydanticSerializationError	[1, 2]