Schema in, validated JSON out: extract structured data from messy text with a validate-and-repair loop, bounded retries and a source span for every field.
Status: v0.1.0, working. JSON repair, schema validation, the retry loop, source spans and a small CLI are implemented and tested. Per-field confidence scores and the TypeScript version are not built yet.
You bring the model as a plain function, complete(prompt: str) -> str. This library has no dependencies and never imports a vendor SDK, so it works with any provider, a local model, or a fake in your tests.
from structured_extract import extract
result = extract(Invoice, text, complete=my_llm)
result.value # an Invoice (dataclass or Pydantic model), validated
result.spans # where each field was found in the text
result.attempts # every prompt, reply and rejection reason- Your schema (a JSON Schema dict, a dataclass or a Pydantic v2 model) is turned into a prompt together with the document.
- The reply is parsed. Code fences, chatty prose around the JSON, single quotes, trailing commas, unquoted keys, Python literals and cut-off output are repaired locally, without another model call.
- Near-miss types are coerced when it is lossless (
"764.88"to764.88for a number field). - The result is validated against the schema. If it fails, the exact errors are sent back and the model gets another try, up to
max_retriestimes. - Each string and number in the result is looked up in the source text, so you can show where it came from, or spot values that appear nowhere in the document.
Python 3.10 or newer.
pip install git+https://github.com/narayana006/structured-extractPydantic is optional. If you pass Pydantic models, install it yourself (pip install "pydantic>=2").
Clone the repository to get the examples; they run offline against a scripted stand-in for a model (examples/fake_llm.py) whose first answer is deliberately wrong.
git clone https://github.com/narayana006/structured-extract
cd structured-extract
pip install -e .
python examples/quickstart.pyOutput:
INV-2024-0317: Brightside Dental Group owes 764.88 USD
issued 14 Mar 2024, 3 line items, first: LineItem(description='A4 copy paper, 80gsm (box)', quantity=12, amount=462.0)
attempt 1: rejected
- total: required field is missing
- issue_date: is not a valid date: 'March 14, 2024'
attempt 2: accepted
repairs on the accepted reply: ['removed a trailing comma']
invoice_number -> chars 120-133: 'INV-2024-0317'
vendor -> chars 0-26: 'NORTHWIND PAPER SUPPLY CO.'
total -> chars 845-851: '764.88'
line_items[1].quantity -> chars 520-521: '4'
not found in the source: []
The first reply was missing total and had a date in the wrong format, so both errors were sent back. The second reply still had a trailing comma, which was fixed locally instead of costing a third call. Note that vendor was matched case-insensitively against the letterhead.
The core of examples/quickstart.py:
import datetime
from dataclasses import dataclass
from typing import Optional
from structured_extract import extract
@dataclass
class LineItem:
description: str
quantity: int
amount: float
@dataclass
class Invoice:
invoice_number: str
vendor: str
customer: str
issue_date: datetime.date
currency: str
total: float
line_items: list[LineItem]
due_date: Optional[str] = None
result = extract(Invoice, text, complete=fake_llm.reply)
invoice = result.valueTo use a real model, replace fake_llm.reply with your own function. It receives the prompt as a string and must return the reply as a string; how it calls the model is up to you.
extract(schema, text, complete, *, max_retries=2, instructions="", coerce_types=True, require_grounding=False)
Returns an Extraction:
| Attribute | Contents |
|---|---|
data |
The validated result as plain JSON data (dicts, lists, strings, numbers). |
value |
The same result as an instance of your dataclass or Pydantic model. For a JSON Schema dict it is data. |
spans |
{path: Span or None} for every string and number, e.g. spans["line_items[0].amount"]. A Span has start, end and text. |
ungrounded |
Paths whose value was not found in the source. |
attempts |
A list of Attempt(prompt, response, errors, repairs), one per model call. |
repairs |
Fixes applied to the accepted reply. |
complete is called at most max_retries + 1 times. If no reply validates, ExtractionError is raised; its .attempts holds the full history.
instructionsadds your own guidance to the prompt (for example "amounts are in GBP").coerce_types=Falseturns off type coercion, so"36"for an integer field is an error.require_grounding=Truetreats any string or number that cannot be found in the text as an error and sends it back to the model. Use it when every field should be copied verbatim; leave it off when fields are normalised or inferred (ISO dates from "March 14", categories, booleans).
- JSON Schema dict. Supported keywords:
type,properties,required,additionalProperties,items,minItems,maxItems,enum,const,minimum,maximum,exclusiveMinimum,exclusiveMaximum,minLength,maxLength,pattern,format(date,date-time,email),anyOf,oneOf,allOfand local$ref. Other keywords are ignored. - Dataclass. Fields may be
str,int,float,bool,datetime.date,Optional[...],Literal[...],list[...],dictor another dataclass. Fields without a default are required.field(metadata={"description": "..."})is passed to the model. - Pydantic v2 model. The schema comes from
model_json_schema()and the final object frommodel_validate(), so your own validators run too, and their error messages are fed back to the model like any other validation error.
Runs extract over several documents in order and returns a list with an Extraction or an ExtractionError in each slot, so one bad document does not stop the batch. It is sequential; there is no concurrency or rate limiting yet.
Each step is usable on its own:
from structured_extract import parse_json, coerce, validate, find_span
parse_json("Sure! ```json\n{'name': 'Ada', age: 36,}\n```")
# ({'name': 'Ada', 'age': 36},
# ['removed a Markdown code fence', 'converted non-standard quotes to double quotes',
# 'quoted an unquoted key', 'removed a trailing comma'])
schema = {"type": "object", "required": ["age"],
"properties": {"age": {"type": "integer", "minimum": 0}}}
coerce({"age": "36"}, schema)
# ({'age': 36}, ["age: converted string '36' to number"])
[str(issue) for issue in validate({"age": -1}, schema)]
# ['age: must be >= 0, got -1']
find_span(1250.0, "Total due: 1,250.00 EUR")
# Span(start=11, end=19, text='1,250.00')check_output(schema, reply, source) runs parse, coerce, validate and span lookup on a reply you already have, without calling a model. build_prompt(schema, text) returns the prompt.
The package installs a structured-extract command with three subcommands. Schemas on the command line are JSON Schema files.
Check a reply you already have. examples/model_reply.txt is a typical chatty reply: prose, a code fence, single quotes on one line, a trailing comma and the total as a string.
$ structured-extract check examples/model_reply.txt --schema examples/invoice.schema.json | head -9
{
"valid": true,
"errors": [],
"repairs": [
"removed a Markdown code fence",
"converted non-standard quotes to double quotes",
"removed a trailing comma",
"total: converted string '764.88' to number"
],The full report also contains the cleaned data. Add --source examples/invoice.txt to get a spans section, and --require-grounding to fail when a value is not in the source. The exit status is 0 when valid and 1 when not, so it can gate a pipeline.
Run the whole loop with any command as the model. The command receives the prompt on stdin and must print the reply on stdout:
$ structured-extract run examples/invoice.txt --schema examples/invoice.schema.json --llm-cmd "python3 examples/fake_llm.py" | head -8
{
"invoice_number": "INV-2024-0317",
"vendor": "Northwind Paper Supply Co.",
"customer": "Brightside Dental Group",
"issue_date": "2024-03-14",
"due_date": "2024-04-13",
"total": 764.88,
"currency": "USD",Options: --max-retries N (default 2), --timeout SECONDS per call (default 120), --require-grounding, and --report to print repairs, spans and per-attempt errors along with the data.
Print the prompt to paste into a chat window or inspect:
structured-extract prompt examples/invoice.txt --schema examples/invoice.schema.json- JSON Schema, dataclass and Pydantic v2 schemas (Python)
- JSON parsing with local repair of malformed model output
- Validate-and-repair loop with bounded retries
- Source span for each extracted string and number
- Sequential batch helper (
extract_many) - CLI:
check,run,prompt - Per-field confidence scores
- Concurrent batch mode with rate limiting
- Zod schemas and a TypeScript version
- Chunking for documents longer than the model's context window
- No confidence scores. The only signal today is whether a value was found in the source (
spans,ungrounded). - Span lookup is literal. Strings match exactly or ignoring case and whitespace; numbers match numerals of equal value. A value the model normalised (
"2024-03-14"from "March 14, 2024") or inferred has no span. When a string occurs several times, the first occurrence is reported; for a repeated number, the occurrence closest to a string field of the same object is chosen, which is a heuristic and can pick the wrong one. - A span shows that the value appears in the text, not that it is the right value. The model can still copy the wrong number.
- The JSON Schema support is a subset (listed above). There is no
patternProperties,if/then,dependentRequired,uniqueItems,multipleOfor remote$ref. - JSON repair is best effort. Badly broken output can be repaired into something that parses but is not what the model meant; the schema check is the safety net, so write a strict schema (
required,additionalProperties: false). - The whole document is sent in one prompt. Splitting long documents is up to you.
- Prompts are in English and not tuned for any particular model.
Bug reports are most useful with the model reply that was mis-parsed or the schema and value that were validated wrongly. Pull requests are welcome:
pip install -e ".[dev]"
pytestTests use scripted fake models only; nothing in the test suite calls a real model or the network.
Built and maintained by Narayana Vemburaj.