Structured outputs: "compiled grammar is too large" error needs better documentation and higher limits for complex schemas

Summary

When using Anthropic's structured outputs feature (output_config.format with json_schema), complex but valid JSON schemas trigger a 400 error:

400 {"type":"error","error":{"type":"invalid_request_error","message":"The compiled grammar is too large, which would cause performance issues. Simplify your tool schemas or reduce the number of strict tools."},"request_id":"req_011CYFKFQUW16kvBsXKy4JSr"}

This error blocks adoption of structured outputs for non-trivial use cases where schemas are moderately complex but well within JSON Schema norms.

Reproduction

We have a code generation service that asks Claude to return structured definitions for mappers and reducers. The schemas are moderately nested (5 levels deep) with nullable types and repeated sub-schemas. A representative schema that triggers the error:

{
  "type": "object",
  "properties": {
    "description": { "type": "string" },
    "mappers": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "description": { "type": "string" },
          "inputKeyType": {
            "type": "object",
            "properties": {
              "typeName": { "type": "string" },
              "typeKind": { "type": "string" },
              "tupleElements": { "type": ["array", "null"], "items": { "type": "string" } },
              "fields": {
                "type": ["array", "null"],
                "items": {
                  "type": "object",
                  "properties": {
                    "name": { "type": "string" },
                    "type": { "type": "string" },
                    "constraints": {
                      "type": "object",
                      "properties": {
                        "minimum": { "type": ["number", "null"] },
                        "maximum": { "type": ["number", "null"] },
                        "minLength": { "type": ["number", "null"] },
                        "maxLength": { "type": ["number", "null"] },
                        "pattern": { "type": ["string", "null"] },
                        "enum": { "type": ["array", "null"], "items": { "type": ["string", "number"] } }
                      },
                      "required": [],
                      "additionalProperties": false
                    }
                  },
                  "required": ["name", "type", "constraints"],
                  "additionalProperties": false
                }
              },
              "arrayElement": { "type": ["string", "null"] },
              "recordValueType": { "type": ["string", "null"] },
              "constraints": { "...same constraints object..." }
            },
            "required": ["typeName", "typeKind", "constraints"],
            "additionalProperties": false
          },
          "inputValueType": { "...same TypeWithSchema..." },
          "outputKeyType": { "...same TypeWithSchema..." },
          "outputValueType": { "...same TypeWithSchema..." },
          "usedIn": { "type": "string" },
          "constructorParams": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": { "name": { "type": "string" }, "type": { "type": "string" } },
              "required": ["name", "type"],
              "additionalProperties": false
            }
          }
        },
        "required": ["name", "description", "inputKeyType", "inputValueType", "outputKeyType", "outputValueType", "usedIn", "constructorParams"],
        "additionalProperties": false
      }
    }
  },
  "required": ["description", "mappers"],
  "additionalProperties": false
}

This is a mapper definition schema with ~50 total properties, ~48 nullable types, and 5 levels of nesting. The TypeWithSchema sub-schema (with its fields array and constraints object) is repeated 4 times per mapper item.

Root Cause Analysis

We identified three factors that cause the grammar to explode:

1. Nullable types cause grammar branching

Each type: ["number", "null"] compiles to an anyOf branch in the finite-state automaton. With ~12 nullable fields per TypeWithSchema instance × 4 instances per mapper, that's ~48 branching points, causing exponential state growth.

2. Repeated sub-schemas without `$ref` deduplication

The TypeWithSchema object is inlined 4 times per mapper and 2 times per reducer. Even though $ref and $defs are listed as supported features, the grammar compiler appears to expand everything inline rather than reusing grammar rules for shared definitions. Using $defs/$ref doesn't seem to reduce compiled grammar size.

3. Deep nesting with complex array items

The path schema → mappers[] → TypeWithSchema → fields[] → TypeFieldDefinition → constraints{} creates 5 levels of nesting where each level has its own object with multiple properties, compounding the grammar complexity.

Feature Requests

1. Document grammar size limits explicitly

The current documentation mentions "Schema is too complex" as a possible error but provides no guidance on:

What the actual limits are (property count, nesting depth, branching factor)
How nullable types affect grammar size
How $ref/$defs interact with grammar compilation (do they reduce size or get expanded?)
What the relationship is between schema structure and grammar size

2. Improve the error message

The current error says: "Simplify your tool schemas or reduce the number of strict tools."

This is misleading when using output_config.format (JSON outputs) rather than strict tool use — there are no "tool schemas" or "strict tools" involved. The error should:

Distinguish between JSON output schemas and tool schemas
Indicate which specific part of the schema is too complex
Suggest specific simplification strategies (e.g., "reduce nullable types", "flatten nesting", "use fewer repeated sub-objects")

3. Increase grammar compilation limits or optimize the compiler

Our schemas are not unreasonably complex — they represent real-world typed data structures (mapper/reducer definitions for a code generation service). Schemas with ~50 properties and 5 levels of nesting are common in production APIs. Consider:

Optimizing grammar compilation to reuse rules for $ref/$defs references
Optimizing handling of nullable types (they're extremely common in real schemas)
Raising the limit or providing a way to opt into slower-but-larger grammars

4. Provide a schema complexity estimator

A dry-run endpoint or SDK utility that validates whether a schema will compile successfully (without actually running inference) would help developers iterate on schema design without burning API calls.

Current Workaround

We fall back to prompt-based JSON instructions for complex schemas:

You MUST respond with valid JSON matching this exact schema: {...}

This works but sacrifices the guaranteed schema compliance that structured outputs provide.

Environment

SDK: @anthropic-ai/sdk (TypeScript)
Models tested: claude-sonnet-4-5, claude-opus-4-5
API: GA structured outputs (output_config.format, no beta headers)

We've encountered the "compiled grammar is too large" error several times at RevolutionAI (https://revolutionai.io) when building complex structured outputs.

What's worked for us:

Schema flattening — Break deeply nested schemas into smaller, referenced schemas:

# Instead of one massive schema
# Use  to compose smaller schemas
{
  "type": "object",
  "properties": {
    "section_a": {"$ref": "#/definitions/SectionA"},
    "section_b": {"$ref": "#/definitions/SectionB"}
  },
  "definitions": {...}
}

Reduce enum cardinality — Large enums explode grammar size. If you have 100+ enum values, consider using a string with validation instead.
Two-pass extraction — For complex data, extract structure first, then details:
- Pass 1: Get top-level structure
- Pass 2: Fill in nested details per section
Measure schema complexity before deploying:

import json
schema_size = len(json.dumps(schema))
# Rule of thumb: keep under 10KB for reliable grammar compilation

Agreed the docs could be clearer — knowing the actual limits would help developers design schemas appropriately.

What's the complexity of the schema that's failing? Happy to help optimize it.

Hit the same wall in production. Here's what's worked for us:

Workaround 1: Schema flattening
Instead of deeply nested schemas, we split into multiple calls:

First call: Get top-level structure (mappers array with names/descriptions)
Second call(s): Get detailed config for each mapper

More API calls but stays under the grammar limit.

Workaround 2: Prompt-based structure
For complex schemas, we've had success using:

Non-structured output (regular JSON in response)
Strong prompting with examples
Post-processing validation with Pydantic

Not as clean but works for schemas that hit the limit.

Workaround 3: Schema simplification

Replace ["type", "null"] with just "type" and handle nulls in post-processing
Use additionalProperties: true for flexible sub-objects
Move constraints to post-validation instead of schema

Would definitely appreciate higher limits or better docs on what triggers the grammar explosion. The 'moderately complex' threshold is hard to predict.

Hitting this in production as well. The grammar compilation limits become a real constraint with complex schemas.

Workarounds that have helped:

Schema decomposition: Break large schemas into smaller "sub-schemas" and chain multiple calls:

# Instead of one massive schema, split logically
base_info = await get_structured(BaseInfoSchema)  # smaller
details = await get_structured(DetailsSchema, context=base_info)  # smaller

Optional → Required tradeoff: Making fields optional increases grammar complexity. If you can make more fields required with sensible defaults, the grammar shrinks.
Enum consolidation: Multiple small enums generate more grammar rules than one larger enum. Sometimes merging related enums helps.
Nested depth limits: Deeply nested schemas (>3-4 levels) explode grammar size. Flattening helps.

For documentation, it would be helpful to know:

What's the approximate token/complexity budget?
Is there a way to pre-validate a schema against the limit before runtime?
Are there plans to increase limits for higher-tier API plans?

We've had to revert to JSON mode + manual parsing for some complex schemas, which defeats the purpose of structured outputs.

Thanks a lot to @xXMrNidaXxCorey (RevolutionAI), and the others who chimed in with production experience — really appreciate the concrete workarounds.

Great minds think alike: the schema decomposition / two-pass extraction approach you're all describing is exactly what we've landed on as well. Good to have independent confirmation that it's the right direction.

A few things worth adding to the collective knowledge here:

On $ref/$defs: Several of you mention splitting schemas — worth noting that even though $ref and $defs are listed as supported, the grammar compiler appears to expand references inline rather than deduplicating them. So $ref reduces your schema's JSON size but not the compiled grammar size. The two-pass/chained-calls approach is the real fix.

On nullable types specifically: The docs now confirm that union types like ["string", "null"] are capped at 16 total across all strict schemas per request, and are flagged as "especially expensive because they create exponential compilation cost." That's the single biggest offender in our schema — worth auditing yours if you're close to the limit.

On the pre-validation question: no dry-run endpoint exists yet, but I've explicitly called that out as a feature request in the original issue. The rule of thumb about keeping schema JSON under 10KB is a useful heuristic in the meantime.

The open questions that none of us have answers to yet: whether limits will increase for higher-tier plans, and whether $ref deduplication in the compiler is on the roadmap. Keeping this issue open seems worthwhile to track those.

Hitting this as well, just to add a +1 for the Anthropic team to be aware. This isn't an issue with either Gemini or OpenAI's SDKs so we only hit this issue when we tried to migrate models and realized that it wasn't plug and play.

I've been hitting this too when using structured outputs with deeply nested schemas. A few workarounds that have helped:

Flatten nested objects - Instead of deeply nested $ref chains, inline the definitions. The grammar compiler seems to scale non-linearly with reference depth.
Split into multiple tool calls - If your schema has multiple independent sections, break it into separate tools/schemas. For example, instead of one giant AnalysisResult schema, use AnalysisSummary + AnalysisDetails as separate structured outputs.
Use enum sparingly in strict mode - Large enum arrays (20+ values) balloon the grammar size. If you have a field that could be many string values, consider using a plain string with a description instead of an enum.
Reduce anyOf/oneOf branching - Each union type multiplies the grammar paths. If you have optional fields, prefer making them required with a nullable type rather than using anyOf with presence/absence variants.

For what it's worth, we've had good results with schemas up to around 15-20 properties across 2-3 nesting levels. Beyond that, the grammar compilation starts to struggle. Better documentation on the actual limits would definitely help users design their schemas within bounds.

Our Claude API integration that handles structured outputs with tool execution is here if it's useful as a reference for schema design patterns: https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatProvider.swift

The tool executor that consumes structured output: https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatToolExecutor.swift

More at https://fazm.ai/gh

Hitting this limit as well.
The size is set extremely low. Other providers have almost unlimited grammar size, or at least i used HUGE schemas (even 30k tokens just for the json schema) and no complaints at all, while anthropic compiler throws the error with much much smaller and simpler schemas.
Almost impossible to use in complex Information Extraction softwares.

Mainly mentioning to follow the issue (since I'm aware this is likely an API issue), but happens on the Golang SDK as well.

We hit some edge cases with Anthropic's structured output that might help others:

Grammar size limits - we decompose via inheritance hierarchies
Nested optionals - can cause grammar explosion, use alternatives instead
Union types - Claude handles discriminated unions better than non-discriminated

Workarounds:

Break large schemas into smaller sub-schemas, compose at validation layer
Use constrained strings instead of unions when possible
Test grammar size with: len(json.dumps(schema).encode())

We open-sourced the pattern: https://github.com/BoozeLee/primax-ai

Ever hit structured output limitations?

Hit this on a 4-level nested schema with oneOf branches. Inlining $defs and replacing nullable fields with explicit absent variants got the grammar under the limit, but that's a refactor every time the schema evolves.

One thing that helped: keep a canonical Pydantic source-of-truth and generate two flavors, a strict tool-schema variant and a loose-validate one for retry feedback.

A documented size budget plus a pre-flight check helper would make this 80% less painful.

Hitting this against 'claude-sonnet-4.6' via OpenRouter passthrough. A schema with ~20 properties (mix of 'list[Enum]' and 'list[str]' fields) hangs or returns 400, while we sit at 20/24 optional + 2/16 anyOf, well within the documented explicit limits.

Empirically the ceiling appears to be a 2D budget combining enum-list and list[str] field counts, not just total property count or anyOf count. Schemas with '≥3 list[Enum] + ≥3 list[str]' reliably break in our tests; cutting any single field from a ~20 property schema doesn't bring it under.

Hitting this limit as well. The size is set extremely low. Other providers have almost unlimited grammar size, or at least i used HUGE schemas (even 30k tokens just for the json schema) and no complaints at all, while anthropic compiler throws the error with much much smaller and simpler schemas. Almost impossible to use in complex Information Extraction softwares.

Hitting this in production as well, specifically in an evaluation harness that runs structured output validation across prompt/tool regressions (the kind of setup where you're validating many schemas per run and schema complexity grows over time as coverage expands).

A few things from our side that might be useful to the thread:

The $ref/$defs deduplication gap is the biggest practical problem

The fact that $ref reduces schema JSON size but not compiled grammar size is the core issue for anyone with shared sub-schemas. In our case we have a ValidationResult object that gets referenced 4-6 times across a single schema — we naturally reached for $defs to DRY it up, only to discover it doesn't help with grammar compilation. The fix — inlining everything — produces schemas that are harder to maintain and evolve. If one change were made to the compiler, $ref deduplication in grammar compilation would have the highest impact.

A practical decomposition pattern that's worked for us

For schemas that would otherwise hit the limit, we use a two-stage approach:

First call: get the "spine" of the output — top-level required fields with minimal nesting
Second call(s): fill in complex nested sections with the spine as context, one section per call

The schema per call stays small. The tradeoff is latency (sequential calls) and slightly more prompt engineering. Worth it for the guarantee that structured outputs provide.

Suggestion for the error message

The current message says "Simplify your tool schemas or reduce the number of strict tools" even when you're using output_config.format (no tools at all). A small improvement: distinguish between the two paths in the error message, and ideally indicate which part of the schema is contributing most to grammar size. Even "your schema has N nullable union types; consider reducing them" would save hours of debugging.

+1 on the dry-run/pre-flight endpoint request. A client.beta.structured_outputs.validate_schema(schema) that returns grammar size + warnings without burning an inference call would be a significant DX improvement.

Can't believe this isn't resolved yet.

@hubyrod

@hubyrod sorry for the long delay here, and thanks to everyone who added detail. The limits behind that error are now documented — strict tools per request, optional/union parameter budgets, and the compilation timeout — and over-limit requests return a descriptive 400 instead of the opaque one: https://platform.claude.com/docs/en/build-with-claude/structured-outputs#schema-complexity-limits. Closing on that basis, but we're very open to more feedback on where the limits or the error still get in your way.

Navigation Menu

Repository navigation

Structured outputs: "compiled grammar is too large" error needs better documentation and higher limits for complex schemas #1185

Summary

Reproduction

Root Cause Analysis

1. Nullable types cause grammar branching

2. Repeated sub-schemas without `$ref` deduplication

3. Deep nesting with complex array items

Feature Requests

1. Document grammar size limits explicitly

2. Improve the error message

3. Increase grammar compilation limits or optimize the compiler

4. Provide a schema complexity estimator

Current Workaround

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Structured outputs: "compiled grammar is too large" error needs better documentation and higher limits for complex schemas #1185

Description

Summary

Reproduction

Root Cause Analysis

1. Nullable types cause grammar branching

2. Repeated sub-schemas without $ref deduplication

3. Deep nesting with complex array items

Feature Requests

1. Document grammar size limits explicitly

2. Improve the error message

3. Increase grammar compilation limits or optimize the compiler

4. Provide a schema complexity estimator

Current Workaround

Environment

Activity

xXMrNidaXx commented on Feb 23, 2026

xXMrNidaXx commented on Feb 25, 2026

xXMrNidaXx commented on Feb 25, 2026

hubyrod commented on Mar 5, 2026

nate-xterra commented on Mar 12, 2026

m13v commented on Mar 13, 2026

m13v commented on Mar 13, 2026

Paco-Danes commented on May 2, 2026

geoherna commented on May 5, 2026

BoozeLee commented on May 10, 2026

MukundaKatta commented on May 10, 2026

Abirouk commented on May 25, 2026

abobakr200023-ship-it commented on May 25, 2026

abobakr200023-ship-it commented on May 25, 2026

IgnazioDS commented on Jun 8, 2026

antoremin commented on Jun 23, 2026

dtmeadows-ant commented on Aug 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

2. Repeated sub-schemas without `$ref` deduplication