Support PostgreSQL hex bytea literals

[xiangfu0]

PR flow

Parses PostgreSQL :: bytea casts, normalizes to binary literals, and flows them to broker, expression, planner, and integration paths.

flowchart TD
  N0["Grammar extension for INFIX#95;CAST #58;#58; and extraBinaryExpressions #40;F1#44; F2#41;"]:::stAdded
  N1["Parsing yields SqlCall for #58;#58; or CAST #40;F2#41;"]:::stAdded
  N2["PostgreSqlCastRewriter rewrites BYTEA casts to SqlBinaryStringLiteral #40;F3#44; F4#41;"]:::stAdded
  N3["Literal#45;only broker path processes resulting BYTES literal #40;F5#41;"]:::stAdded
  N4["Expression compilation uses rewritten literal #40;F3#41;"]:::stAdded
  N5["Query planner#47;type inference validates binary literal type #40;F11#41;"]:::stAdded
  N6["Integration queries execute with bytea literals #40;F9#44; F10#44; F12#41;"]:::stAdded
  N0 -->|"enables"| N1
  N1 -->|"feeds"| N2
  N2 -->|"supplies"| N3
  N2 -->|"supplies"| N4
  N2 -->|"supplies"| N5
  N2 -->|"supplies"| N6
  classDef stAdded fill:#dafbe1,stroke:#1a7f37,color:#1f2328,stroke-width:2px
  classDef stModified fill:#fff8c5,stroke:#9a6700,color:#1f2328,stroke-width:2px
  classDef stRemoved fill:#ffebe9,stroke:#cf222e,color:#1f2328,stroke-width:2px
  classDef stUnchanged fill:#f6f8fa,stroke:#656d76,color:#1f2328,stroke-width:1px

Loading

AI-generated · Green: added · Yellow: modified · Red: removed · Gray: existing

Diff evidence

• F1: pinot-common/src/main/codegen/config.fmpp — before · after
• F2: pinot-common/src/main/codegen/includes/parserImpls.ftl — before · after
• F3: pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java — before · after
• F4: pinot-common/src/main/java/org/apache/pinot/sql/parsers/PostgreSqlCastRewriter.java — after
• F5: pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/LiteralOnlyBrokerRequestTest.java — before · after
• F9: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/BytesMvTypeTest.java — before · after
• F10: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/BytesTypeTest.java — before · after
• F11: pinot-query-planner/src/test/java/org/apache/pinot/query/QueryCompilationTest.java — before · after
• F12: pinot-query-runtime/src/test/resources/queries/BinaryTypes.json — before · after

• Regenerate PR flow

Summary

Accept PostgreSQL hex-format bytea constants in both the infix '\x0102'::bytea form and the standard CAST('\x0102' AS BYTEA) spelling, and normalize them to Pinot's existing binary-literal representation before either the single-stage or multi-stage engine plans the query.

The SQL-standard X'0102' form is unchanged, and the two are fully interchangeable — including mixed within a single expression. The rewriter converts PostgreSQL constants to Calcite binary literals at parse time, so downstream there is no separate "PostgreSQL path": every spelling below produces the identical literal. Servers never see the new syntax, so there is no mixed-version exposure.

Supported syntax

All of these produce the identical BYTES literal 0x0102:

SELECT X'0102';                     -- SQL standard (unchanged)
SELECT x'0102';                     -- lowercase prefix
SELECT '\x0102'::bytea;             -- PostgreSQL infix cast (new)
SELECT '\x0102'::BYTEA;             -- type name is case-insensitive
SELECT CAST('\x0102' AS BYTEA);     -- standard CAST spelling (new)
SELECT CAST(X'0102' AS BYTEA);      -- spellings may be mixed

Constant formatting:

SELECT '\xdeadbeef'::bytea;         -- upper- or lowercase hex digits
SELECT '\xDE AD BE EF'::bytea;      -- ASCII whitespace between byte pairs
SELECT '\x'::bytea;                 -- empty -> zero-length BYTES

Arrays, including mixed spellings in one array:

SELECT ARRAY[X'00', X'0102'];
SELECT ARRAY['\x00'::bytea, '\x0102'::bytea];
SELECT ARRAY[X'00', '\x0102'::bytea];
SELECT ARRAY['\x00'::bytea, CAST('\x0102' AS BYTEA)];

Usable anywhere a binary literal is valid — predicates, IN lists, CASE branches, GROUP BY, and function arguments:

SELECT id FROM events WHERE payload = '\x0102'::bytea;
SELECT id FROM events WHERE payload IN ('\x01'::bytea, X'02');
SELECT CASE WHEN a = 1 THEN '\x01'::bytea ELSE X'02' END FROM events;
SELECT ARRAY['\x02'::bytea], COUNT(*) FROM events GROUP BY ARRAY['\x02'::bytea];

SELECT id, byte_values
FROM events
WHERE ARRAYS_OVERLAP(byte_values, ARRAY['\x0102'::bytea, '\xCAFE'::bytea]);

Result values remain hex strings, and an array result reports BYTES_ARRAY metadata.

Not supported

Rejected	Error	Use instead
CAST(col AS BYTEA)	BYTEA casts are supported only for quoted hex constants	CAST(col AS BYTES)
col::bytea	BYTEA casts are supported only for quoted hex constants	CAST(col AS BYTES)
1::int	:: casts are supported only for BYTEA hex constants	CAST(1 AS INT)
'0102'::bytea	must begin with \x	X'0102' or CAST('0102' AS BYTES)
'\046'::bytea	must begin with \x (legacy octal escape format)	—

Two boundaries are deliberately asymmetric:

• BYTEA accepts only quoted constants, while the existing BYTES type still accepts a dynamic per-row cast. A per-row STRING-to-BYTES conversion under BYTEA would have to be implemented separately in each engine and the two would be easy to drift apart, so it is not introduced here.
• :: is BYTEA-only. This is a scope limit, not a technical one: SqlLibraryOperators.INFIX_CAST is an ordinary SqlCastOperator of kind CAST, and intCol::double does compile to the same cast function as CAST(intCol AS DOUBLE). Supporting the full operator means committing to :: type-name semantics across both engines, which belongs in its own PR. The grammar accepts :: so this rewriter can report a clear error rather than a bare parse failure.

Only ASCII hex digits and ASCII whitespace are accepted. Character.digit and Character.isWhitespace also accept full-width and non-Latin forms, which PostgreSQL rejects.

Pre-existing conversions are unaffected: CAST('0102' AS BYTES), CAST(strCol AS BYTES), and hexToBytes(...) all behave as before.

Parsing notes

InfixCast appends the operator and its target type to the enclosing expression list as a SqlParserUtil.ToTreeListItem, so precedence is resolved by Expression2 — the same shape as the adjacent SqlAtTimeZone production and Calcite's own Babel InfixCast. This matters: collapsing the accumulated list eagerly makes :: bind everything to its left, so WHERE bytesCol = '\x01'::bytea parses as CAST(bytesCol = '\x01' AS BYTEA) and fails to compile. There is direct coverage for bytea constants as the right operand of =, <>, >, <= and inside compound predicates.

The [] and .ident postfix forms need no handling in this production; Expression2 already owns those branches.

No Thrift or protobuf schema changes.

Validation

Local runs on this commit:

• CalciteSqlParserTest, LiteralSerDeTest, RequestUtilsTest — 76 passed
• LiteralOnlyBrokerRequestTest — 10 passed (includes scalar bytea constants and constant folding on the broker's literal-only path)
• QueryCompilationTest — 241 passed
• ResourceBasedQueriesTest (includes BinaryTypes.json against H2, both optimizer variants) — 3691 passed
• BytesTypeTest, BytesMvTypeTest (real Avro array<bytes> ingestion, both query engines) — 32 passed
• spotless:apply, checkstyle:check, license:format, license:check clean on all five affected modules; no compiler warnings on added lines

testPostgreSqlByteaLiteralMatchesStandardCastSpelling pins the ::, CAST and X'...' spellings to the same value across both the query and expression parse paths.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 83.33333% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.73%. Comparing base (e6787e1) to head (4e70acf).

Files with missing lines	Patch %	Lines
...ache/pinot/sql/parsers/PostgreSqlCastRewriter.java	82.60%	2 Missing and 6 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19263      +/-   ##
============================================
+ Coverage     67.69%   67.73%   +0.03%     
  Complexity     1430     1430              
============================================
  Files          3488     3489       +1     
  Lines        224306   224354      +48     
  Branches      35416    35433      +17     
============================================
+ Hits         151850   151956     +106     
+ Misses        60447    60383      -64     
- Partials      12009    12015       +6     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	67.73% <83.33%> (+0.03%)	⬆️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	67.73% <83.33%> (+0.03%)	⬆️
unittests	67.72% <83.33%> (+0.03%)	⬆️
unittests1	57.83% <83.33%> (+0.02%)	⬆️
unittests2	39.43% <39.58%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[yashmayya]

Nice piece of work — the grammar production is a faithful port of Calcite Babel's InfixCast, the ToTreeListItem precedence handling is right, and the PR description is unusually clear about scope. I checked a few things that came back clean:

• The new :: token takes nothing away. <COLON> is used only in AddOptionalColonPath, which is gated on isColonFieldAccessAllowed() (false for every SqlConformanceEnum, including the BABEL one Pinot sets), and in JsonNameAndValue with a single colon. a::b was already a parse error.
• Both parser entry points are patched, and they are the only two in the repo, so no engine bypasses the rewrite.
• SqlShuttle and SqlBasicVisitor have no instance fields, so the static INSTANCE is safe to share, as the Javadoc says.
• normalizeHex matches PostgreSQL's hex_decode: whitespace only at even byte boundaries, ASCII digits only.
• I parsed about 50 query shapes (CTE, UNION, EXISTS, FILTER, GROUPING SETS, window, JOIN, IN, CASE, BETWEEN, SET, OPTION) and every one reconstructs correctly.

One blocker, commented inline.

Blocker: the shuttle runs over the whole statement list and downgrades Pinot's custom statement nodes. EXPLAIN IMPLEMENTATION PLAN FOR a query with a bytea literal silently returns the logical plan, and CREATE MATERIALIZED VIEW with one is typed DQL instead of DDL. Details and two suggested fixes are in the inline comment.

Non-blocking notes, all at your discretion:

1. col::bytea error names no alternative. Its two sibling messages both end with a remedy. This one leaves a PostgreSQL user at a dead end, and Pinot has what they want: hexToBytes(<expr>) converts a hex string column to BYTES per row. Worth appending.
2. compileToExpression hides these messages. The rewrite call sits inside the catch (Throwable) at line 553, which does not append e.getMessage() — unlike line 148, which does. That is why the test has to assert on getStackTrace(). Moving the call out of the try fixes it, since it already throws SqlCompilationException.
3. Two untested branches. The source instanceof SqlBinaryStringLiteral early return is the only support for X'0102'::bytea and CAST(X'0102' AS BYTEA), both advertised in the description, and neither has a test. The \x prefix check is case-sensitive on purpose (PostgreSQL is too), but nothing pins it — one '\X0102'::bytea row in invalidPostgreSqlByteaLiterals would.
4. Optional: guard the shuttle on a text scan. I measured the walk at 0.5–1.4 us against a 25–260 us parse, so about 0.5% — small. Worth knowing that compileToExpression is also called per segment on the server for JSON_MATCH and theta-sketch filters, so the multiplier there is segments x queries. sql.indexOf("::") >= 0 || StringUtils.containsIgnoreCase(sql, "BYTEA") has no false negatives and costs tens of nanoseconds.
5. One line in the description is misleading. The table lists '0102'::bytea as rejected and suggests X'0102' or CAST('0102' AS BYTES). In PostgreSQL '0102'::bytea is valid escape-format input and gives four bytes (0x30 0x31 0x30 0x32); both substitutes give two. Better to mark it unsupported with no substitute, the way '\046'::bytea already is one row below.
6. Worth a maintainer call: is :: meant to stay BYTEA-only? You note in the description that intCol::double would in fact plan and run. Accepting the token and then rejecting every other target type is a permanent public-grammar decision, so it is worth one committer confirming the limit rather than settling it inside this PR. If the limit stays, an issue link in the class Javadoc would help the next reader.

[yashmayya]

This runs the shuttle over the whole statement list, so it also walks Pinot's own statement nodes.

SqlShuttle rebuilds every ancestor of a changed node with operator.createCall(...), and that call returns the operator's own node type. SqlPhysicalExplain inherits SqlExplain.OPERATOR, whose createCall hardcodes new SqlExplain(...). SqlPinotCreateMaterializedView uses a bare SqlSpecialOperator with no createCall override, so it rebuilds as SqlBasicCall. One bytea literal anywhere inside such a statement replaces the statement node with a different class.

I ran both cases against this branch:

EXPLAIN IMPLEMENTATION PLAN FOR SELECT a FROM t WHERE b = X'01'
  -> SqlPhysicalExplain

EXPLAIN IMPLEMENTATION PLAN FOR SELECT a FROM t WHERE b = '\x01'::bytea
  -> org.apache.calcite.sql.SqlExplain

QueryEnvironment:967 picks the physical plan with explain instanceof SqlPhysicalExplain. After the downgrade that test is false, so the broker returns the logical plan. The user gets no error.

CREATE MATERIALIZED VIEW mv AS SELECT a FROM t WHERE b = X'01'
  -> SqlPinotCreateMaterializedView, sqlType=DDL

CREATE MATERIALIZED VIEW mv AS SELECT a FROM t WHERE b = '\x01'::bytea
  -> SqlBasicCall, sqlType=DQL

extractSqlNodeAndOptions matches on the node class, so the statement becomes DQL and the DDL path never runs.

CAST('\x01' AS BYTEA) reproduces both, so :: is not needed to hit this. A bytea literal in a nested subquery of the view body triggers it too.

Two ways to correct this:

1. Classify the statement first, then rewrite only the query node. For SqlExplain, rewrite operand 0 in place with setOperand.
2. Give each custom node's operator a createCall override that rebuilds the concrete class, and give SqlPhysicalExplain its own OPERATOR.

Option 1 is smaller. Option 2 also protects these nodes from the next shuttle someone adds.

Please add regression tests that assert the node class and PinotSqlType for EXPLAIN IMPLEMENTATION PLAN FOR and for CREATE MATERIALIZED VIEW with a bytea literal. The current tests cover only a plain SELECT, so both defects pass CI.