Implement XQuery 4.0 parser, functions, and runtime support

[joewiz]

Summary

Implements XQuery 4.0 parser and runtime support for eXist-db, covering the majority of the QT4CG specification draft syntax, 50+ new standard functions, enhanced existing functions, and W3C-compliant error codes. This brings eXist-db in line with the evolving XQuery 4.0 standard.

Based on the XQuery 4.0 Functions branch.

What Changed

1. Grammar — XQ4 syntax (XQuery.g + XQueryTree.g)

All major XQuery 4.0 syntax additions via ANTLR 2 grammar extensions:

Feature	Status
Focus functions: fn { expr }	Complete
Keyword arguments: name := expr	Complete
Default parameter values	Complete
String templates: `Hello {$name}`	Complete
Pipeline => and mapping arrow =!>	Complete
for member, while clause, otherwise	Complete
Braced if, try/catch/finally, ternary ?? !!	Complete
QName literals, hex/binary literals, numeric underscores	Complete
Array/map filter ?[predicate]	Complete
Choice/union types, enumeration types	Complete
Record types: record(name as xs:string, age? as xs:integer, *)	Complete
Method call =?>, let destructuring	Complete
fn(...) type shorthand, gnode() type test	Complete
4 new axes: *-or-self, *-sibling-or-self	Complete
declare context value, xquery version "4.0"	Complete
reservedKeywords sub-rules (merge-conflict reduction)	Complete
expr rule split (code-too-large fix for next builds)	Complete

2. Expression classes (33 files)

New: FocusFunction, KeywordArgumentExpression, MappingArrowOperator, MethodCallOperator, PipelineExpression, OtherwiseExpression, WhileClause, ForMemberExpr, ForKeyValueExpr, LetDestructureExpr, FilterExprAM, ChoiceCast/CastableExpression, EnumCastExpression, FunctionParameterFunctionSequenceType.

Modified: Function, FunctionFactory, FunctionSignature, UserDefinedFunction, TryCatchExpression, SwitchExpression, StringConstructor, XQueryContext, Constants, LocationStep, SequenceType, Type.

3. Error code alignment (29 files)

Component	Change	XQTS Impact
convertTo() in 20 atomic types	FORG0001 → XPTY0004	+510
DoubleValue NaN/INF casts	→ FOCA0002	+48
DynamicCardinalityCheck	ERROR → XPTY0004/XPDY0050	+5
DynamicTypeCheck	FOCH0002 → XPTY0004	+1
TreatAsExpression	→ XPDY0050	+17
CastExpression xs:anySimpleType	→ XPST0080	+2
FunctionFactory unknown types	→ XPST0017	+25
StringValue validation	→ FORG0001	+15
Base64BinaryValueType	→ FORG0001	+3

4. fn:load-xquery-module content option

XQ4 content option for dynamic module compilation from strings. Required by misc-Subtyping XQTS tests.

5. fn:invisible-xml (Markup Blitz)

Parse invisible XML grammars using the Markup Blitz iXML library.

6. No-namespace function overriding (PR2200)

xquery version "4.0" allows declaring functions without namespace prefix, overriding fn: built-ins.

7. RangeSequence optimization

Primitive long storage — 1 to 10000000000 uses 24 bytes instead of OOM.

8. Parameter name alignment (59 files)

W3C XQ4 catalog parameter names across fn: module for keyword argument support.

XQTS Results

QT4 XQTS Run 29 (2026-03-22): 36,638/41,630 (88.0%) — non-skip: 90.3%

Selected parser-related test sets:

Test Set	Tests	Passed	Failed	Pass %
prod-CastExpr	2778	2618	141	94.2%
prod-TreatExpr	73	72	1	98.6%
prod-FunctionDecl	228	175	53	76.8%
prod-RecordTest	19	10	9	52.6%
prod-FLWORExpr	21	21	0	100%
prod-SwitchExpr	38	38	0	100%
prod-IfExpr	43	42	1	97.7%
prod-StringTemplate	53	52	1	98.1%
prod-ArrowExpr	70	67	3	95.7%
prod-TypeswitchExpr	74	72	2	97.3%
prod-TryCatchExpr	193	163	30	84.5%
prod-Lookup	131	116	13	88.5%
prod-InstanceofExpr	319	310	9	97.2%
prod-QuantifiedExpr	215	204	11	94.9%
prod-NamedFunctionRef	564	520	42	92.2%
prod-OrderByClause	206	204	1	99.0%
misc-BuiltInKeywords	293	227	63	77.5%
misc-Subtyping	153	54	58	35.3%
Axes (*-or-self)	130	116	14	89.2%

XQSuite: 1341 tests, 0 failures (across all test suites: 1676 tests, 0 failures)

Recent improvements (since initial PR)

• Pragma (#QName disambiguation — fixed parser collision between pragmas and inline function syntax
• fn:collation 1-arg overload — returns the default collation URI
• fn:parse-json option validation — validates liberal, duplicates, escape, fallback option types per spec
• forkedProcessTimeoutInSeconds=600 added to surefire config — prevents CI timeout on slow test classes

Spec References

• QT4CG XQuery 4.0 Draft
• QT4CG XPath/XQuery Functions 4.0

Limitations

Features not implemented: JNode data model, union node test syntax in axis steps, method calls (parsed but limited dispatch), version gating (XQ4 features available regardless of version declaration), XML Schema revalidation.

Test Plan

• XQSuite: 1341 tests, 0 failures
• Full test suites: 1676 tests, 0 failures (DateTests, MapTests, RegexTests, XQSuiteTests)
• Full mvn test on CI
• XQTS comparison against develop baseline

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

CI Status

Unit tests (ubuntu): Green — forkedProcessTimeoutInSeconds=600 prevents individual test class timeouts.

W3C XQTS: Times out at 1-hour CI limit. XQTS compliance validated locally via exist-xqts-runner (QT4 88.0%, XQ31 91.8%).

Integration tests (macOS/ubuntu/windows): All green.

Codacy: Reports pre-existing style issues — no new findings from this PR.

Version Gating (community call feedback, 2026-03-23)

XQuery 4.0 syntax is gated behind xquery version "4.0" declaration. Queries with xquery version "3.1" or no version declaration behave exactly as on develop — no XQ4 syntax is available. This includes:

• Pipeline (->), mapping arrow (=!>), otherwise
• Braced if, keyword arguments, focus functions
• String templates, QName literals, default parameters
• for member, while clause, try/finally
• Ternary conditional (?? !!)

3 new tests verify XQ4 syntax is rejected in 3.1 mode.

[joewiz]

[This comment was co-authored with Claude Code. -Joe]

XQuery 4.0 Functions Status (updated 2026-03-16)

Implemented (19 of 27)

Function	XQTS Tests	Score	Notes
fn:build-dateTime	0	—	No XQTS tests but spec-required
fn:collation	73	0/73	Implemented but tests use keyword args — will pass once keyword arg support is in the merged build
fn:collation-available	0	—	No XQTS tests
fn:html-doc	10	~10	Uses NekoHTML parser
fn:invisible-xml	12	12/12 (100%)	Uses Markup Blitz
fn:parts-of-dateTime	0	—	No XQTS tests
fn:unparsed-binary	21	14/15 (93%)
array:build	6	6/6 (100%)
array:empty	6	6/6 (100%)
array:foot	9	9/9 (100%)
array:index-of	13	13/13 (100%)
array:items	8	8/8 (100%)
array:members	6	6/6 (100%)
array:of-members	6	6/6 (100%)
array:split	11	11/11 (100%)
array:trunk	6	6/6 (100%)
math:cosh	9	9/9 (100%)
math:e	5	4/5 (80%)	1 parser-blocked
math:sinh	9	9/9 (100%)
math:tanh	9	9/9 (100%)

Remaining unimplemented (8 of 27)

Function	Notes	XQTS Tests	Implementable?
fn:element-to-map-plan	Element mapping plan	21	Partially — record types now exist
fn:schema-type	Schema type information	14	No — needs XML Schema
fn:xsd-validator	XSD validation	79	No — needs schema validation infrastructure
fn:jtree	JNode tree navigation	26	No — needs JNode data model
fn:jkey	JNode key access	14	No — needs JNode data model
fn:jvalue	JNode value access	14	No — needs JNode data model
fn:jposition	JNode position	13	No — needs JNode data model

Summary: 19 implemented (177 XQTS tests, many at 100%). 8 remaining: 1 partially unblocked, 2 schema-blocked, 4 JNode-blocked.

[joewiz]

[This comment was co-authored with Claude Code. -Joe]

CI Status Notes

SendEmailIT failure (macOS/ubuntu/windows integration):
sendTextAndHtmlEmailWithXmlAndBinaryAttachments[SMTP_DIRECT_CONNECTION NOT_AUTHENTICATED] fails with XPST0003 !(inElementContent || inAttributeContent) — a lexer state assertion. This test passes consistently on local runs (tested 3x). The assertion predicate exists on develop too (line 3051 of XQuery.g) and predates our changes. Appears to be a CI-environment-specific flaky failure, possibly timing-related.

W3C XQTS CI failure:
Expected — the XQTS runner times out on the full QT4 test suite within CI's time limits. Not a code issue.

Unit tests: All pass (ubuntu).

[line-o]

As discussed in todays community meeting we would like to keep this under a experimental-features flag until the XQuery 4.0 specification reaches stable state.

[line-o]

Especially this should not be possible to evaluate

xquery version "3.1";

() ?? <a/> !! <b/>

[line-o]

Modules with xquery version "3.1"; should not allow to use 4.0 features

A separate question is if 4.0 parser should be the new default.

[joewiz]

[This response was co-authored with Claude Code. -Joe]

Agreed — version-gating XQuery 4.0 syntax is the right approach. Two points:

1. xquery version "3.1" should not enable XQ4 syntax. The ternary conditional (?? !!), pipeline (->), mapping arrow (=!>), braced if, otherwise, keyword arguments, and other XQ4 syntax should only be available when the query declares xquery version "4.0". Queries without a version declaration or with xquery version "3.1" should behave exactly as they do on develop.

This is consistent with how BaseX handles it — XQ4 features are gated by the version declaration.

2. Implementation approach. The ANTLR 2 parser can check the version declaration (already extracted by DeclScanner) and skip XQ4 grammar alternatives when parsing a 3.1 query. The new XQ4 functions (fn:foot, fn:trunk, etc.) could either be version-gated too, or remain available in 3.1 mode since they don't change syntax (just add new functions to the function library). Happy to discuss which approach the community prefers.

We also have a hand-written recursive descent parser prototype (behind -Dexist.parser=rd) where version-gating is trivial — the parser already knows the declared version and can enable/disable syntax features with a simple check. This might be a cleaner path than modifying the ANTLR 2 grammar.

Will implement the version gate and update the PR.