promql-compliance-tester fails on victoriametrics

[monotek]

I've just tried to run the promql compatibility test with victoriametrics 1.83.0 but run into the following error.

Tried thanos and prometheus as reference config.
Both fail.

Result is also non deterministic as the error alwasy fails at a different step. See:

/ # ./promql-compliance-tester -config-file vm.yaml -config-file ./promql-test-queries.yml -output-passing
403 / 548 [------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->___________________________________________________________________] 73.54% 307 p/sFATA[0001] Error running comparison: expected reference API query "label_replace(demo_num_cpus, \"instance\", \"\", \"\", \"\")" to fail, but succeeded  source="main.go:137"

/ # ./promql-compliance-tester -config-file vm.yaml -config-file ./promql-test-queries.yml -output-passing
385 / 548 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->___________________________________________________________________________] 70.26% 363 p/sFATA[0001] Error running comparison: expected reference API query "label_replace(demo_num_cpus, \"instance\", \"\", \"\", \"\")" to fail, but succeeded  source="main.go:137"

/ # ./promql-compliance-tester -config-file vm.yaml -config-file ./promql-test-queries.yml -output-passing
445 / 548 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->_______________________________________________] 81.20% 267 p/sFATA[0001] Error running comparison: expected reference API query "label_replace(demo_num_cpus, \"instance\", \"\", \"\", \"\")" to fail, but succeeded  source="main.go:137"

/ # ./promql-compliance-tester -config-file vm.yaml -config-file ./promql-test-queries.yml -output-passing
431 / 548 [-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->______________________________________________________] 78.65% 348 p/sFATA[0001] Error running comparison: expected reference API query "label_replace(demo_num_cpus, \"instance\", \"\", \"\", \"\")" to fail, but succeeded  source="main.go:137"

[anish-katukam]

Were you ever able to find a resolution to this? Attempting to run the tool and encountering the same issues.

[monotek]

Nope. But we use VictoriaMetrics now since i tried it and we could not find any incompatablities while doing so.

[jessechencx]

anyone resolved it?