Real-window compatibility findings

These findings cover populated-query regressions and resource guards following the v1.67.0 security integration. They do not establish full LogQL parity. Each correction has a focused regression; the remaining limits below define the scope of the compatibility claim.

Timestamp defect

The exhaustive helper sent millisecond integers to Loki, which interprets integers as nanoseconds. Loki therefore queried 1970 while the proxy queried current data. RFC3339Nano timestamps now address the same populated window. A literal canary also requires both freshly ingested lines from both backends. See the Loki timestamp contract.

Isolated baseline

A fresh Compose project with Loki 3.7.7, VictoriaLogs 1.50.0 and a pre-release build of these corrections, without the UI generator, produced 283/284 query checks and 64/70 error checks. The query failure was implicit many-to-one matching. The six error failures were four invalid IP cases and a duplicated parser-error case.

The default test/e2e-compat/docker-compose.yml pins Loki 3.7.7 and VictoriaLogs v1.52.0, the same Loki release as the docker-compose.review.yml override; LOKI_IMAGE selects another Loki release.

The earlier 255/284 query result came from a long-running UI generator stack with repeated ingestion. Its reference timeouts and empty-result findings were not isolated reproductions. A later deterministic test reproduced the regexp capture/filter failure after the stored-field inventory was warmed: a query-created http_method capture was incorrectly rewritten to a stored http.method field. Query-local capture names now remain independent of that inventory.

The exhaustive checks compare status, result type and non-emptiness. Strict quantile canaries demonstrated wrong grouping and values even when those checks passed. They are insufficient proof of labels, samples or timestamps.

Corrections

• IP line filters reject invalid single addresses, prefixes, ranges and unsupported operators. Substring filters preserve literal regex metacharacters, including text resembling ip(...).

• Grouped quantiles retain by (labels) and by (), interpolate over trailing windows and include samples at the evaluation timestamp. The exact adapter uses bounded raw samples; it does not establish native quantile performance.

• Binary matching checks cardinality independently at each timestamp. Empty grouping modifiers retain their meaning, disjoint streams do not conflict, and set operations retain their many-to-many exemption. Original operands execute through the scoped query handlers, preserving trailing windows, error propagation, extraction aliases and comparison semantics.

• Additive ungrouped sums reduce all streams instead of leaking intermediate backend groups. Regexp captures remain visible as parsed fields in ordinary and categorized log responses.

• Tumbling range aggregations (range equal to step) served from VictoriaLogs stats_query_range buckets are relabelled to Loki's evaluation timestamps. VictoriaLogs labels a bucket by its start and covers [T, T+step); Loki's sample at T covers (T-range, T]. The proxy fetches from start-range and moves each bucket forward by one range, instead of trimming the bucket Loki's first sample needs. It covers rate and bytes_rate in any aggregation, and ungrouped sum(count_over_time(...)) and sum(bytes_over_time(...)).

• Sliding range aggregations (count_over_time, rate, bytes_over_time, bytes_rate with a range different from the step) served from stats_query_range buckets sum buckets of gcd(step, range) anchored to the request start and shifted by one nanosecond through the offset argument, so each bucket covers (T, T+bucket] and every window (t-range, t] is an exact union of buckets, including lines on window edges and unaligned starts. A step whose window holds no line is absent, as in Loki, for every client including Drilldown. Buckets below 1 ms, stats responses over the byte limit, and unaligned grids on a backend older than VictoriaLogs v1.45 or of undetected version use the raw-sample evaluator, which applies the same (t-range, t] boundaries to these functions for range and instant queries.

• Bare parser range aggregations without an outer aggregation (count_over_time({...} | logfmt [90s]), also regexp, pattern and json with parameters) use the same anchored gcd(step, range) buckets: stats_query_range grouped by stream for count_over_time, rate, bytes_over_time and bytes_rate with a range greater than or equal to the step (byte sums add a count() presence column, so a window holding only empty lines is a zero sample), VictoriaLogs /hits with the same offset argument for sliding count_over_time and rate when stream label fields are declared, and bucket values for sum_over_time, max_over_time and min_over_time over unwrap. Tumbling queries on this path are no longer merged into one unlabelled series. Unaligned grids on a backend without offset support use the raw evaluator. Bare | json without parameters keeps the ordered JSON evaluator.

• Binary expressions joining two vector operands evaluate both on the step-aligned grid, as Loki's query frontend does when align_queries_with_step is enabled. The compatibility stack enables it; a default Loki evaluates at start+k*step, so results with an unaligned start differ by less than one step. Operands with different ranges take different execution paths (sliding evaluation versus tumbling buckets), and an unaligned start previously left no common timestamps to join.

• unwrap duration(...) and unwrap bytes(...) over bare parsers convert unit strings such as 15ms and 1024B instead of dropping every sample.

• Range operands grouped by (level) keep Loki's level key, so on(level) matching works for range queries as it already did for instant queries.

• Scalar results carry second timestamps and all sample values use Loki's fixed-point rendering (1234000, not 1.234e+06).

• Metric queries over | regexp and | pattern keep the labels those stages extract in the series identity, as Loki does, so an existing query now returns one series per captured value instead of one per stream (a minute of one generator app: 450 series instead of 30, same total). | json and | logfmt are unchanged by default; -exact-parser-series-identity opts in.

These corrections reject previously accepted invalid queries and change incorrect numeric results. They do not imply unchanged behavior for all clients. IP validation is eager: Loki can bypass invalid pipeline construction for historical empty ranges, while the proxy rejects the invalid expression.

Ordered parser error state

VictoriaLogs JSON unpacking does not produce Loki error labels. Filtering those labels after pushdown can lose malformed lines or include lines that Loki rejects. Stage order and aggregation hints affect the result:

Pipeline inside a metric	Measured Loki behavior
JSON with surviving malformed input	Pipeline error in ordinary grouped/raw evaluation
JSON then empty-error filter	Valid parsed samples only
JSON then nonempty-error filter	Pipeline error
JSON then drop error	Valid and malformed samples accepted
JSON, nonempty-error filter, then drop error	Malformed samples accepted
Drop error before nonempty-error filter	Empty result
Drop error details only	Error state survives

Aggregation can suppress parsing or retain error labels. A blanket syntax rejection or malformed-JSON preflight does not implement this contract. The original strict canary passed 24/24 Loki checks and only 5/24 proxy checks. The ordered evaluator now preserves error state, aggregation hints, drop/keep ordering, structured-metadata collisions and Loki string decoding for eligible count/rate/byte metrics. Subsequent canaries also check both window boundaries and the proven native-aggregation optimization.

Eligibility is deliberately limited to supported bare or sum-wrapped JSON count/rate/byte pipelines. Explicit JSON extraction, mixed parsers, unwrap, compound or numeric predicates and non-sum wrappers retain their existing paths. Passing this coverage does not claim those paths have identical parser-error semantics.

Sources: Loki parser, parser hints, evaluator, VictoriaLogs JSON unpacking.

Resource and operational findings

Raw metric collectors request one extra row and reject overflow rather than returning a successful partial calculation. A final LogsQL limit pipe avoids VictoriaLogs' implicit timestamp sort from the HTTP limit argument; complete samples are sorted locally. See the VictoriaLogs query contract.

Guard	Setting	Client-visible result
Raw rows per manual metric call	-manual-range-metric-row-limit (default 1,000,000)	HTTP 502, errorType unavailable, manual range metric row limit exceeded
Metric query series	-max-stats-query-series (default 500)	HTTP 400, Loki's maximum number of series (N) reached for a single query; Logs Drilldown gets a partial result with the ... returning partial results warning

Every metric path applies the series limit the same way: raw and ordered-parser evaluators, stats and hits buckets, sliding windows, binary expressions and unions, multi-tenant merges, range and instant. Logs Drilldown's partial result holds the busiest series by total value, where Loki keeps the first series it encounters. Logs Drilldown label and field breakdowns take the same path: every series up to the limit with Loki's values, and over it the busiest series, ranked by VictoriaLogs in a bounded second call, with Loki's warning.

Binary evaluation limits nesting to 64 and child evaluations to 1,024. It shares budgets across children: 256 MiB of captured response data, two million decoded arrays, one million constructed output samples and 64 MiB of label-processing work. Individual encoded results are capped at 64 MiB. These are conservative work limits: repeated labels and nested intermediate results consume budget, so a valid large expression can now fail explicitly.

The release UI pass also found a local Compose configuration mismatch: a 1 GiB VL cap retained settings intended for the repository's 5 GiB profile. Wide Fields queries OOM-killed VL, after which the dropdown displayed no options. Restoring the repository memory/restart settings restored all seven label ranges. Independently, a broad bare-parser unwrap query exposed excessive proxy output and required resource guards on that separate path. A backend failure can cause the two ingestion targets to diverge; comparisons after that failure require a fresh paired fixture.

Remaining limits

Valid label ip() syntax and exact matching of all IPv6/non-octet CIDR/range forms need further compatibility work; argument validation does not fix approximate translation. Regexp capture collisions with existing labels and learned aliases when only an alternate stored spelling exists remain separate limits. The grouped quantile canary uses a shared aligned evaluation axis; arbitrary frontend range alignment is not established by it. Outside binary joins, native tumbling results remain on the step-aligned grid rather than Loki's unaligned start+k*step grid, and a bucket boundary is inclusive at its start where Loki's window is inclusive at its end. The same edge limit applies to sliding range aggregations on VictoriaLogs releases older than v1.45, whose stats_query_range ignores offset. Grouped tumbling count_over_time and bytes_over_time, served by the Drilldown hits and hybrid paths, are not relabelled, so their labels mark bucket starts while ungrouped sums use Loki's evaluation timestamps. Bare parser range aggregations served from buckets label each series with its stream labels (or the declared fields on the /hits route), while Loki also adds every parsed label, such as level from | logfmt; their values match Loki when parsing adds no labels. A uniform offset is applied before binary alignment, which differs from Loki only when the offset is not a multiple of the step. Wide raw-sample workloads remain subject to explicit resource limits.

The existing label sanitizer normalizes a stored __name__ label to _name. Direct-ingestion probes that require Loki's reserved metric-name identity are therefore outside the demonstrated grouping parity. Metric label_format identity rewrites also remain a separate compatibility limit. Empty without() reduction is tested independently of these unsupported input transformations; this work does not change the global label-normalization contract.

Reproduce

Use a fresh isolated Compose project for each parity shard. Do not enable the UI generator or repeatedly ingest shared fixtures into the same volumes. Wait for Loki and the proxy to report ready.

cd test/e2e-compat
docker compose down -v && docker compose up -d && ../../scripts/ci/wait_e2e_stack.sh 180
cd ../..
go test -v -tags=e2e ./test/e2e-compat -run '^TestLogQL_Exhaustive_' -count=1

Set LOKI_URL, PROXY_URL and VL_URL only when the stack is published on non-default ports.

Unique-fixture TestHardeningLive_ canaries are selected by the existing security regression CI script. Record failures separately from passes; HTTP 200 or a nonempty chart is insufficient proof of parity.