Skip to main content

LogsQL Typed AST — Architecture Reference

Overview​

internal/logsql provides a typed AST, recursive-descent parser, capability-aware builder, and version gating for VictoriaLogs LogsQL queries. It mirrors the structure of internal/logql (the existing LogQL parser) and targets VictoriaLogs v1.40 and later.

The package is used in two modes:

  1. Direct construction — callers assemble logsql.Expr nodes with the builder API and call String() to emit the final query.
  2. Parse + re-emit — callers parse an existing LogsQL string into an AST, inspect or transform nodes, and re-emit via String().

Package Structure​

FileResponsibility
ast.goAll typed AST nodes — interfaces, filter nodes, pipe stages, stats functions
scanner.goLexer — tokenises LogsQL syntax
parser.goRecursive-descent parser — Parse(string) (*Query, error), ParseFilter(string) (FilterExpr, error)
capabilities.goCapabilities struct, CapabilitiesFor(semver string) Capabilities
builder.goBuilder API — NewQuery(), NewBuilder(caps), constructor helpers, BestTopN(), BestIPv4Range()

AST Node Hierarchy​

Parser Flow​

Query Construction Flow​

VictoriaLogs Version Capability Matrix​

The package targets v1.40 minimum. All features in the v1.40 baseline are always available.

Version rangeFeature additions
v1.40–v1.43Baseline: PipeHits, PipeRunning, PipeBlock, PipeUniq, PipeTop, StatsHistogram
v1.44rate_sum() stats function (Capabilities.StatsRateSum)
v1.45–v1.48ipv4_range() field filter (Capabilities.FieldIPv4Range)
v1.49Metadata substring filter (Capabilities.MetadataSubstring)
v1.50+Dense pattern windowing (Capabilities.DensePatternWindowing)

CapabilitiesFor(semver) is called once at proxy startup from storeBackendVersion(). Unsupported versions (pre-v1.40, malformed) return a zero Capabilities (all false), which is the safest possible degraded state.

FieldOp Reference​

ConstantLogsQL syntaxVL version
FieldOpExactfield:="value"v1.40+
FieldOpRegexpfield:~"pattern"v1.40+
FieldOpPrefixfield:prefix*v1.40+
FieldOpSubstringfield:*sub*v1.40+
FieldOpEmptyfield:""v1.40+
FieldOpAnyfield:*v1.40+
FieldOpGTfield:>valv1.40+
FieldOpGTEfield:>=valv1.40+
FieldOpLTfield:<valv1.40+
FieldOpLTEfield:<=valv1.40+
FieldOpRangefield:range(min,max)v1.40+
FieldOpInfield:in(a,b,c)v1.40+
FieldOpIPv4Rangefield:ipv4_range(first,last)v1.45+

Integration Path​

Typed AST-to-AST translator (internal/logql/translate.go)​

Translate(expr Expr, opts TranslateOptions) (string, error) is a second, typed translation path that maps LogQL AST nodes directly to internal/logsql types — no String() roundtrip through the LogQL printer.

TranslateOptions carries three fields:

  • LabelFn func(string) string — optional label-name rewriter (e.g. OTel dotted→underscore)
  • StreamFields []string — fields declared as stream labels; routed to FilterExpr instead of pipes
  • Caps *logsql.Capabilities — gates VL version-specific constructs

Node mapping:

LogQL nodeLogsQL output
*LogQueryFilterExpr + []Pipe via stream selector + pipeline stages
*StreamSelectorExprlogsql.FilterExpr (label matchers)
*LineFilterStagelogsql.PipeFilter
*LabelFilterStagelogsql.PipeFilter (label conditions)
*JSONParserStagelogsql.PipeUnpackJSON
*LogfmtParserStagelogsql.PipeUnpackLogfmt
*RegexpParserStagelogsql.PipeExtractRegexp
*DropStage (bare labels)logsql.PipeDelete
*KeepStage (bare labels)logsql.PipeKeep
*LineFormatStage (simple)logsql.PipeFormat
*RangeAggregation, *VectorAggregation, *BinOpExpr, *OpaqueMetricExprerrFallthrough → delegate to existing string translator

errFallthrough is a package-private sentinel error. When returned, callers fall back to the existing translator.TranslateLogQLWithCapabilities string path unchanged — metric nodes are not yet handled by the typed path and route to the existing Tier 1/2 translators.

The proxy's main translation path (translator.TranslateLogQLWithCapabilities) is unchanged. logql.Translate is wired but not yet called by handlers; handler migration is planned for a follow-on PR.

Builder integration path​

Once the AST-to-AST path is wired to handlers, the full pipeline becomes:

  1. logql.Translate(expr, opts) maps LogQL AST → logsql AST
  2. Builder's Build* methods return logsql.FilterExpr or []logsql.Pipe
  3. logsql.NewQuery(filter, pipes...) assembles the full AST
  4. q.String() emits the final LogsQL query at the HTTP boundary

Test Coverage Summary​

FileTestsCoverage focus
ast_test.go84All String() methods
scanner_test.go12Token stream, quoted strings, raw strings
capabilities_test.go13Version matrix, edge cases
builder_test.go10Direct construction, BestTopN, BestIPv4Range
parser_test.go38Round-trip (25 cases), ParseFilter, error cases, stats funcs
edge_cases_test.go~110All FieldOps, nesting, boundary versions, malformed semver
fuzz_test.go4 fuzzNever panics on arbitrary input
translate_test.go17Translate(): stream selectors, line/label filters, parsers, drop/keep/lineformat, errFallthrough for metric nodes
Total299

Design Notes​

internal/logsql is built in three layers:

  1. Typed AST (ast.go) — 30+ node types whose String() methods emit valid LogsQL
  2. Parser (scanner.go + parser.go) — recursive-descent parser that round-trips what the translator emits
  3. Capability-aware builder (capabilities.go + builder.go) — selects the best LogsQL construct for the detected VL version

Callers import the package and construct queries using the builder API. The minimum supported VictoriaLogs version is v1.40.