parser-and-nodes

# Parser & Node-types — operational

Companion docs: `knowledge/idioms/parser-pipeline.md`, `knowledge/idioms/node-types-and-lists.md`.

## 0. The three-tree pipeline — AST → Query → Plan

Every SQL statement flows through three distinct tree shapes; nodes
in each header live or die at a specific stage. Confuse the stages
and your patch will silently fail in a downstream walker.

```
SQL text
  │
  │  gram.y / scan.l / kwlist.h
  ▼
1. Raw parsetree (AST)         — parsenodes.h utility-stmt nodes
   Examples: SelectStmt,         + A_Expr / A_Const / ColumnRef /
   InsertStmt, A_Expr,             FuncCall (raw expression shapes)
   ColumnRef, FuncCall          → walker: raw_expression_tree_walker
                                  No types resolved, no relations
                                  looked up, no operators resolved.
  │
  │  parser/analyze.c           → transformStmt → transform<Foo>Stmt
  │  parser/parse_*.c           → adds rangetable, resolves names,
  │                               picks operator + cast OIDs,
  │                               applies coercions
  ▼
2. Query tree                  — parsenodes.h Query + analyzed
   Examples: Query, RangeTblEntry, expression nodes (primnodes.h)
   Var, OpExpr, Const, FuncExpr,  Examples: Var, OpExpr, Const,
   TargetEntry, FromExpr, JoinExpr  FuncExpr, Aggref, WindowFunc
                                → walker: expression_tree_walker
                                  + query_tree_walker
                                  + QTW_* flags for rtable/CTE descent
  │
  │  rewrite/                  → rules, views (Query → Query)
  │  optimizer/planner.c        → planner: Query → Plan
  ▼
3. Plan tree                   — plannodes.h
   Examples: SeqScan,            Examples: SeqScan, IndexScan, HashJoin,
   IndexScan, HashJoin,            Sort, Agg, Limit, Gather, GatherMerge
   Sort, Agg, Limit             → walker: plan_tree_walker
                                + planstate_tree_walker (executor-time)
                                  Shipped to parallel workers via
                                  nodeToString / stringToNode.
```

Operational rules per shape:

| Shape | Key invariants |
|---|---|
| **Raw parsetree** | No catalog lookups (parser runs without an open transaction in some paths); `A_*` and `ColumnRef` are the pre-resolution placeholders. Don't try to compute types here. |
| **Query tree** | Types, collations, operator OIDs, cast OIDs resolved. `Var` carries `varno`+`varattno` into the rangetable. Subqueries are nested `Query`s in `RangeTblEntry`. Stored in `pg_rewrite.ev_action` for views/rules — bump catversion when changing serialized fields. |
| **Plan tree** | Materialized from `Path`s by `createplan.c`. `Plan`'s expression trees use `Var`s indexed into `targetlist` slots, NOT rangetable slots. Shipped to workers via the out/read funcs — bumps to plan-node fields don't need catversion (not stored on disk) but DO need correct out/read coverage. |

The `executor-and-planner` skill covers the Plan-tree side (createplan,
add_path costing, ExecInit/ProcNode dispatch). This skill covers raw
parsetree and Query tree work, plus the node-machinery infrastructure
(`gen_node_support.pl`, copy/equal/out/read funcs).

## Tools for working with an existing tree

- `copyObject(p)` — deep copy (preserves argument type via `typeof_unqual` macro, `nodes.h:228-233`). Dispatcher `copyObjectImpl` in `copyfuncs.c:177-212`, with `check_stack_depth()` guard (`:185`). By-ref Datums go through `_copyConst` → `datumCopy`; `T_List` is deep-copied via `list_copy_deep`, `T_IntList`/`T_OidList`/`T_XidList` shallow via `list_copy`. Extensible nodes dispatch to a registered `nodeCopy` callback (`_copyExtensibleNode`). Allocated in `CurrentMemoryContext` — caller controls the lifetime.
- `equal(a, b)` — deep structural compare (`equalfuncs.c`).
- `nodeToString(p)` / `stringToNode(s)` — Lisp-ish text serialization; load-bearing for plan cache, rule storage (`pg_rewrite.ev_action`), parallel-worker plan shipping. The round-trip `copyObject(stringToNode(nodeToString(p)))` underpins the `debug_write_read_parse_plan_trees` GUC.
- `expression_tree_walker` / `_mutator` — polymorphic recursion over expression trees (post-analysis: `Var`, `OpExpr`, …). `raw_expression_tree_walker` for the pre-analysis shape (`A_Expr`, `ColumnRef`). `query_tree_walker` / `_mutator` wrap for Query, with `QTW_*` flags (`nodeFuncs.h:22-34`) to control rtable/CTE descent. `planstate_tree_walker` is the executor-time analogue. Mutator contract: return `NULL` to delete a subtree; walker contract: callback returns `true` to short-circuit, `false` to keep descending. The macro wrappers (`nodeFuncs.h:155-183`) let callbacks be typed without `-Wincompatible-pointer-types` warnings.

## When you're adding a new SQL statement

Touch four layers, in order. Build between each so you fail fast.

1. **Grammar — `src/backend/parser/gram.y`**
   - Add the keyword(s) to `unreserved_keyword` / `col_name_keyword` / etc. AND to `src/include/parser/kwlist.h` (alphabetical, with category).
   - Add a top-level rule producing your new `*Stmt` node, wire it into `stmt:` (search for `stmt:` in `gram.y`). Use `makeNode(FooStmt)` in the action, set the location with `@1`, fill fields from `$N`.
   - Match an existing simple statement as template (e.g. `DropStmt`, `VariableSetStmt`). Do NOT invent fresh helpers in gram.y if `parser/parse_utilcmd.c` already has one.

2. **Lexer — `src/backend/parser/scan.l`** *(only if you introduced a new token shape; new keywords alone do not need scan.l changes — they go through `kwlist.h`)*.

3. **Parse node — `src/include/nodes/parsenodes.h`** *(utility stmts)* or `primnodes.h` *(expressions)* or `plannodes.h` *(plan nodes)*
   - First field `NodeTag type;` (or another node struct, for "inheritance").
   - Last field for top-level statements is typically `ParseLoc location;`.
   - Annotate fields if needed: `pg_node_attr(...)` on the struct, per-field attrs like `query_jumble_ignore`, `equal_ignore`. See `nodes.h:43-125` for the full list.

4. **Analyze / execute**
   - Optimizable statement → add a `transformFooStmt(ParseState *, FooStmt *)` in `parser/analyze.c` and wire it into the `switch (nodeTag(parseTree))` inside `transformStmt` (analyze.c:334-451, switch at :368). **Caution (analyze.c:363-367)**: any change to that switch must also be reflected in `stmt_requires_parse_analysis()` (`:469-505`) and `analyze_requires_snapshot()` (`:513-529`) — three sites, one logical change.
   - Utility statement → no transform needed; just dump into a Query with `CMD_UTILITY` (the default path). Execution lives in `src/backend/commands/` via `ProcessUtility` / `standard_ProcessUtility` (`src/backend/tcop/utility.c`).

5. Run `gen_node_support.pl` (done automatically by the build) and verify the generated `copyfuncs.funcs.c`, `equalfuncs.funcs.c`, `outfuncs.funcs.c`, `readfuncs.funcs.c` chunks for your new struct look sane.

6. Add a regression test under `src/test/regress/sql/` (and a matching expected file). See `.claude/skills/testing/SKILL.md`.

## When you're adding a new Node type (no SQL change)

From `src/backend/nodes/README` "Steps to Add a Node":

1. Put the struct in the right header. The file must be in `gen_node_support.pl`'s `@all_input_files` list (parsenodes.h, primnodes.h, plannodes.h, pathnodes.h, execnodes.h, value.h, etc. — see `gen_node_support.pl:53-77`). The `T_Foo` tag is generated automatically into `nodes/nodetags.h` (do NOT hand-edit it).
2. First field is `NodeTag type;` — unless you "inherit" by embedding another node struct as the first field (e.g. `Plan plan;` in a new plan node).
3. Build. Inspect the generated `copyfuncs.funcs.c` / `equalfuncs.funcs.c` / `outfuncs.funcs.c` / `readfuncs.funcs.c` / `queryjumblefuncs.funcs.c` entries for your node. Add `pg_node_attr(...)` and per-field attrs to fix anything wrong.
4. If a field needs special treatment, common annotations:
   - `pg_node_attr(custom_copy_equal)` — write your own bodies in `copyfuncs.c` / `equalfuncs.c` and the generator will skip yours.
   - `pg_node_attr(no_copy_equal, no_read, no_query_jumble)` — opt out entirely (executor state nodes typically use `nodetag_only`).
   - Per-field: `array_size(otherfield)`, `copy_as(VALUE)`, `equal_ignore`, `read_write_ignore`, `query_jumble_ignore`, `query_jumble_location`.
5. If your node is an **expression** (lives in primnodes.h or has child Expr fields), add cases to **every** giant switch in `nodeFuncs.c`:
   - `exprType` (`:42+`), `exprTypmod` (`:304+`), `exprCollation` / `exprSetCollation` / `exprInputCollation` (`:826+`, `:1140+`, `:1092+`), `exprLocation` (`:1403+`) — type/typmod/collation/location introspection.
   - `expression_tree_walker_impl` (`:2111+`), `expression_tree_mutator_impl` (`:3018+`) — recursion into children.
   - `raw_expression_tree_walker_impl` (`:4115+`) — only if the node can appear in raw parsetrees.
   - `set_opfuncid` family (`:1890+`) — only if your node holds an operator OID needing resolution.
   Grep `T_<SiblingNode>` (e.g. `T_OpExpr`) to find every site; five-to-eight maintenance points per node, miss one and walkers silently misbehave.
6. **Adding a node type renumbers existing tags.** Recompile the whole tree (`--enable-depend` helps). No initdb needed — node numbers never go to disk. BUT if the node can appear in a *stored* parse tree (rule actions, view definitions), bump `CATALOG_VERSION_NO` in `src/include/catalog/catversion.h`.
7. Test with `debug_copy_parse_plan_trees=on`, `debug_write_read_parse_plan_trees=on`, `debug_raw_expression_coverage_test=on` (set via `PG_TEST_INITDB_EXTRA_OPTS`).

## When you're modifying an existing Node type

- **Adding a field** to a node that appears in stored catalog parse trees (e.g. `Query`, anything in views / rules) → bump catversion. Add a `pg_node_attr` if the new field needs special read/write handling (e.g. `read_as(...)` for backward-compat when reading old serialized trees from extensions — usually not needed because catversion changes invalidate stored trees).
- **Removing or reordering fields** → also catversion bump if stored.
- **Changing only Plan/Path internal fields** (never serialized to catalogs) → no catversion bump needed. But Plan nodes ARE serialized to parallel workers, so you still need correct out/read funcs.

## Pre-submit smoke

```
cd dev/build-debug && ninja && \
  meson test --suite regress -q
```

Plus the debug-tree GUCs above on a separate initdb if you touched anything tree-shape related.

## Node-type reference table

The major node families and where each lives. When in doubt about
which header your new field belongs in, check this table.

| Header | Shape | Examples | When you'd add here |
|---|---|---|---|
| `nodes/parsenodes.h` | Raw + Query | `SelectStmt`, `InsertStmt`, `A_Expr`, `ColumnRef`, `Query`, `RangeTblEntry`, `TargetEntry`, `FromExpr`, `JoinExpr` | New SQL statement; new fields on `Query` (catversion bump if serialized in `pg_rewrite`) |
| `nodes/primnodes.h` | Expression (analyzed) | `Var`, `Const`, `OpExpr`, `FuncExpr`, `Aggref`, `WindowFunc`, `SubLink`, `CaseExpr`, `CoalesceExpr` | New expression node (must touch every `nodeFuncs.c` switch — see §"new Node type") |
| `nodes/plannodes.h` | Plan | `Plan`, `SeqScan`, `IndexScan`, `HashJoin`, `Sort`, `Agg`, `Limit`, `Gather`, `Append`, `MergeAppend` | New plan node (also wire executor under `nodeXxx.c` — see `executor-and-planner`) |
| `nodes/pathnodes.h` | Path + RelOptInfo | `Path`, `IndexPath`, `JoinPath`, `RelOptInfo`, `PlannerInfo` | New path shape considered by the optimizer |
| `nodes/execnodes.h` | PlanState (runtime) | `PlanState`, `SeqScanState`, `EState`, `ExprContext`, `ProjectionInfo` | New runtime state for an executor node |
| `nodes/value.h` | Leaf values | `Integer`, `Float`, `Boolean`, `String`, `BitString` | Almost never — the four primitive value carriers |
| `nodes/pg_list.h` | Lists | `List`, `IntList`, `OidList`, `XidList` | Almost never (collection infrastructure) |
| `nodes/extensible.h` | Extension nodes | `ExtensibleNode`, `CustomScan` | Custom AM / custom planner extension |

`nodes/nodetags.h` is auto-generated by `gen_node_support.pl`; **never
hand-edit it.** A new struct in any of the headers above (with the
file listed in `gen_node_support.pl @all_input_files`) gets its `T_Foo`
tag automatically.

## Walker / mutator quick reference

| Function | Where | Use for |
|---|---|---|
| `raw_expression_tree_walker` | `nodes/nodeFuncs.c` | Pre-analysis (`A_Expr`, `ColumnRef`, `FuncCall`); used by `parser/parse_*.c` |
| `expression_tree_walker` | `nodes/nodeFuncs.c` | Analyzed expressions in a Query tree |
| `expression_tree_mutator` | `nodes/nodeFuncs.c` | Same shape, but returns new nodes; rewriter / planner use this |
| `query_tree_walker` | `nodes/nodeFuncs.c` | Wraps `expression_tree_walker` to also walk Query subexpressions; controlled by `QTW_*` flags |
| `query_tree_mutator` | `nodes/nodeFuncs.c` | Mutator counterpart |
| `range_table_walker` / `_mutator` | `nodes/nodeFuncs.c` | Just the rangetable, with per-RTE field control flags |
| `plan_tree_walker` | `nodes/nodeFuncs.c` | Plan-tree (`plannodes.h`) recursion |
| `planstate_tree_walker` | `executor/execProcnode.c` | Executor-time (`PlanState`) recursion; used by EXPLAIN |

Conventions to remember:

- **Walker callback** returns `bool`. Return `true` to short-circuit
  the whole walk; return `false` to continue descending.
- **Mutator callback** returns `Node *`. Return `NULL` to delete a
  subtree; return the input pointer to keep it unchanged; return a
  new node to replace.
- **`QTW_*` flags** (`nodeFuncs.h:22-34`) control rtable / CTE
  descent. The default skips RTE subqueries — opt in with
  `QTW_EXAMINE_RTES_BEFORE` / `_AFTER` when you actually want them.
- The typed macro wrappers (`nodeFuncs.h:155-183`) let your
  callback be typed (`bool foo_walker(Node *n, MyCtx *ctx)`) without
  `-Wincompatible-pointer-types` warnings.

## Files-examined rows

| file | depth | produced |
| --- | --- | --- |
| `src/backend/parser/README` | full | SKILL §"new SQL statement" |
| `src/backend/parser/gram.y` lines 13625-13706 | scanned + 1 rule | SKILL §grammar |
| `src/backend/parser/scan.l` 1-40 | top | SKILL §lexer |
| `src/backend/parser/parse_node.c` 1-50 | top | SKILL §analyze |
| `src/backend/parser/analyze.c` 1-80, 334-433 | transformStmt full switch | SKILL §analyze |
| `src/backend/nodes/README` | full | SKILL §"new Node" |
| `src/include/nodes/nodes.h` | full | SKILL §annotations |
| `src/include/nodes/pg_list.h` 1-120, 380-485 | List API + foreach | idioms |
| `src/include/nodes/value.h` | full | idioms |
| `src/backend/nodes/gen_node_support.pl` 1-140 | top + file list | SKILL §generator |

## Cross-references

- `.claude/skills/executor-and-planner/SKILL.md` — Plan-tree side of the three-tree pipeline; `createplan.c`, `ExecInitNode`/`ExecProcNode` for new plan nodes.
- `.claude/skills/catalog-conventions/SKILL.md` — `CATALOG_VERSION_NO` bump rules when serialized parse-tree fields change.
- `.claude/skills/testing/SKILL.md` — `debug_copy_parse_plan_trees`, `debug_write_read_parse_plan_trees`, `debug_raw_expression_coverage_test` GUCs for tree-shape regressions.
- `.claude/skills/coding-style/SKILL.md` — node-header style, `pg_node_attr` placement, `T_Foo` enum convention.
- `.claude/skills/fmgr-and-spi/SKILL.md` — function-call node (`FuncExpr`) interaction with fmgr; SPI's parse-tree consumption.
- `knowledge/idioms/parser-pipeline.md` — long-form pipeline narrative.
- `knowledge/idioms/node-types-and-lists.md` — `List` / `foreach` patterns + node-tag invariants.
- `source/src/backend/nodes/README` — canonical "Steps to Add a Node" reference.
- `source/src/backend/parser/README` — gram.y conventions, shift/reduce conflict resolution.