Cascade

A code-to-column change-impact knowledge graph for AI coding agents.

View the Project on GitHub alexsoft-hq/Cascade

SQL lane setup

The SQL lane is the half of the engine that reads your database side. It turns a DDL file (or a fetched catalog snapshot) into tables and columns, MyBatis XML into statements, and each statement into the columns it reads and writes. That gives you mapper → statement → table → column.

It needs Python and one library. It never opens a database connection while analyzing, so you can run it on a laptop with no access to the real server.

For the other half, endpoint → controller → service → mapper, see java-lane.md. The Java lane is a separate, JDK-only, parse-only lane, and you do not need it to get value out of this one.

1. Python and sqlglot

What Pin Why
Python 3.12.x the reference stack, and what sqlglot is tested against here
an interpreter with sqlglot CASCADE_PYTHON, else .venv/ at the repo root, else <cascade home>/venv a run tries those three in that order, and cascade doctor prints which one it found
sqlglot sqlglot==30.17.0 the SQL parser, dialect by dialect, for Oracle, MySQL and PostgreSQL
node bin/cascade.mjs setup       # builds the interpreter and installs the pinned sqlglot
node bin/cascade.mjs doctor      # tells you what is still missing, and the fix

setup builds <engine>/.venv inside a checkout of this repository and <cascade home>/venv anywhere else, because an installed copy lives in a directory the next upgrade replaces. --force rebuilds one; --home puts it in the tool home even inside a checkout. setup repairs a managed environment whose parser is missing, broken or at a different version; --force also reinstalls a matching environment. It verifies a replacement before discarding the backup, restores the previous environment on failure, and preserves data outside the venv. See installation and recovery.

Doing it by hand still works:

python3 -m venv .venv
.venv/bin/pip install -r adapters/sql/requirements.txt

On Windows the interpreter lands in .venv\Scripts\python.exe, and that is where a run looks for it there; the same two lines read py -m venv .venv and .venv\Scripts\python.exe -m pip install -r adapters\sql\requirements.txt. CASCADE_PYTHON names another interpreter on any platform. Both setup and doctor require its exact pinned sqlglot version. setup treats this external interpreter as read-only, including with --force: install the reported pin yourself using that interpreter’s -m pip, or unset CASCADE_PYTHON and let setup manage an environment.

The version is pinned in adapters/sql/requirements.txt, not chosen at install time. sqlglot ships often, and a different version parses some statements differently, which would move the pack digest. The digest has to be reproducible, so the pin comes first and the upgrade is its own change: bump the file, run the suite, and read the diff.

Live-catalog drivers (PyMySQL, psycopg, oracledb) are listed in the same file but commented out. Install one only if you actually run cascade catalog fetch.

2. Dialect, and what counts as the same name

profile.sqlDialects.main names your database. Two separate things follow from it: which parser sqlglot uses, and the rule that decides when two spellings mean one object. The second is a property of the database rather than of the parser, so we declare it here instead of inheriting whatever the parser happens to do.

sqlDialects.main sqlglot parser identifier case where the rule comes from
mysql, mariadb mysql fold-lower MySQL always compares column, index, trigger and event names case-insensitively. Table and database names follow the server’s lower_case_table_names (0 stores and compares as written, 1 lower-cases, 2 compares insensitively). A DDL file does not carry that setting, so the lane takes the portable value the MySQL manual recommends for cross-platform schemas.
postgres, postgresql postgres fold-lower PostgreSQL folds unquoted identifiers to lower case (manual §4.1.1).
oracle, oracle-11g, oracle-19c oracle fold-upper Oracle stores and compares unquoted identifiers in upper case.
hsqldb sqlglot’s default (ANSI) parser fold-upper HSQLDB follows the SQL standard, where unquoted identifiers fold to upper case. sqlglot 30.17.0 ships no HSQLDB parser, so we use the standard-SQL one and say so. The case rule still comes from HSQLDB.
h2 sqlglot’s default (ANSI) parser fold-upper H2’s classic default (DATABASE_TO_UPPER=TRUE) folds unquoted identifiers to upper case. Same parser note as hsqldb.
tibero, altibase, goldilocks oracle exact three databases the Korean market runs and sqlglot has never heard of. Each was built to be Oracle-compatible, so the Oracle grammar is the honest reading of their SQL. The identifier rule is not inherited with it: this engine has no citation for how any of the three folds an unquoted name, so it fails closed on exact and says so. Declare sqlIdentifierCase if you know your deployment’s rule.
cubrid mysql exact the same, for the database that was built MySQL-compatible.
anything else     refused. A dialect we cannot route is an error rather than a quiet fall back to MySQL: a wrong dialect mis-parses every statement, and the result would still look fine.

Two rules apply under every row.

profile.sqlIdentifierCase overrides the table with fold-lower, fold-upper or exact. The default, null, takes the dialect’s own rule. Use exact for a case-sensitive MySQL deployment (lower_case_table_names=0) whose schema really does tell Item from ITEM. Changing the key changes what matches, so it invalidates the cached lineage of every statement. Every key a profile can hold, with an example of each, is listed in concepts.md.

If two catalog names fold onto one key, the run prints a folded_identifier_collision warning naming both, the first declaration keeps the key, and both tables stay in the pack. Nothing is merged in silence.

What an ALTER leaves unsaid is the database’s

The DDL reader (catalog_ddl.py) parses the files with the same grammar the statement lane uses, the standard grammar for H2 and HSQLDB included (--dialect); for H2 and HSQLDB a CREATE TABLE that grammar cannot read is read by one of their compatibility modes’ grammars (see below). It is told which database the files are for (--database, the database sqlDialects.main names, when that is not the grammar’s own name), and, when sqlDialects.main is not declared, that the database is only assumed (--database-assumed). Each of these is part of the cached catalog’s key. The second matters because a migration’s ALTER depends on rules the file does not write, and those rules belong to the database, not to the grammar it was parsed with. DROP CONSTRAINT x drops the primary key only if x is the key’s name, and the CREATE often never gave it one.

Rule PostgreSQL MySQL, MariaDB Oracle
a key’s name when the DDL gives none <table>_pkey, the server’s convention, so a drop matched through it says so; a name past 63 bytes is not known always PRIMARY, whatever the DDL wrote not known: the server numbers it (SYS_C...)
dropping one column of a key drops the whole key the column leaves the key, the rest stays drops the whole key
ALTER INDEX a RENAME TO b renames the key a is the index of    
MODIFY c ...   restates the whole column, so what it leaves out (NOT NULL, a comment) is gone changes only what it says

With no database declared, the MySQL column applies, as an assumption: each conclusion that rests on one of its rules (a key’s name is PRIMARY, one column dropped from a key of several, what a MODIFY leaves out) is said as alter_rule_assumed, whose sentence ends by asking to declare sqlDialects.main if the files are for another database. The overview counts them in its catalog-rules-assumed gap, the catalog axis is degraded with the cause rules-assumed, and the fix it names is to declare sqlDialects.

A blank cell is a rule this reader does not hold for that database. A database with no rules of its own here (H2, HSQLDB, CUBRID, Tibero, Altibase, Goldilocks) borrows none from the grammar it is parsed with: speaking MySQL’s or Oracle’s SQL does not show that a database follows MySQL’s or Oracle’s other rules. Wherever a rule is missing, a key the DDL did not name has no known name, and a statement whose effect hangs on the missing rule keeps what the catalog had and says so: alter_primary_key_unknown (a drop or a rename that may hit the key, one column dropped from a key of several) and alter_modify_unsaid_unknown (a MODIFY that leaves out the type, the nullability or the comment). Before this, H2 and HSQLDB files were read as MySQL, rules included, so dropping an H2 key by the name its CREATE gave it left the key in place, said as changing nothing.

The reader also applies what it used to skip: PostgreSQL’s RENAME c TO d without COLUMN (a column, not the table), DROP c without COLUMN, Oracle’s DROP (c, d), the USING INDEX ... ENABLE an Oracle export writes after a key and MODIFY c NOT NULL ENABLE, H2’s ALTER COLUMN c RENAME TO d, MariaDB’s MODIFY and CHANGE COLUMN IF EXISTS, and clauses sqlglot keeps as one piece of text (DROP c, DROP d). It also applies MySQL’s ALTER TABLE ... COMMENT (the table comment), H2’s DROP COLUMN (c, d) (each column), and H2’s and HSQLDB’s ALTER COLUMN c <definition>, read as MODIFY is: what it leaves out is kept and said (alter_modify_unsaid_unknown). An Oracle constraint state in an ALTER clause (NOT DEFERRABLE ... VALIDATE) is read without it. An option the grammar splits off, such as SET STATISTICS, or a MySQL table option (ENGINE, CHARSET), is named as written in alter_clause_unsupported; a clause that cannot be read is quoted as written. IF EXISTS on a column the table does not have changes nothing, as in the database, and is said as alter_if_exists_absent. A RENAME it cannot read is alter_unreadable.

A CREATE TABLE the grammar cannot read

An Oracle export (NOT NULL ENABLE, storage clauses), an HSQLDB .script, an H2 file written for MODE=MySQL: a CREATE TABLE the grammar cannot read as written used to be lost without a word. Now it is read, or named.

What the reader said is in the pack. The reader’s diagnostics used to go to the terminal only. Its header now carries them, so a catalog read back from the cache says them too. The pack groups them by code (meta.laneStats.catalog: each code’s count, the first ten tables it named and its first five sentences) with one diagnostic per kind in meta.diagnostics (CATALOG_CREATE_TABLE_UNREADABLE, CATALOG_RULE_ASSUMED, CATALOG_FILE_PART_NOT_HELD, CATALOG_IF_EXISTS_ABSENT and the rest), so a schema that gives three hundred of one is one line with its count. The overview counts them into three gaps: catalog-tables-unread, catalog-rules-assumed and catalog-read-in-part; a clause the catalog does not hold and the reader’s notes count into none. Tables lost or a database assumed make the catalog axis degraded, with the cause (tables-unread, rules-assumed); the fix named is sqlDialects for an assumed database and none for lost tables. ruoyi-vue-pro’s SQL Server schema read with MySQL’s grammar gave 0 of its 60 tables and the axis said shipped; it now says degraded, cause tables-unread. dolphinscheduler’s H2 schema went from 57 to 64 tables and from 583 to 622 columns.

The catalog worker is catalog-ddl/13, so a cached catalog is read again once.

One schema, shipped once per vendor

A repository that has to run on seven databases ships its schema seven times:

DATABASE/
  altibase/all_ddl_altibase.sql
  cubrid/all_ddl_cubrid.sql
  mysql/all_ddl_mysql.sql
  oracle/all_ddl_oracle.sql
  postgres/all_ddl_postgres.sql
  tibero/all_ddl_tibero.sql
  goldilocks/all_ddl_goldilocks.sql

That is every eGovFrame project, and reading all seven is not a fuller catalog: it is the same tables declared seven times. Measured on eGovFramework/egovframe-common-components before this rule existed: 182 tables declared eight times over, 13,505 duplicate-declaration warnings, and four vendors’ files failing to parse under a grammar they were not written for.

cascade init groups the DDL by the vendor each file is FOR (a vendor-named directory or a vendor name in the file name), and when the tree names one vendor, that vendor’s files become catalog.ddl and the rest are recorded as catalog.ddlAlternatives:

"catalog": {
  "source": "file",
  "ddl": ["../DATABASE/mysql/all_ddl_mysql.sql"],
  "ddlAlternatives": {
    "oracle": ["../DATABASE/oracle/all_ddl_oracle.sql"],
    "tibero": ["../DATABASE/tibero/all_ddl_tibero.sql"]
  }
}

Swapping to Oracle is then an edit of catalog.ddl, not a walk of the tree.

Which vendor, and where the answer comes from, in this order:

  1. sqlDialects.main, when somebody already wrote it down. Their word stands.
  2. a property that names the database outright: a key ending in db-type, database-type, db-dialect or dbms, whose value is a vendor this engine routes. This is eGovFrame’s Globals.DbType = mysql, and it is the one that is true, because the jdbc url beside it is commented out six times and live once.
  3. the scheme of the jdbc url, when every connection file in the tree agrees on one vendor. Several urls naming several databases is a deployment with several, and picking one by counting would decide which database a project talks to.

No answer means no answer: catalog.source stays "none", the diagnostic names the vendors it found, and you pick.

A tree that ships ONE vendor’s schema has nothing to choose between, but it still has a dialect, and init writes it to sqlDialects.main in this order: the dialect a profile already declares, then a MySQL marker in the DDL itself (backquotes, ENGINE=), then the vendor every jdbc url agrees on, then the dialect the schema file’s own text is written in (SERIAL, a ::type cast or OWNER TO for PostgreSQL; VARCHAR2( or NUMBER( for Oracle). With none of them the key stays empty, and the run reads the SQL as MySQL and says so: PROFILE_DEFAULT_ASSUMED when sqlDialects is empty, and alter_rule_assumed wherever an ALTER is decided by one of MySQL’s rules. Measured on a PostgreSQL MES whose schema is one pg_dump: before this rule its catalog was parsed as MySQL and not one of its tables was read.

One mapper, shipped once per vendor

The schema is not the only thing a project like that ships seven times. The SQL is too:

mapper/let/sym/prm/
  EgovProgrmManage_SQL_altibase.xml
  EgovProgrmManage_SQL_cubrid.xml
  EgovProgrmManage_SQL_hsql.xml
  EgovProgrmManage_SQL_mysql.xml
  EgovProgrmManage_SQL_oracle.xml
  EgovProgrmManage_SQL_postgres.xml
  EgovProgrmManage_SQL_tibero.xml

Seven files, one <mapper namespace="progrmManageDAO"> between them. Read together they are not seven mappers: they are one mapper whose statements overwrite each other, so the SQL a statement ends up with is whichever copy the walk read last — and that copy is then parsed under the ONE dialect the run chose. Measured on eGovFramework/egovframe-enterprise-business-template before this rule existed: 189 mapper files, 1,423 statements collapsing onto 205 ids, 47 of them failing to parse because they were written for a database this project does not run.

cascade init groups the mappers the same way the DDL is grouped. Two things have to be true for a set, and neither alone is enough: the files declare the same namespace, and their paths are the same path apart from a vendor’s name. EgovProgrmManage_SQL_mysql.xml and EgovProgrmManageDtl_SQL_mysql.xml share a namespace and stay two different mappers, because cutting the vendor word out leaves two different paths.

The copy for the vendor catalog.ddl chose is read; the rest go into mappers.alternatives, and the statement lane reads past them by name:

"mappers": {
  "alternatives": {
    "oracle": ["../src/main/resources/mapper/let/sym/prm/EgovProgrmManage_SQL_oracle.xml"],
    "tibero": ["../src/main/resources/mapper/let/sym/prm/EgovProgrmManage_SQL_tibero.xml"]
  }
}

The run says both numbers:

mapper XML: 27 file(s), 162 left out as other vendors' copies (profile mappers.alternatives)

A set with no copy for the chosen vendor keeps the first copy by sorted path and says so (MAPPER_VENDOR_UNMATCHED): dropping the set would lose the statements, and choosing by reading the SQL would be this engine deciding which database a project runs on. Move a vendor’s files out of mappers.alternatives to read that vendor instead.

2b. iBATIS 2

The statement lane reads two elements, not one. <mapper namespace="…"> is MyBatis 3; <sqlMap namespace="…"> is iBATIS 2, which every eGovFrame project written before 3.x runs on, and a tree can hold both at once. One worker reads both, and one framework pack (mybatis-xml) declares the lane for either.

What is different about the older element, and what this lane does with it:

iBATIS 2 read as
<select> <insert> <update> <delete> <procedure> <statement> all six are SQL the runtime sends
#name# a bind parameter, the same as MyBatis’ #{name}
$name$ a raw substitution, the same as ${name}, and the statement is flagged
$dbMain$.tb_user a schema qualifier when schema.propertyNames declares it
<dynamic prepend="WHERE"> the prepend is folded in, and the first AND/OR under it is dropped
<dynamic> with no prepend the body is kept as it stands, conjunctions and all
<isNotNull> <isEqual> <isNotEmpty> … the affixes plus the body; the test is not modelled, exactly as <if test> is not
<iterate open close conjunction> the body ONCE, with open/close folded in
<include refid> / <sql id> resolved, in the same index the MyBatis fragments use

What a MyBatis statement is called is its namespace plus its id, except that an id already starting with its namespace is not prefixed again, the rule MyBatis applies itself: <mapper namespace="loginDAO"> with <select id="loginDAO.actionLogin"> is loginDAO.actionLogin.

What an iBATIS statement is called depends on one setting. <sqlMapConfig><settings useStatementNamespaces="true"/> makes the runtime key namespace.id; without it — and iBATIS’ own default is without it — the key is the bare id, global across every sqlMap file in the deployment. The worker reads that setting out of any <sqlMapConfig> among the lane’s inputs (discovery hands it the directory the file sits in), and two configurations that disagree are reported with the default left standing. Two files declaring one id is a duplicate_statement_id warning naming both.

The Java side of the same shape is in docs/setup/java-lane.md: which receivers run an iBATIS statement, and how a bare id binds.

2c. Stored routines

A mapper that writes {call p_close_order(?, ?)} names a procedure, and the tables are inside the procedure. Korean SI systems keep a large share of their business logic that way, in Oracle packages and PostgreSQL functions, and a lineage that stops at the call stops before the tables. So the catalog reads the routines the DDL declares, and a statement that calls one is read through it.

What the catalog reads, as text, from the same DDL files:

written read as
CREATE [OR REPLACE] FUNCTION\|PROCEDURE name(...) ... AS $$ body $$ a PostgreSQL routine; any $tag$ quote works, and LANGUAGE is recorded
CREATE [OR REPLACE] PROCEDURE\|FUNCTION name ... IS\|AS ... END; then / an Oracle, Tibero or Altibase routine
CREATE [OR REPLACE] PACKAGE BODY pkg IS PROCEDURE p ... FUNCTION f ... one routine per member, named pkg.p and pkg.f

A comment or a string literal that says CREATE FUNCTION is not a routine. Each one is a routine record in the catalog stream, beside the tables.

What a routine runs is every SELECT, INSERT, UPDATE, DELETE, MERGE and WITH its body holds, taken out of the control flow around it (IF … THEN, FOR r IN … LOOP, cursor declarations). PL/SQL’s own SELECT a INTO v FROM t loses its INTO v, so the rest is SQL. A branch is not modelled: every statement a body holds is one it CAN run, which is the reading a MyBatis <if> already gets.

How a statement reaches one. A statement names a routine by {call x}, {? = call x}, CALL x, EXEC x, PERFORM x, a BEGIN x(...); END; block, or x(...) inside its SQL. The name is looked up case-insensitively, whole (app.p_close_order) or bare (p_close_order). A bare name two routines share is not looked up by its bare spelling: which one an unqualified call reaches is the database’s search path, and it is not guessed. A routine that calls a routine is followed four deep, and a cycle stops at the names already read.

What the routines reach joins the statement’s own facts marked "via": "routine", and the statement’s record lists what was read:

"routines": [
  { "name": "app.p_close_order", "depth": 1, "statements": 3, "parsed": 3 },
  { "name": "app.f_log", "depth": 2, "statements": 2, "parsed": 2 }
]

On the graph those are EXECUTES, READS and WRITES edges graded SOUND_SET, with the routine names in the evidence. The statement’s own SQL stays EXACT. When the catalog holds any routine, a statement that is nothing but a call is not handed to the parser at all, so it records no parse failure. A statement that calls no routine is byte-identical to what it was.

What it does not do. A routine the DDL does not declare is not read, and routines are read from DDL files only, not from a live database. SQL a body builds as a string and runs (EXECUTE IMMEDIATE, EXECUTE format(...)) is not read. A trigger is not followed: a statement that writes a table does not reach what a trigger on that table runs.

3. Where the table and column comments come from

Column answers carry business meaning only when the catalog carries comments. There are two ways to get them, and you pick one per project.

A file in the repo. If the tree ships a schema dump with CREATE TABLE and COMMENT, point catalog.source: "file" and catalog.connectionFrom at it. This needs nothing but Python and sqlglot, and it is where to start.

A snapshot you fetch once. For a schema that lives only in a running database, cascade catalog fetch opens one read-only connection, reads the tables, columns and comments, and writes a snapshot into .cascade/. It then writes catalog.source: "jdbc" into the profile itself, so nothing is left for you to hand-edit. Analysis reads the snapshot, never the server. Read db-catalog.md for the connection and credential rules. A password is never committed, and never written into .cascade/, the pack or a log: the one place it may be kept is ~/.cascade/credentials, at mode 0600.

With no catalog at all, the lane still works: a table or column exists in the pack wherever a statement named it. The pack then declares the catalog axis not-shipped and says why, so you can see what you are missing rather than guess.

4. A first project to try it on

macrozheng/mall is a good first target: Apache-2.0, Spring Boot with MyBatis XML mappers, a MySQL dump in the repo, and a matching Vue admin. About 30 tables, which is big enough to show one column reaching many screens and small enough to iterate on.

Keep target checkouts outside this repository, in a sibling folder. The name target-examples/ says what it is: disposable, safe to delete later without anyone having to investigate first.

git clone --depth 1 https://github.com/macrozheng/mall ../target-examples/mall

node bin/cascade.mjs init    --root ../target-examples/mall --project mall
node bin/cascade.mjs analyze --root ../target-examples/mall
node bin/cascade.mjs view    --project mall

init writes .cascade/manifest.json (the checkout path and its pinned commit) and .cascade/profile.json (the build tool, the framework packs, the dialect, and where the catalog comes from). analyze takes no lane flags: it reads those two files plus what discovery found. If you want to know what a run would ship before you run it, cascade estimate says so axis by axis.

The venv (.venv/) and __pycache__/ are gitignored engine runtime, and are not committed.

5. Testing the lane

.venv/bin/python -m unittest discover -s adapters/sql -q   # the Python side
npm run test:quick                                          # the Node side

scripts/generality-gate.mjs runs the whole engine over a pinned corpus of real repositories that nobody wrote it for, Apache DolphinScheduler and JeecgBoot among them. That corpus exists to catch a rule that only works on the project it was written against; it is not a fallback target, and it is not a demo. It clones outside this repository and is skipped unless you ask for it.