Add full stage manifests

2026-08-04 11:37:20 +08:00
parent 0b7e69770c
commit 5874cf9379
9 changed files with 516 additions and 21 deletions
--- a/.trellis/spec/pipeline/cli-and-stages.md
+++ b/.trellis/spec/pipeline/cli-and-stages.md
@@ -300,9 +300,10 @@ const gate = classifyAreaQuality(result);

 ### 1. Scope / Trigger

-Stage manifest 是构建阶段完成后的机器可读产物契约。第一版覆盖 GLB 相关阶段：
-`cesium` 和 `compress`。它用于诊断产物是否存在、是否 stale、体量是否超预算，以及后续
-`check:area` / 增量构建判断。
+Stage manifest 是构建阶段完成后的机器可读产物契约。它覆盖完整区域链：
+`intermediates`、`reimport`、`blender`、`cesium`、`preview` 和 `compress`。
+它用于诊断产物是否存在、是否 stale、体量是否超预算，以及后续 `check:area` /
+增量构建判断。

 ### 2. Signatures

@@ -315,7 +316,11 @@ Manifest 路径固定：
 当前会写：

 ```text
+<areaDir>/_pipeline/stages/intermediates.manifest.json
+<areaDir>/_pipeline/stages/reimport.manifest.json
+<areaDir>/_pipeline/stages/blender.manifest.json
 <areaDir>/_pipeline/stages/cesium.manifest.json
+<areaDir>/_pipeline/stages/preview.manifest.json
 <areaDir>/_pipeline/stages/compress.manifest.json
 ```

@@ -357,18 +362,65 @@ File records use this shape:
 }
 ```

+`intermediates` manifest:
+
+- `inputs.config`
+- `inputs.osm`
+- `outputs.derivedConfig`
+- `outputs.geojsonDir`
+- `outputs.<nine scene layers>`、`outputs.osm2streets_scene.geojson`、
+  `outputs.osm2streets_scene_style.json`
+- `outputs.gpkg`、`outputs.qgisProject`、可选 `outputs.qgisPreview`
+- `summary.geojson`：各 layer / merged scene 的 feature count
+
+`reimport` manifest:
+
+- `inputs.config`
+- `inputs.derivedConfig`
+- `inputs.gpkg`
+- `outputs.geojsonDir`
+- `outputs.<nine scene layers>`、merged scene/style
+- `summary.geojson`
+
+`intermediates` 和 `reimport` 是同一批 GeoJSON 的互斥 ownership：
+成功运行 `intermediates` 删除旧 `reimport.manifest.json`；成功运行 `reimport` 删除旧
+`intermediates.manifest.json`。诊断只要求当前 ownership 路径的 manifest，不要求两者同时存在。
+
+`blender` manifest:
+
+- `inputs.config`
+- `inputs.osm`
+- `inputs.geojsonDir`、`inputs.<nine scene layers>`、merged scene/style
+- `outputs.blend`
+- `outputs.render`
+- `summary.geojson`、`summary.blendBytes`、`summary.renderBytes`
+
 `cesium` manifest:

 - `inputs.blend`
 - `outputs.glb`
 - `outputs.metadata`
- `outputs.cesiumPreview`
- `outputs.vehicleRoute`
- `outputs.vehicleModel`
 - `summary.glb.fileBytes`
 - `summary.glb.counts`
 - `summary.glb.extensionsUsed`

+`cesium` 会调用 preview 生成函数，但 preview HTML / route / vehicle model 的 freshness
+所有权属于独立 `preview` manifest。否则单跑 `--stages preview` 会把 Cesium manifest
+错误判 stale。
+
+`preview` manifest:
+
+- `inputs.config`
+- `inputs.osm`
+- `inputs.glb`
+- `inputs.metadata`
+- `inputs.previewCss`
+- `inputs.previewJs`
+- `outputs.cesiumPreview`
+- `outputs.vehicleRoute`
+- `outputs.vehicleModel`
+- `summary.glbName`、`summary.metadataName`、`summary.routeSegments`
+
 `compress` manifest:

 - `inputs.glb`
@@ -395,12 +447,19 @@ Manifest files are written atomically via `*.tmp` then `renameSync`.
 | Manifest JSON 无法解析 | 诊断报告 invalid warning |
 | Manifest 记录的 input / output sha/bytes 与当前文件不同 | 诊断报告 stale warning |
 | GLB size / nodes / images 超保守预算 | manifest `warnings[]` 记录，诊断继续独立报警 |
+| `intermediates` 后留有旧 reimport manifest | 成功后删除旧 reimport manifest |
+| `reimport` 后留有旧 intermediates manifest | 成功后删除旧 intermediates manifest |

 ### 5. Good/Base/Bad Cases

- Good: `--stages cesium` 成功后写 `cesium.manifest.json`，诊断显示 `ok cesium`。
+- Good: `--stages intermediates` 成功后写 `intermediates.manifest.json`，诊断显示
+  `ok intermediates`。
+- Good: `--stages blender` 成功后写 `blender.manifest.json`，诊断显示 `ok blender`。
+- Good: `--stages cesium` 成功后写 `cesium.manifest.json` 和 `preview.manifest.json`。
+- Good: `--stages preview` 只更新 preview manifest，不让 cesium manifest stale。
 - Good: `--stages compress` 成功后写 `compress.manifest.json`，summary 记录压缩比和节省字节。
- Base: 旧产物没有 manifest，诊断显示 expected manifest missing，提示重跑对应阶段。
+- Base: 旧产物没有当前 ownership 路径的 manifest，诊断显示 expected manifest missing，
+  提示重跑对应阶段。
 - Bad: 手工编辑 GLB 后不重跑阶段，诊断显示 manifest stale。

 ### 6. Tests Required
@@ -408,10 +467,15 @@ Manifest files are written atomically via `*.tmp` then `renameSync`.
 - `node --check scripts/lib/stage-manifest.js`
 - `node --check scripts/build-area.js`
 - `node --check scripts/diagnose-area.js`
+- `node --check scripts/check-area.js`
+- `npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages intermediates`
+- `npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages blender`
 - `npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages cesium`
+- `npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages preview`
 - `npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages compress`
 - `npm run diagnose:area -- --config config/areas/nantaizi-lake-innovation-valley.json`
- JSON parse both generated manifests and assert `manifestVersion`, `stage`, `inputs`,
+- `npm run check:area -- --config config/areas/nantaizi-lake-innovation-valley.json`
+- JSON parse generated manifests and assert `manifestVersion`, `stage`, `inputs`,
  `outputs`, `summary`, and `warnings` fields exist.

 ### 7. Wrong vs Correct
--- a/.trellis/spec/pipeline/index.md
+++ b/.trellis/spec/pipeline/index.md
@@ -38,14 +38,17 @@ config/areas/<id>.json
      │                     → osm2streets_scene.geojson + _scene_style.json
      │                     → ogr2ogr 导入 <id>.gpkg
      │                     → QGIS 生成 .qgz + -preview.png
+      │                     → _pipeline/stages/intermediates.manifest.json
      │
      ├─[reimport]──────▶ reimport-gpkg.js         （反向，与 intermediates 互斥）
      │                     ogr2ogr 从 .gpkg 导出 → 校验 → 覆写 *.geojson
      │                     → 重建 scene.geojson + scene_style.json
+      │                     → _pipeline/stages/reimport.manifest.json
      │
      ├─[blender]───────▶ Blender + blender/generate_scene.py
      │                     读 .osm + osm2streets_web_out/
      │                     → <id>.blend + <id>.png
+      │                     → _pipeline/stages/blender.manifest.json
      │
      ├─[cesium]────────▶ Blender + blender/export_cesium.py
      │                     读 .blend → <id>.glb + <id>.json
@@ -55,6 +58,7 @@ config/areas/<id>.json
      ├─[preview]───────▶ 生成 <id>-cesium-preview.html
      │                     + 拷贝 lib/cesium-preview.{js,css}
      │                     + 车辆巡航路线与模型
+      │                     → _pipeline/stages/preview.manifest.json
      │
      └─[compress]──────▶ 生成并列压缩 GLB / metadata / preview
                            → _pipeline/stages/compress.manifest.json
@@ -83,10 +87,10 @@ config/areas/<id>.json

 | 文件 | 行数 | 职责 |
 |---|---|---|
-| `build-area.js` | 815 | 主入口：区域配置读取、阶段调度、Cesium 预览页、车辆巡航和 stage manifest 写入 |
+| `build-area.js` | 961 | 主入口：区域配置读取、阶段调度、Cesium 预览页、车辆巡航和 stage manifest 写入 |
 | `diagnose-area.js` | 36 | 快速诊断入口：调用共享 area diagnostics 并打印完整报告 |
 | `check-area.js` | 74 | 区域质量门入口：调用共享 area diagnostics，输出 PASS/FAIL 并设置退出码 |
-| `lib/area-diagnostics.js` | 660 | 共享区域诊断事实源：OSM、产物、metadata、stage manifest、GLB digest 和质量门分类 |
+| `lib/area-diagnostics.js` | 776 | 共享区域诊断事实源：OSM、产物、metadata、stage manifest、GLB digest 和质量门分类 |
 | `lib/area-config.js` | 135 | 区域配置归一化与输出路径推导，供 build / diagnose 复用 |
 | `lib/stage-manifest.js` | 100 | stage manifest 路径、文件记录、GLB budget warning 和原子 JSON 写入 |
 | `build-osm2streets-qgis.js` | 1468 | intermediates：osm2streets 解析、图层拆分、人行道转角合成、GeoPackage 与 QGIS 工程生成 |
--- a/.trellis/tasks/08-04-add-full-stage-manifests/design.md
+++ b/.trellis/tasks/08-04-add-full-stage-manifests/design.md
@@ -0,0 +1,77 @@
+# Design
+
+## Scope
+
+The existing manifest infrastructure already supports file records, atomic JSON
+write, digest summaries, and diagnostics-side freshness checks. This task
+extends that same contract to the earlier and read-only/patched stages without
+adding a new manifest format.
+
+## Stage Boundaries
+
+`intermediates`
+
+- Written by `build-area.js` after `build-osm2streets-qgis.js` succeeds.
+- Inputs: area config, source OSM XML.
+- Outputs: derived config, GeoJSON directory, all scene-layer GeoJSON files,
+  merged scene/style files, GeoPackage, QGIS project, QGIS preview if present.
+- Summary: GeoJSON feature counts by file.
+
+`reimport`
+
+- Written by `build-area.js` after `reimport-gpkg.js` succeeds.
+- Inputs: area config, GeoPackage.
+- Outputs: GeoJSON directory, all scene-layer GeoJSON files, merged scene/style
+  files.
+- Summary: GeoJSON feature counts by file.
+
+`blender`
+
+- Written by `build-area.js` after Blender scene generation succeeds.
+- Inputs: area config, source OSM XML, GeoJSON directory, all scene-layer
+  GeoJSON files, merged scene/style files.
+- Outputs: `.blend`, render PNG.
+- Summary: parsed `SCENE_DONE` payload when available, plus blend/render bytes.
+
+`preview`
+
+- Written by `build-area.js` after preview generation succeeds. Since `cesium`
+  automatically calls preview generation, this manifest should also be written
+  during `--stages cesium`.
+- Inputs: area config, source OSM XML, GLB, metadata, preview runtime JS/CSS.
+- Outputs: Cesium preview HTML, vehicle route JSON, vehicle model glTF.
+- Summary: vehicle route count and asset filenames.
+
+## Diagnostics
+
+`scripts/lib/area-diagnostics.js` will extend `stageManifestStatus()` from two
+hard-coded GLB stages to the full ordered stage list. Expected status should
+derive from stage outputs:
+
+- `intermediates`: expected when GeoJSON dir, GeoPackage, or QGIS project exists.
+- `reimport`: expected when GeoJSON dir exists and `reimport.manifest.json`
+  already exists; it is an alternate path and should not be required for normal
+  intermediates builds.
+- `blender`: expected when blend exists.
+- `cesium`: expected when baseline GLB exists.
+- `preview`: expected when Cesium preview exists.
+- `compress`: expected when compressed GLB exists.
+
+Only expected manifests are fatal in `check:area`. Optional absent manifests
+remain ignored.
+
+## Compatibility
+
+- Existing `cesium.manifest.json` and `compress.manifest.json` remain valid.
+- Existing outputs without new manifests will show missing expected manifests
+  until their stages are rerun.
+- No stage should overwrite its primary outputs differently just to write a
+  manifest.
+
+## Tradeoffs
+
+- Manifest writing lives in `build-area.js` for this task because it already
+  owns normalized area paths and stage timing. Moving preview generation and
+  vehicle assets out of `build-area.js` is a later refactor.
+- Reimport remains optional/alternate to avoid making normal `check:area` fail
+  for users who never use the reimport workflow.
--- a/.trellis/tasks/08-04-add-full-stage-manifests/implement.md
+++ b/.trellis/tasks/08-04-add-full-stage-manifests/implement.md
@@ -0,0 +1,30 @@
+# Implementation Plan
+
+1. Read current manifest and stage code.
+2. Add reusable helpers in `build-area.js` for:
+   - file record collection
+   - GeoJSON feature-count summaries
+   - manifest duration/timestamps
+   - writing `intermediates`, `reimport`, `blender`, and `preview` manifests
+3. Wire manifest writes after successful stage functions only.
+4. Extend `area-diagnostics.js` manifest expected/input/output definitions to
+   cover all stages.
+5. Update README, changelog, and `.trellis/spec/pipeline/*`.
+6. Validate with:
+   - `node --check scripts/build-area.js`
+   - `node --check scripts/lib/area-diagnostics.js`
+   - `node --check scripts/check-area.js`
+   - `node --check scripts/diagnose-area.js`
+   - `npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages preview`
+   - `npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages blender`
+   - `npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages intermediates`
+   - `npm run diagnose:area -- --config config/areas/nantaizi-lake-innovation-valley.json`
+   - `npm run check:area -- --config config/areas/nantaizi-lake-innovation-valley.json`
+   - controlled stale/missing manifest failure
+   - `git diff --check`
+
+## Rollback
+
+- Remove added manifest write calls and diagnostics stage definitions.
+- Delete generated `_pipeline/stages/{intermediates,reimport,blender,preview}.manifest.json`
+  from outputs if needed; outputs are ignored and not committed.
--- a/.trellis/tasks/08-04-add-full-stage-manifests/prd.md
+++ b/.trellis/tasks/08-04-add-full-stage-manifests/prd.md
@@ -0,0 +1,50 @@
+# Add full stage manifests
+
+## Goal
+
+Extend the stage manifest contract from GLB-only stages to the full area
+pipeline so `diagnose:area` and `check:area` can reason about stage freshness
+across intermediates, reimport, blender, preview, cesium, and compress.
+
+## Requirements
+
+- Write stage manifests for:
+  - `intermediates`
+  - `reimport`
+  - `blender`
+  - `preview`
+- Preserve existing `cesium` and `compress` manifest behavior.
+- Do not change default build outputs except for adding/updating manifest files
+  under `<areaDir>/_pipeline/stages/`.
+- Stage manifests must record enough inputs and outputs to detect stale files
+  using the existing bytes/sha freshness logic.
+- `diagnose:area` must display these additional manifests.
+- `check:area` must fail when an expected full-pipeline manifest is missing,
+  invalid, or stale.
+- `reimport` and `intermediates` remain mutually exclusive.
+- Missing optional `compress` artifacts still must not make `check:area` fail.
+- README, changelog, and Trellis pipeline spec must document the expanded
+  manifest coverage.
+
+## Acceptance Criteria
+
+- [x] Running `npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages preview`
+      writes a fresh `preview.manifest.json`.
+- [x] Running `npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages blender`
+      writes a fresh `blender.manifest.json`.
+- [x] Running `npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages intermediates`
+      writes a fresh `intermediates.manifest.json`.
+- [x] `diagnose:area` shows `intermediates`, `blender`, `cesium`, `preview`, and
+      optional `compress` manifest status.
+- [x] `check:area` passes for current nantaizi outputs after the required
+      manifests are generated.
+- [x] A controlled stale/missing manifest case makes `check:area` exit non-zero.
+- [x] Syntax checks pass for changed Node scripts.
+- [x] README, changelog, and pipeline spec describe full stage manifest coverage.
+
+## Notes
+
+- Validate only `nantaizi-lake-innovation-valley` unless explicitly asked.
+- Ignore `hanyang-block`.
+- This task is about pipeline contracts and diagnostics, not lane-arrow visual
+  behavior.