Add OSM area preflight
This commit is contained in:
@@ -12,6 +12,7 @@
|
|||||||
| `scripts/build-osm2streets-qgis.js` | intermediates 阶段的实现 | 由 build-area 调起;`npm run build:qgis` 可单跑 |
|
| `scripts/build-osm2streets-qgis.js` | intermediates 阶段的实现 | 由 build-area 调起;`npm run build:qgis` 可单跑 |
|
||||||
| `scripts/reimport-gpkg.js` | reimport 阶段的实现 | 由 build-area 调起 |
|
| `scripts/reimport-gpkg.js` | reimport 阶段的实现 | 由 build-area 调起 |
|
||||||
| `scripts/diagnose-area.js` | 快速区域诊断;只读配置、OSM 和已有产物 | `npm run diagnose:area` |
|
| `scripts/diagnose-area.js` | 快速区域诊断;只读配置、OSM 和已有产物 | `npm run diagnose:area` |
|
||||||
|
| `scripts/preflight-area.js` | OSM 构建前预检;只读配置和 OSM,并写成功预检记录 | `npm run preflight:area` |
|
||||||
| `scripts/check-area.js` | 区域质量门;复用诊断事实源并给出 PASS/FAIL 退出码 | `npm run check:area` |
|
| `scripts/check-area.js` | 区域质量门;复用诊断事实源并给出 PASS/FAIL 退出码 | `npm run check:area` |
|
||||||
|
|
||||||
`scripts/parity.js` 和 `scripts/glb-digest.js` 是校验工具,不属于构建链,见
|
`scripts/parity.js` 和 `scripts/glb-digest.js` 是校验工具,不属于构建链,见
|
||||||
@@ -121,6 +122,73 @@ npm run compress:glb -- --input outputs/a/a.glb --output outputs/a/a-compressed-
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## OSM 预检命令
|
||||||
|
|
||||||
|
### 1. Scope / Trigger
|
||||||
|
|
||||||
|
`preflight:area` 用于手工编辑 OSM 后、运行 QGIS / Blender / Cesium 前的快速结构门槛。
|
||||||
|
它不是构建阶段,不进入 `build-area --stages`,不会启动外部工具或写场景产物。
|
||||||
|
|
||||||
|
### 2. Signatures
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run preflight:area
|
||||||
|
npm run preflight:area -- --config config/areas/<area>.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Contracts
|
||||||
|
|
||||||
|
- 通过 `readAreaConfig()` 读取区域配置,只读取配置文件和 OSM XML。
|
||||||
|
- OSM 解析和错误分类来自 `scripts/lib/area-diagnostics.js`,不得另建 XML 解析器。
|
||||||
|
- `<node>`、`<way>`、`<relation>` 带 `action="delete"` 时属于历史删除对象,解析统计和
|
||||||
|
几何检查必须忽略;否则旧 OSM 导出会把已删除的建筑误报为损坏面。
|
||||||
|
- 无效 bounds、任意 way 缺失 node、破损 building way、破损 building multipolygon、
|
||||||
|
不可解析为正米数的显式 building `height` 都是 error,并以非零退出。
|
||||||
|
- 不填 `height` / `building:levels` 合法;无效 `building:levels` 是 warning。
|
||||||
|
- 仅在 error 为零时原子写 `<areaDir>/_pipeline/stages/preflight.manifest.json`。失败不覆盖
|
||||||
|
上次成功 manifest。manifest 记录 config / OSM 文件摘要、预检 summary、errors、warnings
|
||||||
|
和耗时。
|
||||||
|
- `diagnose:area` 显示该 manifest;`check:area` 仅在 manifest 存在时检查其有效性和
|
||||||
|
freshness,故采用前不会使旧区域质量门失败。
|
||||||
|
|
||||||
|
### 4. Validation & Error Matrix
|
||||||
|
|
||||||
|
| 条件 | 结果 |
|
||||||
|
|---|---|
|
||||||
|
| 无效 bounds、缺失 node、破损建筑几何、坏显式 height | 输出 error,非零,不覆盖旧 manifest |
|
||||||
|
| 无效 `building:levels` | 输出 warning,仍可成功写 manifest |
|
||||||
|
| `action="delete"` 的节点 / way / relation | 忽略,不计入当前 OSM 或报错 |
|
||||||
|
| OSM / config 不存在 | `readAreaConfig()` 或文件读取抛错,非零 |
|
||||||
|
|
||||||
|
### 5. Good/Base/Bad Cases
|
||||||
|
|
||||||
|
- Good: 手工编辑后预检 PASS,再运行 `build-area` 的任意重型阶段。
|
||||||
|
- Base: 未运行过预检的旧区域,`check:area` 仍可按既有规则通过。
|
||||||
|
- Bad: 预检失败后仍更新 manifest,或把预检加入 `all` 导致每次构建修改验证记录。
|
||||||
|
|
||||||
|
### 6. Tests Required
|
||||||
|
|
||||||
|
- `npm run test:preflight`
|
||||||
|
- `npm run preflight:area -- --config config/areas/nantaizi-lake-innovation-valley.json`
|
||||||
|
- `npm run diagnose:area -- --config config/areas/nantaizi-lake-innovation-valley.json`
|
||||||
|
- `npm run check:area -- --config config/areas/nantaizi-lake-innovation-valley.json`
|
||||||
|
|
||||||
|
### 7. Wrong vs Correct
|
||||||
|
|
||||||
|
Wrong:
|
||||||
|
|
||||||
|
```js
|
||||||
|
const xml = fs.readFileSync(area.input, "utf8");
|
||||||
|
const missingNodes = xml.match(/<nd[^>]*ref=/g).length;
|
||||||
|
```
|
||||||
|
|
||||||
|
Correct:
|
||||||
|
|
||||||
|
```js
|
||||||
|
const osm = parseOsm(fs.readFileSync(area.input, "utf8"));
|
||||||
|
const preflight = analyzeOsmPreflight(osm);
|
||||||
|
```
|
||||||
|
|
||||||
## 区域诊断命令
|
## 区域诊断命令
|
||||||
|
|
||||||
### 1. Scope / Trigger
|
### 1. Scope / Trigger
|
||||||
@@ -300,8 +368,8 @@ const gate = classifyAreaQuality(result);
|
|||||||
|
|
||||||
### 1. Scope / Trigger
|
### 1. Scope / Trigger
|
||||||
|
|
||||||
Stage manifest 是构建阶段完成后的机器可读产物契约。它覆盖完整区域链:
|
Stage manifest 是区域构建阶段或独立验证通过后的机器可读产物契约。它覆盖预检记录与完整区域链:
|
||||||
`intermediates`、`reimport`、`blender`、`cesium`、`preview` 和 `compress`。
|
`preflight`、`intermediates`、`reimport`、`blender`、`cesium`、`preview` 和 `compress`。
|
||||||
它用于诊断产物是否存在、是否 stale、体量是否超预算,以及后续 `check:area` /
|
它用于诊断产物是否存在、是否 stale、体量是否超预算,以及后续 `check:area` /
|
||||||
增量构建判断。
|
增量构建判断。
|
||||||
|
|
||||||
@@ -316,6 +384,7 @@ Manifest 路径固定:
|
|||||||
当前会写:
|
当前会写:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
<areaDir>/_pipeline/stages/preflight.manifest.json
|
||||||
<areaDir>/_pipeline/stages/intermediates.manifest.json
|
<areaDir>/_pipeline/stages/intermediates.manifest.json
|
||||||
<areaDir>/_pipeline/stages/reimport.manifest.json
|
<areaDir>/_pipeline/stages/reimport.manifest.json
|
||||||
<areaDir>/_pipeline/stages/blender.manifest.json
|
<areaDir>/_pipeline/stages/blender.manifest.json
|
||||||
@@ -362,6 +431,15 @@ File records use this shape:
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`preflight` manifest:
|
||||||
|
|
||||||
|
- `inputs.config`
|
||||||
|
- `inputs.osm`
|
||||||
|
- `outputs` 固定为空对象(预检不拥有场景产物)
|
||||||
|
- `summary`:bounds、node / way / relation、building way / multipolygon 与问题计数
|
||||||
|
- `errors` 固定为空数组(只有通过的预检会写 manifest)和 `warnings`
|
||||||
|
- manifest 不存在时不算质量门失败;存在但 input 摘要漂移或 JSON 损坏时,`check:area` 失败
|
||||||
|
|
||||||
`intermediates` manifest:
|
`intermediates` manifest:
|
||||||
|
|
||||||
- `inputs.config`
|
- `inputs.config`
|
||||||
|
|||||||
1
.trellis/tasks/08-04-add-osm-area-preflight/check.jsonl
Normal file
1
.trellis/tasks/08-04-add-osm-area-preflight/check.jsonl
Normal file
@@ -0,0 +1 @@
|
|||||||
|
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||||
30
.trellis/tasks/08-04-add-osm-area-preflight/design.md
Normal file
30
.trellis/tasks/08-04-add-osm-area-preflight/design.md
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
# OSM area preflight design
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
`scripts/lib/area-diagnostics.js` 保持 OSM 解析和预检分类的唯一事实源。新增 `scripts/preflight-area.js` 只负责 CLI 参数、报告和退出码;它不读取产物,也不调起外部工具。现有 `diagnose-area.js`、`check-area.js` 不复制预检逻辑。
|
||||||
|
|
||||||
|
## Data Flow
|
||||||
|
|
||||||
|
```text
|
||||||
|
area config + OSM XML
|
||||||
|
-> analyzeOsmPreflight()/shared parseOsm()
|
||||||
|
-> { summary, errors, warnings }
|
||||||
|
-> preflight-area CLI report + exit code
|
||||||
|
-> successful preflight.manifest.json
|
||||||
|
-> diagnose/check manifest freshness reporting
|
||||||
|
```
|
||||||
|
|
||||||
|
预检 manifest 是一次成功验证的记录,不是输入所有权或构建依赖。它记录 config/OSM 文件摘要、错误/警告、统计和耗时;预检失败时不写入,也不删除旧 manifest,以保留最后一次成功记录。
|
||||||
|
|
||||||
|
## Classification
|
||||||
|
|
||||||
|
Blocking error 仅限肯定会使场景几何或范围不可靠的 OSM 结构错误:无效 bounds、任意 way 的缺失 node 引用、破损的建筑 way / building multipolygon。没有 `height` / levels 不是错误(渲染已有默认值);填写但不可解析的显式 `height` 是错误。`building:levels` 的非正值或不可解析值将作为 warning,除非现有 Blender 行为表明它会产生错误几何。
|
||||||
|
|
||||||
|
## Compatibility
|
||||||
|
|
||||||
|
现有 `diagnose:area` 保持全量只读报告;它添加 preflight manifest 一行。`check:area` 仅在 manifest 存在时才要求其有效和 fresh,确保现有历史构建仍可通过。新命令的默认配置规则与其他 area CLI 一致。
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
删除 `preflight:area` CLI、预检 manifest 定义和 diagnostics 展示即可回到当前行为。预检 manifest 是可再生文件,保留或删除均不影响构建。
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||||
23
.trellis/tasks/08-04-add-osm-area-preflight/implement.md
Normal file
23
.trellis/tasks/08-04-add-osm-area-preflight/implement.md
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
# OSM area preflight implementation plan
|
||||||
|
|
||||||
|
1. 确认 Blender 对 `building:levels` 的解析语义;缺失 node 引用按已确认策略一律阻断。
|
||||||
|
2. 在 `area-diagnostics.js` 抽取结构化 OSM 预检分类,并保持 diagnose/check 共用解析。
|
||||||
|
3. 新增 `preflight-area.js` 和 `package.json` 命令;成功后原子写 preflight manifest。
|
||||||
|
4. 扩展 stage manifest 状态与 `diagnose:area` / `check:area` 对 preflight 的显示、freshness 与 failure 规则。
|
||||||
|
5. 为纯 OSM 预检逻辑添加 Node fixture 或等价轻量测试,覆盖关键破损输入。
|
||||||
|
6. 更新 README、pipeline CLI spec、changelog;运行质量检查。
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node --check scripts/lib/area-diagnostics.js
|
||||||
|
node --check scripts/preflight-area.js
|
||||||
|
node --check scripts/diagnose-area.js
|
||||||
|
node --check scripts/check-area.js
|
||||||
|
npm run preflight:area -- --config config/areas/nantaizi-lake-innovation-valley.json
|
||||||
|
npm run diagnose:area -- --config config/areas/nantaizi-lake-innovation-valley.json
|
||||||
|
npm run check:area -- --config config/areas/nantaizi-lake-innovation-valley.json
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
只验证 `nantaizi-lake-innovation-valley`。破损 fixture 使用临时目录,不能改写桌面上的 OSM 或正式 outputs。
|
||||||
46
.trellis/tasks/08-04-add-osm-area-preflight/prd.md
Normal file
46
.trellis/tasks/08-04-add-osm-area-preflight/prd.md
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
# Add OSM area preflight
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
在调用 QGIS、Blender 或 Cesium 前,快速验证区域 OSM 是否包含会导致建筑或场景几何错误的人工编辑问题。使用者应得到明确的通过/失败结果和可定位的问题列表,而不必先跑重型构建阶段。
|
||||||
|
|
||||||
|
## Confirmed Facts
|
||||||
|
|
||||||
|
- `scripts/lib/area-diagnostics.js` 已是区域配置和 OSM 解析的事实源;它已覆盖 bounds、building multipolygon、缺失 node 引用及高度标签,并被 `diagnose:area` 和 `check:area` 共同复用。
|
||||||
|
- `diagnose:area` 是全文诊断,包含既有产物、GLB 与 manifest;它不适合成为编辑 OSM 后的轻量独立门槛。
|
||||||
|
- 当前 `nantaizi-lake-innovation-valley` 是唯一验证目标。忽略 `action=delete` 的历史对象后,其 OSM 有 1 个健康的建筑 multipolygon 和 23 个有效显式高度标签。
|
||||||
|
- 全阶段 manifest 已存在;新预检不应被实现为 `build-area --stages` 的构建阶段。
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
1. 新增 `npm run preflight:area -- --config config/areas/<area>.json`,只读取区域配置和 OSM XML;不得调用 QGIS、Blender、Cesium、压缩或写入场景产物。
|
||||||
|
2. 从共享 diagnostics 模块提供结构化 OSM 预检结果,避免新增第二套 XML/建筑关系解析。
|
||||||
|
3. 预检应检查:有效 bounds、way 引用的 node、building way 闭合性和最小节点数、建筑 multipolygon 的 outer/inner member、成员存在性、可拼接闭环,以及 building `height` / `building:levels` 的基本可解析性。
|
||||||
|
4. 命令输出简短摘要、错误和警告;存在 blocking error 时以非零退出。
|
||||||
|
5. 成功预检写 `<areaDir>/_pipeline/stages/preflight.manifest.json`,记录配置和 OSM 输入的文件摘要、预检摘要、errors 与 warnings。失败预检不得覆盖此前成功 manifest。
|
||||||
|
6. `diagnose:area` 显示预检 manifest 的状态;`check:area` 仅在该 manifest 已存在时检查其新鲜度和有效性,避免使历史区域在首次采用前立即失败。
|
||||||
|
7. README 和 Pipeline spec 记录命令、退出码与 manifest 所有权。
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- [x] 当前南台子 OSM 上 `npm run preflight:area -- --config config/areas/nantaizi-lake-innovation-valley.json` 输出 PASS、写入 fresh `preflight.manifest.json`,且不触发重型工具。
|
||||||
|
- [x] 预检报告包含 buildings、multipolygon、bounds 和几何健康统计。
|
||||||
|
- [x] 通过临时 OSM fixture 验证:缺失/无效 bounds、缺失 node、未闭合 building way、破损 building multipolygon、无效显式 height 都能得到定位明确的 blocking error。
|
||||||
|
- [x] `diagnose:area` 显示 preflight manifest,`check:area` 对存在但 stale/invalid 的 preflight manifest 失败。
|
||||||
|
- [x] 原有 `diagnose:area` 与 `check:area` 继续通过当前南台子区域。
|
||||||
|
- [x] 修改脚本通过 Node 语法检查和 `git diff --check`。
|
||||||
|
|
||||||
|
## Out Of Scope
|
||||||
|
|
||||||
|
- 不修复 OSM 文件,不自动重组关系或补齐标签。
|
||||||
|
- 不检查道路拓扑、转向箭头渲染、osm2streets 的语义兼容性,或任意非建筑面是否合法。
|
||||||
|
- 不把预检加入 `build-area` 的默认阶段,也不要求未运行过预检的历史产物立即失败。
|
||||||
|
- 不验证 `hanyang-block`。
|
||||||
|
|
||||||
|
## Key Decisions
|
||||||
|
|
||||||
|
- 缺失 node 引用一律作为 blocking error。它表示 OSM 引用完整性已损坏,即使当前出现在非建筑 way,也不将问题带入后续构建。
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- 这是跨 CLI、OSM 解析、质量门和 manifest 的改动,按复杂任务处理,需要 design 和 implement 工件后再开始实现。
|
||||||
26
.trellis/tasks/08-04-add-osm-area-preflight/task.json
Normal file
26
.trellis/tasks/08-04-add-osm-area-preflight/task.json
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
{
|
||||||
|
"id": "add-osm-area-preflight",
|
||||||
|
"name": "add-osm-area-preflight",
|
||||||
|
"title": "Add OSM area preflight",
|
||||||
|
"description": "",
|
||||||
|
"status": "in_progress",
|
||||||
|
"dev_type": null,
|
||||||
|
"scope": null,
|
||||||
|
"package": null,
|
||||||
|
"priority": "P2",
|
||||||
|
"creator": "dingkang",
|
||||||
|
"assignee": "dingkang",
|
||||||
|
"createdAt": "2026-08-04",
|
||||||
|
"completedAt": null,
|
||||||
|
"branch": null,
|
||||||
|
"base_branch": "main",
|
||||||
|
"worktree_path": null,
|
||||||
|
"commit": null,
|
||||||
|
"pr_url": null,
|
||||||
|
"subtasks": [],
|
||||||
|
"children": [],
|
||||||
|
"parent": null,
|
||||||
|
"relatedFiles": [],
|
||||||
|
"notes": "",
|
||||||
|
"meta": {}
|
||||||
|
}
|
||||||
14
README.md
14
README.md
@@ -72,13 +72,25 @@ npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json
|
|||||||
npm run diagnose:area -- --config config/areas/nantaizi-lake-innovation-valley.json
|
npm run diagnose:area -- --config config/areas/nantaizi-lake-innovation-valley.json
|
||||||
```
|
```
|
||||||
|
|
||||||
|
每次手工编辑 OSM 后、进入重型构建前,先跑预检。它只读取区域配置和 OSM XML,检查
|
||||||
|
bounds、缺失 node 引用、building way 闭合性、building multipolygon member/ring、以及
|
||||||
|
显式 `height` / `building:levels` 标签:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run preflight:area -- --config config/areas/nantaizi-lake-innovation-valley.json
|
||||||
|
```
|
||||||
|
|
||||||
|
预检有 error 时退出非零且不会更新记录;通过后写 `preflight.manifest.json`,保留本次
|
||||||
|
验证的 config / OSM 文件摘要和检查结果。
|
||||||
|
|
||||||
诊断会输出 OSM bounds、building way / multipolygon relation、显式高度、植被数量、
|
诊断会输出 OSM bounds、building way / multipolygon relation、显式高度、植被数量、
|
||||||
现有产物状态,以及 GLB 的 size / nodes / meshes / materials / images / extensions。
|
现有产物状态,以及 GLB 的 size / nodes / meshes / materials / images / extensions。
|
||||||
缺少已期望的基线产物、异常 building relation、GLB 超过保守预算等会进入 `Warnings`。
|
缺少已期望的基线产物、异常 building relation、GLB 超过保守预算等会进入 `Warnings`。
|
||||||
|
|
||||||
每个成功的区域构建阶段都会写机器可读的 stage manifest:
|
预检和每个成功的区域构建阶段都会写机器可读的 manifest:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
outputs/<area-id>/_pipeline/stages/preflight.manifest.json
|
||||||
outputs/<area-id>/_pipeline/stages/intermediates.manifest.json
|
outputs/<area-id>/_pipeline/stages/intermediates.manifest.json
|
||||||
outputs/<area-id>/_pipeline/stages/reimport.manifest.json
|
outputs/<area-id>/_pipeline/stages/reimport.manifest.json
|
||||||
outputs/<area-id>/_pipeline/stages/blender.manifest.json
|
outputs/<area-id>/_pipeline/stages/blender.manifest.json
|
||||||
|
|||||||
@@ -2,6 +2,11 @@
|
|||||||
|
|
||||||
## 2026-08-04
|
## 2026-08-04
|
||||||
|
|
||||||
|
- 新增 `npm run preflight:area -- --config config/areas/<area>.json`:手工编辑 OSM 后可在
|
||||||
|
QGIS / Blender / Cesium 前检查 bounds、缺失 node、building way 闭合性、building
|
||||||
|
multipolygon member/ring 与显式高度。结构错误会非零退出;成功结果原子写入
|
||||||
|
`_pipeline/stages/preflight.manifest.json`,并由 `diagnose:area` / `check:area` 检查
|
||||||
|
新鲜度。该命令不属于 `build-area` 阶段,也不调用重型工具。
|
||||||
- 扩展 stage manifest 到完整区域链:`intermediates`、`reimport`、`blender` 和
|
- 扩展 stage manifest 到完整区域链:`intermediates`、`reimport`、`blender` 和
|
||||||
`preview` 现在和既有 `cesium` / `compress` 一样,在成功后写
|
`preview` 现在和既有 `cesium` / `compress` 一样,在成功后写
|
||||||
`_pipeline/stages/<stage>.manifest.json`。前段 manifest 记录 config / OSM /
|
`_pipeline/stages/<stage>.manifest.json`。前段 manifest 记录 config / OSM /
|
||||||
|
|||||||
@@ -9,7 +9,9 @@
|
|||||||
"build:qgis": "node scripts/build-osm2streets-qgis.js",
|
"build:qgis": "node scripts/build-osm2streets-qgis.js",
|
||||||
"check:area": "node scripts/check-area.js",
|
"check:area": "node scripts/check-area.js",
|
||||||
"compress:glb": "node scripts/compress-glb.js",
|
"compress:glb": "node scripts/compress-glb.js",
|
||||||
"diagnose:area": "node scripts/diagnose-area.js"
|
"diagnose:area": "node scripts/diagnose-area.js",
|
||||||
|
"preflight:area": "node scripts/preflight-area.js",
|
||||||
|
"test:preflight": "node scripts/test-area-preflight.js"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"osm2streets-js-node": "0.1.4"
|
"osm2streets-js-node": "0.1.4"
|
||||||
|
|||||||
@@ -67,6 +67,7 @@ function parseOsm(xml) {
|
|||||||
const nodePattern = /<node\b([^>]*?)\/>|<node\b([^>]*?)>([\s\S]*?)<\/node>/g;
|
const nodePattern = /<node\b([^>]*?)\/>|<node\b([^>]*?)>([\s\S]*?)<\/node>/g;
|
||||||
for (const match of xml.matchAll(nodePattern)) {
|
for (const match of xml.matchAll(nodePattern)) {
|
||||||
const attrs = xmlAttrs(match[1] || match[2] || "");
|
const attrs = xmlAttrs(match[1] || match[2] || "");
|
||||||
|
if (attrs.action === "delete") continue;
|
||||||
if (attrs.id) nodeIds.add(attrs.id);
|
if (attrs.id) nodeIds.add(attrs.id);
|
||||||
nodeStats.total += 1;
|
nodeStats.total += 1;
|
||||||
const tags = parseTags(match[3] || "");
|
const tags = parseTags(match[3] || "");
|
||||||
@@ -80,6 +81,8 @@ function parseOsm(xml) {
|
|||||||
buildingsWithHeight: 0,
|
buildingsWithHeight: 0,
|
||||||
buildingsWithLevels: 0,
|
buildingsWithLevels: 0,
|
||||||
buildingsWithBadHeight: 0,
|
buildingsWithBadHeight: 0,
|
||||||
|
buildingsWithBadLevels: 0,
|
||||||
|
buildingGeometryIssues: [],
|
||||||
missingNodeRefs: 0,
|
missingNodeRefs: 0,
|
||||||
grass: 0,
|
grass: 0,
|
||||||
scrub: 0,
|
scrub: 0,
|
||||||
@@ -87,6 +90,7 @@ function parseOsm(xml) {
|
|||||||
};
|
};
|
||||||
for (const match of xml.matchAll(/<way\b([^>]*)>([\s\S]*?)<\/way>/g)) {
|
for (const match of xml.matchAll(/<way\b([^>]*)>([\s\S]*?)<\/way>/g)) {
|
||||||
const attrs = xmlAttrs(match[1]);
|
const attrs = xmlAttrs(match[1]);
|
||||||
|
if (attrs.action === "delete") continue;
|
||||||
const body = match[2];
|
const body = match[2];
|
||||||
const tags = parseTags(body);
|
const tags = parseTags(body);
|
||||||
const refs = [];
|
const refs = [];
|
||||||
@@ -110,6 +114,14 @@ function parseOsm(xml) {
|
|||||||
if (isExplicitHeight(tags)) wayStats.buildingsWithHeight += 1;
|
if (isExplicitHeight(tags)) wayStats.buildingsWithHeight += 1;
|
||||||
if (tags["building:levels"]) wayStats.buildingsWithLevels += 1;
|
if (tags["building:levels"]) wayStats.buildingsWithLevels += 1;
|
||||||
if (tags.height && !parseHeightMeters(tags.height)) wayStats.buildingsWithBadHeight += 1;
|
if (tags.height && !parseHeightMeters(tags.height)) wayStats.buildingsWithBadHeight += 1;
|
||||||
|
if (tags["building:levels"] && !parseBuildingLevels(tags["building:levels"])) {
|
||||||
|
wayStats.buildingsWithBadLevels += 1;
|
||||||
|
}
|
||||||
|
if (refs.length < 4) {
|
||||||
|
wayStats.buildingGeometryIssues.push({ id: way.id, issue: "has fewer than 4 node refs" });
|
||||||
|
} else if (!way.closed) {
|
||||||
|
wayStats.buildingGeometryIssues.push({ id: way.id, issue: "is not closed" });
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if (tags.landuse === "grass") wayStats.grass += 1;
|
if (tags.landuse === "grass") wayStats.grass += 1;
|
||||||
if (tags.natural === "scrub") wayStats.scrub += 1;
|
if (tags.natural === "scrub") wayStats.scrub += 1;
|
||||||
@@ -122,11 +134,13 @@ function parseOsm(xml) {
|
|||||||
buildingsWithHeight: 0,
|
buildingsWithHeight: 0,
|
||||||
buildingsWithLevels: 0,
|
buildingsWithLevels: 0,
|
||||||
buildingsWithBadHeight: 0,
|
buildingsWithBadHeight: 0,
|
||||||
|
buildingsWithBadLevels: 0,
|
||||||
healthyBuildingMultipolygons: 0,
|
healthyBuildingMultipolygons: 0,
|
||||||
issues: [],
|
issues: [],
|
||||||
};
|
};
|
||||||
for (const match of xml.matchAll(/<relation\b([^>]*)>([\s\S]*?)<\/relation>/g)) {
|
for (const match of xml.matchAll(/<relation\b([^>]*)>([\s\S]*?)<\/relation>/g)) {
|
||||||
const attrs = xmlAttrs(match[1]);
|
const attrs = xmlAttrs(match[1]);
|
||||||
|
if (attrs.action === "delete") continue;
|
||||||
const body = match[2];
|
const body = match[2];
|
||||||
const tags = parseTags(body);
|
const tags = parseTags(body);
|
||||||
const members = [];
|
const members = [];
|
||||||
@@ -139,6 +153,9 @@ function parseOsm(xml) {
|
|||||||
if (isExplicitHeight(tags)) relationStats.buildingsWithHeight += 1;
|
if (isExplicitHeight(tags)) relationStats.buildingsWithHeight += 1;
|
||||||
if (tags["building:levels"]) relationStats.buildingsWithLevels += 1;
|
if (tags["building:levels"]) relationStats.buildingsWithLevels += 1;
|
||||||
if (tags.height && !parseHeightMeters(tags.height)) relationStats.buildingsWithBadHeight += 1;
|
if (tags.height && !parseHeightMeters(tags.height)) relationStats.buildingsWithBadHeight += 1;
|
||||||
|
if (tags["building:levels"] && !parseBuildingLevels(tags["building:levels"])) {
|
||||||
|
relationStats.buildingsWithBadLevels += 1;
|
||||||
|
}
|
||||||
|
|
||||||
const health = buildingRelationHealth(attrs.id || "", members, ways);
|
const health = buildingRelationHealth(attrs.id || "", members, ways);
|
||||||
if (health.ok) relationStats.healthyBuildingMultipolygons += 1;
|
if (health.ok) relationStats.healthyBuildingMultipolygons += 1;
|
||||||
@@ -163,7 +180,10 @@ function parseBounds(xml) {
|
|||||||
maxLon: Number(attrs.maxlon),
|
maxLon: Number(attrs.maxlon),
|
||||||
maxLat: Number(attrs.maxlat),
|
maxLat: Number(attrs.maxlat),
|
||||||
};
|
};
|
||||||
return Object.values(bounds).every(Number.isFinite) ? bounds : null;
|
return Object.values(bounds).every(Number.isFinite) &&
|
||||||
|
bounds.minLon < bounds.maxLon && bounds.minLat < bounds.maxLat
|
||||||
|
? bounds
|
||||||
|
: null;
|
||||||
}
|
}
|
||||||
|
|
||||||
function isExplicitHeight(tags) {
|
function isExplicitHeight(tags) {
|
||||||
@@ -177,6 +197,48 @@ function parseHeightMeters(value) {
|
|||||||
return Number.isFinite(height) && height > 0 ? height : null;
|
return Number.isFinite(height) && height > 0 ? height : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function parseBuildingLevels(value) {
|
||||||
|
const levels = Number(String(value).trim());
|
||||||
|
return Number.isFinite(levels) && levels > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
function analyzeOsmPreflight(osm) {
|
||||||
|
const errors = [];
|
||||||
|
const warnings = [];
|
||||||
|
if (!osm.bounds) errors.push("OSM has no valid <bounds>.");
|
||||||
|
if (osm.ways.missingNodeRefs) {
|
||||||
|
errors.push(`OSM ways reference ${osm.ways.missingNodeRefs} missing node(s).`);
|
||||||
|
}
|
||||||
|
for (const issue of osm.ways.buildingGeometryIssues) {
|
||||||
|
errors.push(`Building way ${issue.id}: ${issue.issue}.`);
|
||||||
|
}
|
||||||
|
if (osm.ways.buildingsWithBadHeight || osm.relations.buildingsWithBadHeight) {
|
||||||
|
errors.push("Some building height tags could not be parsed as positive meters.");
|
||||||
|
}
|
||||||
|
for (const issue of osm.relations.issues) {
|
||||||
|
errors.push(`Building relation ${issue.id}: ${issue.issues.join("; ")}.`);
|
||||||
|
}
|
||||||
|
const badLevels = osm.ways.buildingsWithBadLevels + osm.relations.buildingsWithBadLevels;
|
||||||
|
if (badLevels) {
|
||||||
|
warnings.push(`${badLevels} building:levels tag(s) could not be parsed as positive numbers.`);
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
errors: uniqueLines(errors),
|
||||||
|
warnings: uniqueLines(warnings),
|
||||||
|
summary: {
|
||||||
|
bounds: osm.bounds,
|
||||||
|
nodes: osm.nodes.total,
|
||||||
|
ways: osm.ways.total,
|
||||||
|
relations: osm.relations.total,
|
||||||
|
buildingWays: osm.ways.buildings,
|
||||||
|
buildingMultipolygons: osm.relations.buildingMultipolygons,
|
||||||
|
missingNodeRefs: osm.ways.missingNodeRefs,
|
||||||
|
buildingWayIssues: osm.ways.buildingGeometryIssues.length,
|
||||||
|
buildingRelationIssues: osm.relations.issues.length,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
function buildingRelationHealth(id, members, ways) {
|
function buildingRelationHealth(id, members, ways) {
|
||||||
const issues = [];
|
const issues = [];
|
||||||
const outerMembers = members.filter((member) => member.type === "way" && member.role === "outer");
|
const outerMembers = members.filter((member) => member.type === "way" && member.role === "outer");
|
||||||
@@ -307,6 +369,15 @@ function stageManifestStatus(area, configPath = null) {
|
|||||||
const hasReimportManifest = fs.existsSync(reimportManifest);
|
const hasReimportManifest = fs.existsSync(reimportManifest);
|
||||||
const derivedConfig = path.join(area.outputs.pipelineDir, "osm2streets-qgis.config.json");
|
const derivedConfig = path.join(area.outputs.pipelineDir, "osm2streets-qgis.config.json");
|
||||||
const stages = [
|
const stages = [
|
||||||
|
{
|
||||||
|
stage: "preflight",
|
||||||
|
expected: fs.existsSync(stageManifestPath(area, "preflight")),
|
||||||
|
inputs: {
|
||||||
|
...(configPath ? { config: configPath } : {}),
|
||||||
|
osm: area.input,
|
||||||
|
},
|
||||||
|
outputs: {},
|
||||||
|
},
|
||||||
{
|
{
|
||||||
stage: "intermediates",
|
stage: "intermediates",
|
||||||
expected: (
|
expected: (
|
||||||
@@ -763,6 +834,7 @@ function formatBytes(bytes) {
|
|||||||
|
|
||||||
module.exports = {
|
module.exports = {
|
||||||
BUDGETS,
|
BUDGETS,
|
||||||
|
analyzeOsmPreflight,
|
||||||
analyzeArea,
|
analyzeArea,
|
||||||
artifactStatus,
|
artifactStatus,
|
||||||
classifyAreaQuality,
|
classifyAreaQuality,
|
||||||
|
|||||||
99
scripts/preflight-area.js
Normal file
99
scripts/preflight-area.js
Normal file
@@ -0,0 +1,99 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
"use strict";
|
||||||
|
|
||||||
|
const fs = require("fs");
|
||||||
|
const path = require("path");
|
||||||
|
const { readAreaConfig } = require("./lib/area-config");
|
||||||
|
const {
|
||||||
|
analyzeOsmPreflight,
|
||||||
|
defaultConfigPath,
|
||||||
|
parseOsm,
|
||||||
|
} = require("./lib/area-diagnostics");
|
||||||
|
const { fileRecord, writeStageManifest } = require("./lib/stage-manifest");
|
||||||
|
|
||||||
|
const repoRoot = path.resolve(__dirname, "..");
|
||||||
|
|
||||||
|
function parseArgs(argv) {
|
||||||
|
const out = {};
|
||||||
|
for (let i = 0; i < argv.length; i += 1) {
|
||||||
|
const arg = argv[i];
|
||||||
|
if (!arg.startsWith("--")) continue;
|
||||||
|
const key = arg.slice(2).replace(/-([a-z])/g, (_, c) => c.toUpperCase());
|
||||||
|
const next = argv[i + 1];
|
||||||
|
if (!next || next.startsWith("--")) {
|
||||||
|
out[key] = "true";
|
||||||
|
} else {
|
||||||
|
out[key] = next;
|
||||||
|
i += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
function main() {
|
||||||
|
const startedAt = new Date().toISOString();
|
||||||
|
const args = parseArgs(process.argv.slice(2));
|
||||||
|
const configPath = path.resolve(args.config || defaultConfigPath(repoRoot));
|
||||||
|
const area = readAreaConfig(configPath, { repoRoot });
|
||||||
|
const osm = parseOsm(fs.readFileSync(area.input, "utf8"));
|
||||||
|
const preflight = analyzeOsmPreflight(osm);
|
||||||
|
|
||||||
|
printPreflightReport(area, configPath, preflight);
|
||||||
|
if (preflight.errors.length) {
|
||||||
|
process.exitCode = 1;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const finishedAt = new Date().toISOString();
|
||||||
|
writeStageManifest(area, {
|
||||||
|
stage: "preflight",
|
||||||
|
status: "ok",
|
||||||
|
config: configPath,
|
||||||
|
startedAt,
|
||||||
|
finishedAt,
|
||||||
|
durationMs: Date.parse(finishedAt) - Date.parse(startedAt),
|
||||||
|
inputs: {
|
||||||
|
config: fileRecord(configPath),
|
||||||
|
osm: fileRecord(area.input),
|
||||||
|
},
|
||||||
|
outputs: {},
|
||||||
|
summary: preflight.summary,
|
||||||
|
errors: preflight.errors,
|
||||||
|
warnings: preflight.warnings,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function printPreflightReport(area, configPath, preflight) {
|
||||||
|
console.log("OSM area preflight");
|
||||||
|
console.log(`Area: ${area.id}`);
|
||||||
|
console.log(`Config: ${configPath}`);
|
||||||
|
console.log(`Input: ${area.input}`);
|
||||||
|
console.log("");
|
||||||
|
console.log(
|
||||||
|
`OSM: ${preflight.summary.nodes} nodes, ${preflight.summary.ways} ways, ` +
|
||||||
|
`${preflight.summary.relations} relations`,
|
||||||
|
);
|
||||||
|
console.log(
|
||||||
|
`Buildings: ${preflight.summary.buildingWays} way(s), ` +
|
||||||
|
`${preflight.summary.buildingMultipolygons} multipolygon relation(s)`,
|
||||||
|
);
|
||||||
|
console.log(`Bounds: ${preflight.summary.bounds ? "valid" : "invalid or missing"}`);
|
||||||
|
console.log("");
|
||||||
|
const status = preflight.errors.length ? "FAIL" : "PASS";
|
||||||
|
console.log(`${status}: ${preflight.errors.length} error(s), ${preflight.warnings.length} warning(s)`);
|
||||||
|
console.log("");
|
||||||
|
printLines("Errors", preflight.errors);
|
||||||
|
console.log("");
|
||||||
|
printLines("Warnings", preflight.warnings);
|
||||||
|
}
|
||||||
|
|
||||||
|
function printLines(label, lines) {
|
||||||
|
console.log(`${label} (${lines.length})`);
|
||||||
|
if (!lines.length) {
|
||||||
|
console.log(" none");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
for (const line of lines) console.log(` - ${line}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
main();
|
||||||
34
scripts/test-area-preflight.js
Normal file
34
scripts/test-area-preflight.js
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
"use strict";
|
||||||
|
|
||||||
|
const assert = require("assert");
|
||||||
|
const { analyzeOsmPreflight, parseOsm } = require("./lib/area-diagnostics");
|
||||||
|
|
||||||
|
function osm({ bounds = '<bounds minlon="0" minlat="0" maxlon="1" maxlat="1"/>', nodes, ways, relations = "" }) {
|
||||||
|
return `<osm>${bounds}${nodes}${ways}${relations}</osm>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function nodes(ids) {
|
||||||
|
return ids.map((id) => `<node id="${id}" lat="0" lon="0"/>`).join("");
|
||||||
|
}
|
||||||
|
|
||||||
|
function way(id, refs, tags = '<tag k="building" v="yes"/>') {
|
||||||
|
return `<way id="${id}">${refs.map((ref) => `<nd ref="${ref}"/>`).join("")}${tags}</way>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function preflight(input) {
|
||||||
|
return analyzeOsmPreflight(parseOsm(input));
|
||||||
|
}
|
||||||
|
|
||||||
|
assert.equal(preflight(osm({ nodes: nodes([1, 2, 3]), ways: way(10, [1, 2, 3, 1]) })).errors.length, 0);
|
||||||
|
assert.match(preflight(osm({ bounds: "", nodes: nodes([1, 2, 3]), ways: way(10, [1, 2, 3, 1]) })).errors.join("\n"), /bounds/);
|
||||||
|
assert.match(preflight(osm({ bounds: '<bounds minlon="1" minlat="0" maxlon="0" maxlat="1"/>', nodes: nodes([1, 2, 3]), ways: way(10, [1, 2, 3, 1]) })).errors.join("\n"), /bounds/);
|
||||||
|
assert.match(preflight(osm({ nodes: nodes([1, 2, 3]), ways: way(10, [1, 2, 9, 1]) })).errors.join("\n"), /missing node/);
|
||||||
|
assert.match(preflight(osm({ nodes: nodes([1, 2, 3]), ways: way(10, [1, 2, 3, 2]) })).errors.join("\n"), /not closed/);
|
||||||
|
assert.match(preflight(osm({ nodes: nodes([1, 2, 3]), ways: way(10, [1, 2, 3, 1], '<tag k="building" v="yes"/><tag k="height" v="many"/>') })).errors.join("\n"), /height/);
|
||||||
|
assert.equal(preflight(osm({ nodes: nodes([1, 2, 3]), ways: '<way id="10" action="delete"><tag k="building" v="yes"/></way>' })).errors.length, 0);
|
||||||
|
|
||||||
|
const brokenRelation = '<relation id="20"><member type="way" ref="99" role="outer"/><tag k="type" v="multipolygon"/><tag k="building" v="yes"/></relation>';
|
||||||
|
assert.match(preflight(osm({ nodes: nodes([1, 2, 3]), ways: "", relations: brokenRelation })).errors.join("\n"), /Building relation 20/);
|
||||||
|
|
||||||
|
console.log("OSM preflight tests passed.");
|
||||||
Reference in New Issue
Block a user