feat(pipeline): make compressed assets the default delivery

This commit is contained in:
2026-08-11 15:30:04 +08:00
parent b68be063ad
commit 0790cbd0a9
11 changed files with 143 additions and 152 deletions

View File

@@ -54,7 +54,7 @@ cp config/examples/template.json config/areas/my-area.json
| `turnLaneArrows` | | 见下 | 从 OSM `turn:lanes:*` 生成自定义车道箭头的发布开关 | | `turnLaneArrows` | | 见下 | 从 OSM `turn:lanes:*` 生成自定义车道箭头的发布开关 |
| `osm2streets` | | 见下 | 透传给 osm2streets 的选项 | | `osm2streets` | | 见下 | 透传给 osm2streets 的选项 |
| `blender` | | 见下 | Blender 侧选项 | | `blender` | | 见下 | Blender 侧选项 |
| `compress` | | 见下 | 显式 `compress` 阶段的 GLB 压缩选项 | | `compress` | | 见下 | 默认交付压缩阶段的 GLB 压缩选项 |
| `budget` | | 见下 | 区域 GLB 性能与体量预算 | | `budget` | | 见下 | 区域 GLB 性能与体量预算 |
| `outputs` | | 从 `id` 推导 | 输出路径覆盖,逃生舱 | | `outputs` | | 从 `id` 推导 | 输出路径覆盖,逃生舱 |
@@ -69,11 +69,10 @@ cp config/examples/template.json config/areas/my-area.json
| `blender` | `true` | | | `blender` | `true` | |
| `cesium` | `true` | | | `cesium` | `true` | |
`reimport``preview``compress` **在这里配也没用**——`normalizeAreaConfig` 把它们 `reimport``preview` **在这里配也没用**——`normalizeAreaConfig` 把它们
硬编码为 `false`,只能靠 `--stages` 显式请求。 硬编码为 `false`,只能靠 `--stages` 显式请求。`compress` 始终默认开启。
> 恢复动作reimport补丁动作preview和替代产物动作compress不该被一份 > 恢复动作reimport补丁动作preview不该被一份配置文件变成默认行为;压缩是标准交付链的一部分。
> 配置文件变成默认行为。
`--stages` 会整体覆盖这里的默认值。 `--stages` 会整体覆盖这里的默认值。
@@ -133,8 +132,8 @@ cp config/examples/template.json config/areas/my-area.json
### `compress` ### `compress`
只影响显式 `--stages compress`。默认压缩链是 texture resize + WebP transcode 完整构建和显式 `--stages compress` 都使用此配置。默认压缩链是 texture resize + WebP
不覆盖默认 `<area-id>.glb` transcode成功后覆盖标准 `<area-id>.glb``.json` 和预览 HTML未压缩源只保留在构建临时目录
| 字段 | 默认 | 说明 | | 字段 | 默认 | 说明 |
|---|---|---| |---|---|---|
@@ -177,7 +176,6 @@ cp config/examples/template.json config/areas/my-area.json
可覆盖的键(`scripts/lib/area-config.js``areaDir``fileStem``geojsonDir``gpkg` 可覆盖的键(`scripts/lib/area-config.js``areaDir``fileStem``geojsonDir``gpkg`
`qgisProject``qgisPreview``blend``render``glb``metadata``cesiumPreview` `qgisProject``qgisPreview``blend``render``glb``metadata``cesiumPreview`
`compressedFileStem``compressedGlb``compressedMetadata``compressedCesiumPreview`
`vehicleRoute``vehicleModel``pipelineDir``stageManifestDir` `vehicleRoute``vehicleModel``pipelineDir``stageManifestDir`
**优先改 `fileStem` 或 `areaDir`**——它们能一次性影响全部派生路径。逐个覆盖容易漏。 **优先改 `fileStem` 或 `areaDir`**——它们能一次性影响全部派生路径。逐个覆盖容易漏。
@@ -221,7 +219,7 @@ cp config/examples/template.json config/areas/my-area.json
| 布尔字段用 `\|\|` 兜底 | `false` 被翻转 | | 布尔字段用 `\|\|` 兜底 | `false` 被翻转 |
| 给新字段造顶层平铺别名 | 扩大历史包袱 | | 给新字段造顶层平铺别名 | 扩大历史包袱 |
| 逐个覆盖 `outputs` 而不用 `fileStem` | 漏掉某个产物路径 | | 逐个覆盖 `outputs` 而不用 `fileStem` | 漏掉某个产物路径 |
| 在 `stages` 里配 `reimport` / `preview` / `compress` | 无效,被硬编码为 false | | 在 `stages` 里配 `reimport` / `preview` | 无效,被硬编码为 false |
| 加数值字段不做范围校验 | 错配置在中途才崩,输出已被破坏 | | 加数值字段不做范围校验 | 错配置在中途才崩,输出已被破坏 |
--- ---

View File

@@ -21,13 +21,13 @@
全部是 CommonJS`package.json``"type": "commonjs"`),无构建步骤、无 TypeScript、 全部是 CommonJS`package.json``"type": "commonjs"`),无构建步骤、无 TypeScript、
零运行时依赖(唯一依赖 `osm2streets-js-node` 只被 `build-osm2streets-qgis.js` 用)。 零运行时依赖(唯一依赖 `osm2streets-js-node` 只被 `build-osm2streets-qgis.js` 用)。
## 可选 GLB 压缩阶段 ## 默认 GLB 压缩阶段
### 1. Scope / Trigger ### 1. Scope / Trigger
`compress`显式请求的替代产物阶段,不属于 `all`,也不覆盖默认 `<area-id>.glb` `compress`完整构建和 `all` 的默认末端阶段。它在已有 Cesium GLB 上执行 texture
它用于在已有 Cesium GLB 上生成并列压缩 GLB、metadata 和预览页。当前主路径是 resize + WebP transcode并以标准 `<area-id>.glb``.json` 和预览 HTML 替换未压缩导出。
texture resize + WebP transcode 未压缩版本只在 `<areaDir>/_pipeline/compress-*` 临时目录中存在,成功或失败后都会清理
`scripts/compress-glb.js` 是该阶段调用的低层脚本,也可单独运行做实验。 `scripts/compress-glb.js` 是该阶段调用的低层脚本,也可单独运行做实验。
@@ -62,15 +62,14 @@ npm run compress:glb -- --input in.glb --output out.glb [options]
### 3. Contracts ### 3. Contracts
- 输入必须是现有 `.glb` 文件;`--output` 必须不同于 `--input` - 输入必须是现有 `.glb` 文件;`--output` 必须不同于 `--input`
- 输出是并列压缩 GLB默认构建产物不被替换 - 低层脚本仍输出到与输入不同的路径;`build-area.js` 使用临时输入/输出路径,避免原地压缩
- `compress` 阶段依赖默认 `glb``metadata``cesiumPreview` 已存在 - `compress` 阶段依赖标准 `glb``metadata``cesiumPreview` 已存在,并在全部临时交付文件
生成和校验后以 `renameSync` 替换它们。
- 默认链固定为 `gltf-transform resize -> gltf-transform webp` - 默认链固定为 `gltf-transform resize -> gltf-transform webp`
- `compress` 阶段默认 `textureSize=768`;低层脚本单独运行时默认 `--texture-size 1024` - `compress` 阶段默认 `textureSize=768`;低层脚本单独运行时默认 `--texture-size 1024`
- `build-area.js` 默认写: - `build-area.js` 最终只保留 `<fileStem>.glb``<fileStem>.json`
- `<fileStem>-compressed-webp768.glb` `<fileStem>-cesium-preview.html`;不会写并列 `-compressed-*` 交付物。
- `<fileStem>-compressed-webp768.json` - 伴生 metadata 的 `asset``id="main"` 资产 URL 保持为标准 GLB 文件名;其余资产(包括
- `<fileStem>-compressed-webp768-cesium-preview.html`
- 伴生 metadata 的 `asset``id="main"` 资产 URL 改为压缩 GLB 文件名;其余资产(包括
`category="semantic"` 的 Cesium 分类检查 GLB必须原样保留。 `category="semantic"` 的 Cesium 分类检查 GLB必须原样保留。
- preview HTML 只替换 `window.OSM_ASSET_PREVIEW_CONFIG``glbName` / - preview HTML 只替换 `window.OSM_ASSET_PREVIEW_CONFIG``glbName` /
`metadataName` 和 loading 文案,不改 preview runtime。 `metadataName` 和 loading 文案,不改 preview runtime。
@@ -88,13 +87,12 @@ npm run compress:glb -- --input in.glb --output out.glb [options]
| `gltf-transform` 退出非零 | 抛错并带上 status / signal | | `gltf-transform` 退出非零 | 抛错并带上 status / signal |
| `--preview` 没有 `--metadata` | 抛错,因为 preview 必须指向存在的 metadata | | `--preview` 没有 `--metadata` | 抛错,因为 preview 必须指向存在的 metadata |
| preview HTML 找不到配置块 | 抛错,不做猜测替换 | | preview HTML 找不到配置块 | 抛错,不做猜测替换 |
| `--stages compress`默认 preview 不存在 | `Cesium preview not found: <path>` | | `--stages compress`标准 preview 不存在 | `Cesium preview not found: <path>` |
### 5. Good/Base/Bad Cases ### 5. Good/Base/Bad Cases
- Good: `--stages cesium,compress` 先重导默认 GLB再生成并列压缩产物。 - Good: `all` 或默认构建先导出 Cesium再压缩成标准交付物。
- Good: `--texture-size 768 --metadata --preview` 生成压缩 GLB、metadata、HTML - Good: `--stages compress` 对已有标准交付物重新压缩,临时源 GLB 不会留在输出目录。
源 GLB 保持不变。
- Base: 只传 `--input --output` 生成压缩 GLB不生成伴生文件。 - Base: 只传 `--input --output` 生成压缩 GLB不生成伴生文件。
- Bad: 使用 `--meshopt` 后没有做 Cesium 兼容性验证就当默认产物发布。 - Bad: 使用 `--meshopt` 后没有做 Cesium 兼容性验证就当默认产物发布。
@@ -104,9 +102,9 @@ npm run compress:glb -- --input in.glb --output out.glb [options]
- `npm run test:compress-glb`:断言压缩 metadata 只替换主资产,不丢失语义资产。 - `npm run test:compress-glb`:断言压缩 metadata 只替换主资产,不丢失语义资产。
- `node --check scripts/build-area.js` - `node --check scripts/build-area.js`
- 对目标区域跑一次 `npm run compress:glb -- ... --metadata --preview` - 对目标区域跑一次 `npm run compress:glb -- ... --metadata --preview`
- 对目标区域跑一次 `npm run build:area -- --stages compress` - 对目标区域跑一次默认 `npm run build:area`
- `node scripts/glb-digest.js <compressed.glb>` 确认可解析结构和扩展 - `node scripts/glb-digest.js <area>.glb` 确认可解析结构和扩展
- 浏览器/Cesium 预览压缩 HTML确认 `EXT_texture_webp` 在目标环境可加载 - 浏览器/Cesium 预览标准 HTML确认 `EXT_texture_webp` 在目标环境可加载
### 7. Wrong vs Correct ### 7. Wrong vs Correct
@@ -116,10 +114,10 @@ Wrong:
npm run compress:glb -- --input outputs/a/a.glb --output outputs/a/a.glb npm run compress:glb -- --input outputs/a/a.glb --output outputs/a/a.glb
``` ```
Correct: Correct for the low-level script:
```bash ```bash
npm run compress:glb -- --input outputs/a/a.glb --output outputs/a/a-compressed-webp768.glb npm run compress:glb -- --input outputs/a/a.glb --output /tmp/a-compressed.glb
``` ```
--- ---
@@ -758,14 +756,11 @@ runtime 与 HTML并写 preview manifest。预览内容实现不得回流到
`compress` manifest: `compress` manifest:
- `inputs.glb` - `outputs.glb`
- `inputs.metadata` - `outputs.metadata`
- `inputs.cesiumPreview` - `outputs.cesiumPreview`
- `outputs.compressedGlb`
- `outputs.compressedMetadata`
- `outputs.compressedCesiumPreview`
- `summary.sourceGlb` - `summary.sourceGlb`
- `summary.compressedGlb` - `summary.glb`
- `summary.options` - `summary.options`
- `summary.compressionRatio` - `summary.compressionRatio`
- `summary.savedBytes` - `summary.savedBytes`
@@ -931,7 +926,7 @@ gpkg: path.resolve(outputOverrides.gpkg || path.join(areaDir, `${fileStem}.gpkg`
| `blender` | OSM + GeoJSON → 场景 | `.osm``osm2streets_web_out/` | `.blend``.png` | | `blender` | OSM + GeoJSON → 场景 | `.osm``osm2streets_web_out/` | `.blend``.png` |
| `cesium` | 场景 → GLB + 元数据 + 预览页 | `.blend` | `.glb``.json`、预览 HTML 及其静态资源 | | `cesium` | 场景 → GLB + 元数据 + 预览页 | `.blend` | `.glb``.json`、预览 HTML 及其静态资源 |
| `preview` | 只补生成预览页 | `.glb``.json` | 预览 HTML 及其静态资源 | | `preview` | 只补生成预览页 | `.glb``.json` | 预览 HTML 及其静态资源 |
| `compress` | 生成并列压缩 Cesium 产物 | `.glb``.json`默认预览 HTML | `-compressed-webp*.glb/json/html` | | `compress` | Cesium 临时导出替换为压缩交付产物 | 标准 `.glb``.json`、预览 HTML | 标准 `.glb/json/html` |
调度是顶层的阶段 `if``build-area.js` 开头),顺序固定,**阶段之间不传内存状态, 调度是顶层的阶段 `if``build-area.js` 开头),顺序固定,**阶段之间不传内存状态,
只通过磁盘产物耦合**。这就是单跑某个阶段能work 的原因。 只通过磁盘产物耦合**。这就是单跑某个阶段能work 的原因。
@@ -952,11 +947,11 @@ gpkg: path.resolve(outputOverrides.gpkg || path.join(areaDir, `${fileStem}.gpkg`
```js ```js
// 'reimport' is deliberately absent from 'all': it is a recovery step for // 'reimport' is deliberately absent from 'all': it is a recovery step for
// hand-edited GeoPackages, never part of a full build. build-area.js:159-160 // hand-edited GeoPackages, never part of a full build. build-area.js:159-160
all: ["intermediates", "blender", "cesium"], all: ["intermediates", "blender", "cesium", "compress"],
``` ```
`preview` 同样不在 `all` 里——`cesium` 已经包含它。`compress` 也不`all`—— `preview` 不在 `all` 里——`cesium` 已经包含它。`compress``all`,确保完整构建
它生成的是替代压缩产物,不是 baseline GLB 以压缩后的标准路径交付
### `intermediates` 与 `reimport` 互斥 ### `intermediates` 与 `reimport` 互斥
@@ -966,9 +961,9 @@ all: ["intermediates", "blender", "cesium"],
这是**显式拒绝而不是警告**——两者同时开,无论谁先跑,另一个的工作都白做。 这是**显式拒绝而不是警告**——两者同时开,无论谁先跑,另一个的工作都白做。
`normalizeAreaConfig``stages.reimport``stages.preview``stages.compress` 硬编码为 `false` `normalizeAreaConfig``stages.reimport``stages.preview` 硬编码为 `false`
**不能从配置文件打开**只能靠 `--stages` 显式请求。 只能靠 `--stages` 显式请求`stages.compress` 为默认 `true`
恢复动作补丁动作和替代产物动作都不该被一份配置文件变成默认行为。 恢复动作补丁动作不该被一份配置文件变成默认行为,压缩则属于标准交付链
--- ---

View File

@@ -60,7 +60,7 @@ config/areas/<id>.json
│ + 车辆巡航路线与模型 │ + 车辆巡航路线与模型
│ → _pipeline/stages/preview.manifest.json │ → _pipeline/stages/preview.manifest.json
└─[compress]──────▶ 生成并列压缩 GLB / metadata / preview └─[compress]──────▶ 临时保存未压缩导出,再以标准 GLB / metadata / preview 路径交付压缩版本
→ _pipeline/stages/compress.manifest.json → _pipeline/stages/compress.manifest.json
``` ```

View File

@@ -8,11 +8,10 @@
- `<area-id>.blend`Blender 场景,包含道路、建筑、水体、植被等 - `<area-id>.blend`Blender 场景,包含道路、建筑、水体、植被等
- `<area-id>.png`Blender 预览渲染 - `<area-id>.png`Blender 预览渲染
- `<area-id>.glb`Cesium 可加载的完整 3D 模型(基线产物) - `<area-id>.glb`Cesium 可加载的完整 3D 模型(默认压缩交付产物)
- `<area-id>-roads.glb``-buildings.glb``-vegetation.glb``-water.glb`Cesium 分类检查用的辅助模型 - `<area-id>-roads.glb``-buildings.glb``-vegetation.glb``-water.glb`Cesium 分类检查用的辅助模型
- `<area-id>.json`Cesium 放置元数据和示例代码 - `<area-id>.json`Cesium 放置元数据和示例代码
- `<area-id>-cesium-preview.html`Cesium 本地预览页 - `<area-id>-cesium-preview.html`Cesium 本地预览页
- `<area-id>-compressed-webp768.glb/json/html`:显式 `compress` 阶段生成的可选压缩预览产物
- `osm2streets_web_out/`osm2streets GeoJSON 中间层 - `osm2streets_web_out/`osm2streets GeoJSON 中间层
- `<area-id>.gpkg` / `<area-id>.qgz` / `<area-id>-preview.png`QGIS 调试资产 - `<area-id>.gpkg` / `<area-id>.qgz` / `<area-id>-preview.png`QGIS 调试资产
@@ -62,15 +61,14 @@ npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json
npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages reimport npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages reimport
``` ```
`intermediates` 会生成 osm2streets GeoJSON、GeoPackage、QGIS 工程和 QGIS 预览图。`blender` 使用 OSM 和 osm2streets GeoJSON 生成 `.blend`/`.png``cesium``.blend` 导出 `.glb`/`.json`,并生成 Cesium 预览 HTML。`preview` 只在已有 `.glb/.json` 时补生成 HTML。`compress` 已有 `.glb/.json/html` 生成并列压缩产物,不覆盖默认 GLB`reimport` 把手工编辑过的 GeoPackage 回导为 GeoJSON不含在 `all` 里,详见 [QGIS 手工修正工作流](#qgis-手工修正工作流)。 `intermediates` 会生成 osm2streets GeoJSON、GeoPackage、QGIS 工程和 QGIS 预览图。`blender` 使用 OSM 和 osm2streets GeoJSON 生成 `.blend`/`.png``cesium``.blend` 导出 `.glb`/`.json`,并生成 Cesium 预览 HTML。完整构建会接着运行 `compress`,以标准 `.glb/.json/html` 名称交付压缩版本;未压缩导出仅存在于构建时临时目录。`preview` 只在已有 `.glb/.json` 时补生成 HTML`reimport` 把手工编辑过的 GeoPackage 回导为 GeoJSON不含在 `all` 里,详见 [QGIS 手工修正工作流](#qgis-手工修正工作流)。
Cesium 预览默认显示完整场景。点击 `Inspect` 后会按道路、建筑、绿化与设施、水体加载辅助 GLB该模式用于单独检查生成结果主 GLB 仍是完整场景和下游使用的基线 Cesium 预览默认显示完整场景。点击 `Inspect` 后会按道路、建筑、绿化与设施、水体加载辅助 GLB该模式用于单独检查生成结果主 GLB 是压缩后的下游交付资产
压缩预览同样保留这些分类检查资产;重跑 `compress` 后应打开新生成的压缩预览 HTML。
`compress` 含在 `all` 里,也不能从配置文件默认开启。需要重导出 Cesium 后立刻生成压缩产物时,显式跑: `compress` 含在完整构建的默认阶段和 `all` 中。需要对已有标准产物重新压缩时,可以显式跑:
```bash ```bash
npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages cesium,compress npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages compress
``` ```
## 区域诊断 ## 区域诊断
@@ -211,8 +209,7 @@ QGIS road-layer knobs:
"areaDir": "/absolute/path/to/custom-area", "areaDir": "/absolute/path/to/custom-area",
"blend": "/absolute/path/to/custom.blend", "blend": "/absolute/path/to/custom.blend",
"glb": "/absolute/path/to/custom.glb", "glb": "/absolute/path/to/custom.glb",
"cesiumPreview": "/absolute/path/to/custom-preview.html", "cesiumPreview": "/absolute/path/to/custom-preview.html"
"compressedGlb": "/absolute/path/to/custom-compressed.glb"
} }
} }
``` ```
@@ -277,8 +274,8 @@ Cesium GLB 低层导出:
## 压缩 GLB ## 压缩 GLB
推荐使用显式 `compress` 阶段。它不会覆盖默认 `<area-id>.glb`,只生成并列的压缩 GLB、 完整构建默认运行 `compress`。它将未压缩 Cesium 导出复制到 `_pipeline` 下的临时目录,
metadata 和预览页 压缩成功后才替换标准 `<area-id>.glb``.json` 和预览页;最终输出目录不保留并列压缩文件
```bash ```bash
npm run build:area -- \ npm run build:area -- \
@@ -291,7 +288,7 @@ npm run build:area -- \
```bash ```bash
npm run compress:glb -- \ npm run compress:glb -- \
--input outputs/nantaizi-lake-innovation-valley/nantaizi-lake-innovation-valley.glb \ --input outputs/nantaizi-lake-innovation-valley/nantaizi-lake-innovation-valley.glb \
--output outputs/nantaizi-lake-innovation-valley/nantaizi-lake-innovation-valley-compressed-webp768.glb \ --output /tmp/nantaizi-lake-innovation-valley-compressed.glb \
--texture-size 768 \ --texture-size 768 \
--metadata outputs/nantaizi-lake-innovation-valley/nantaizi-lake-innovation-valley.json \ --metadata outputs/nantaizi-lake-innovation-valley/nantaizi-lake-innovation-valley.json \
--preview outputs/nantaizi-lake-innovation-valley/nantaizi-lake-innovation-valley-cesium-preview.html --preview outputs/nantaizi-lake-innovation-valley/nantaizi-lake-innovation-valley-cesium-preview.html

View File

@@ -347,42 +347,51 @@ function compressCesiumGlb(area) {
ensureFile(area.outputs.metadata, "Cesium metadata"); ensureFile(area.outputs.metadata, "Cesium metadata");
ensureFile(area.outputs.cesiumPreview, "Cesium preview"); ensureFile(area.outputs.cesiumPreview, "Cesium preview");
ensureFile(path.join(repoRoot, "scripts", "compress-glb.js"), "GLB compressor"); ensureFile(path.join(repoRoot, "scripts", "compress-glb.js"), "GLB compressor");
fs.mkdirSync(path.dirname(area.outputs.compressedGlb), { recursive: true });
fs.mkdirSync(path.dirname(area.outputs.compressedMetadata), { recursive: true });
fs.mkdirSync(path.dirname(area.outputs.compressedCesiumPreview), { recursive: true });
const compressArgs = [
path.join(repoRoot, "scripts", "compress-glb.js"),
"--input",
area.outputs.glb,
"--output",
area.outputs.compressedGlb,
"--texture-size",
String(area.compress.textureSize),
"--quality",
String(area.compress.quality),
"--effort",
String(area.compress.effort),
"--metadata",
area.outputs.metadata,
"--metadata-output",
area.outputs.compressedMetadata,
"--preview",
area.outputs.cesiumPreview,
"--preview-output",
area.outputs.compressedCesiumPreview,
];
if (area.compress.meshopt) {
compressArgs.push("--meshopt");
}
console.log("Stage: compress (Cesium GLB texture resize + WebP)");
const started = Date.now(); const started = Date.now();
const startedAt = new Date(started).toISOString(); const startedAt = new Date(started).toISOString();
fs.mkdirSync(area.outputs.pipelineDir, { recursive: true });
const temporaryDir = fs.mkdtempSync(path.join(area.outputs.pipelineDir, "compress-"));
// The compressor rejects an in-place transform. Preserve the exporter output
// in a temporary directory, then atomically promote the compressed delivery.
const sourceDir = path.join(temporaryDir, "source");
const outputDir = path.join(temporaryDir, "delivery");
const stagedSourceGlb = path.join(sourceDir, path.basename(area.outputs.glb));
const stagedSourceMetadata = path.join(sourceDir, path.basename(area.outputs.metadata));
const stagedSourcePreview = path.join(sourceDir, path.basename(area.outputs.cesiumPreview));
const stagedDeliveryGlb = path.join(outputDir, path.basename(area.outputs.glb));
const stagedDeliveryMetadata = path.join(outputDir, path.basename(area.outputs.metadata));
const stagedDeliveryPreview = path.join(outputDir, path.basename(area.outputs.cesiumPreview));
try {
fs.mkdirSync(sourceDir, { recursive: true });
fs.copyFileSync(area.outputs.glb, stagedSourceGlb);
fs.copyFileSync(area.outputs.metadata, stagedSourceMetadata);
fs.copyFileSync(area.outputs.cesiumPreview, stagedSourcePreview);
const sourceDigest = glbDigest(stagedSourceGlb);
const compressArgs = [
path.join(repoRoot, "scripts", "compress-glb.js"),
"--input", stagedSourceGlb,
"--output", stagedDeliveryGlb,
"--texture-size", String(area.compress.textureSize),
"--quality", String(area.compress.quality),
"--effort", String(area.compress.effort),
"--metadata", stagedSourceMetadata,
"--metadata-output", stagedDeliveryMetadata,
"--preview", stagedSourcePreview,
"--preview-output", stagedDeliveryPreview,
];
if (area.compress.meshopt) compressArgs.push("--meshopt");
console.log("Stage: compress (Cesium GLB texture resize + WebP)");
runCommand(process.execPath, compressArgs, "compress"); runCommand(process.execPath, compressArgs, "compress");
ensureFile(stagedDeliveryGlb, "Compressed delivery GLB");
ensureFile(stagedDeliveryMetadata, "Compressed delivery metadata");
ensureFile(stagedDeliveryPreview, "Compressed delivery preview");
const compressedDigest = glbDigest(stagedDeliveryGlb);
fs.renameSync(stagedDeliveryGlb, area.outputs.glb);
fs.renameSync(stagedDeliveryMetadata, area.outputs.metadata);
fs.renameSync(stagedDeliveryPreview, area.outputs.cesiumPreview);
const finished = Date.now(); const finished = Date.now();
const sourceDigest = glbDigest(area.outputs.glb);
const compressedDigest = glbDigest(area.outputs.compressedGlb);
writeStageManifest(area, { writeStageManifest(area, {
stage: "compress", stage: "compress",
status: "ok", status: "ok",
@@ -390,19 +399,15 @@ function compressCesiumGlb(area) {
startedAt, startedAt,
finishedAt: new Date(finished).toISOString(), finishedAt: new Date(finished).toISOString(),
durationMs: finished - started, durationMs: finished - started,
inputs: { inputs: {},
outputs: {
glb: fileRecord(area.outputs.glb), glb: fileRecord(area.outputs.glb),
metadata: fileRecord(area.outputs.metadata), metadata: fileRecord(area.outputs.metadata),
cesiumPreview: fileRecord(area.outputs.cesiumPreview), cesiumPreview: fileRecord(area.outputs.cesiumPreview),
}, },
outputs: {
compressedGlb: fileRecord(area.outputs.compressedGlb),
compressedMetadata: fileRecord(area.outputs.compressedMetadata),
compressedCesiumPreview: fileRecord(area.outputs.compressedCesiumPreview),
},
summary: { summary: {
sourceGlb: glbSummary(sourceDigest), sourceGlb: glbSummary(sourceDigest),
compressedGlb: glbSummary(compressedDigest), glb: glbSummary(compressedDigest),
budget: evaluateGlbBudget(compressedDigest, area.budget), budget: evaluateGlbBudget(compressedDigest, area.budget),
options: { options: {
textureSize: area.compress.textureSize, textureSize: area.compress.textureSize,
@@ -415,6 +420,9 @@ function compressCesiumGlb(area) {
}, },
warnings: glbBudgetWarnings("Compressed", compressedDigest, area.budget), warnings: glbBudgetWarnings("Compressed", compressedDigest, area.budget),
}); });
} finally {
fs.rmSync(temporaryDir, { recursive: true, force: true });
}
} }
function blenderExecutable(area) { function blenderExecutable(area) {

View File

@@ -1,8 +1,8 @@
#!/usr/bin/env node #!/usr/bin/env node
"use strict"; "use strict";
// Experimental GLB compression wrapper. Keeps the source GLB intact and writes // GLB compression wrapper. The caller supplies distinct input and output files;
// a separate compressed artifact for visual comparison in Cesium. // build-area.js uses staging files before promoting the compressed delivery.
const fs = require("fs"); const fs = require("fs");
const os = require("os"); const os = require("os");

View File

@@ -25,8 +25,6 @@ function normalizeAreaConfig(raw, options = {}) {
const fileStem = outputOverrides.fileStem || id; const fileStem = outputOverrides.fileStem || id;
const compress = normalizeCompressConfig(raw.compress); const compress = normalizeCompressConfig(raw.compress);
const budget = normalizeBudgetConfig(raw.budget); const budget = normalizeBudgetConfig(raw.budget);
const compressedFileStem = outputOverrides.compressedFileStem ||
`${fileStem}-compressed-webp${compress.textureSize}${compress.meshopt ? "-meshopt" : ""}`;
const pipelineDir = path.resolve(outputOverrides.pipelineDir || path.join(areaDir, "_pipeline")); const pipelineDir = path.resolve(outputOverrides.pipelineDir || path.join(areaDir, "_pipeline"));
const geojsonDir = path.resolve(outputOverrides.geojsonDir || path.join(areaDir, "osm2streets_web_out")); const geojsonDir = path.resolve(outputOverrides.geojsonDir || path.join(areaDir, "osm2streets_web_out"));
const outputs = { const outputs = {
@@ -47,15 +45,6 @@ function normalizeAreaConfig(raw, options = {}) {
cesiumPreview: path.resolve( cesiumPreview: path.resolve(
outputOverrides.cesiumPreview || path.join(areaDir, `${fileStem}-cesium-preview.html`), outputOverrides.cesiumPreview || path.join(areaDir, `${fileStem}-cesium-preview.html`),
), ),
compressedGlb: path.resolve(
outputOverrides.compressedGlb || path.join(areaDir, `${compressedFileStem}.glb`),
),
compressedMetadata: path.resolve(
outputOverrides.compressedMetadata || path.join(areaDir, `${compressedFileStem}.json`),
),
compressedCesiumPreview: path.resolve(
outputOverrides.compressedCesiumPreview || path.join(areaDir, `${compressedFileStem}-cesium-preview.html`),
),
vehicleRoute: path.resolve(outputOverrides.vehicleRoute || path.join(areaDir, `${fileStem}-vehicle-route.json`)), vehicleRoute: path.resolve(outputOverrides.vehicleRoute || path.join(areaDir, `${fileStem}-vehicle-route.json`)),
vehicleModel: path.resolve(outputOverrides.vehicleModel || path.join(areaDir, `${fileStem}-vehicle-car.gltf`)), vehicleModel: path.resolve(outputOverrides.vehicleModel || path.join(areaDir, `${fileStem}-vehicle-car.gltf`)),
trafficSignalAssemblies: path.resolve( trafficSignalAssemblies: path.resolve(
@@ -80,7 +69,7 @@ function normalizeAreaConfig(raw, options = {}) {
cesium: raw.stages?.cesium ?? true, cesium: raw.stages?.cesium ?? true,
reimport: false, reimport: false,
preview: false, preview: false,
compress: false, compress: true,
}, },
qgis: { qgis: {
arrowScale: raw.qgis?.arrowScale ?? raw.arrowScale ?? 0.8, arrowScale: raw.qgis?.arrowScale ?? raw.arrowScale ?? 0.8,

View File

@@ -334,9 +334,6 @@ function artifactStatus(area) {
["Cesium GLB", area.outputs.glb, true, "file"], ["Cesium GLB", area.outputs.glb, true, "file"],
["Cesium metadata", area.outputs.metadata, true, "file"], ["Cesium metadata", area.outputs.metadata, true, "file"],
["Cesium preview", area.outputs.cesiumPreview, true, "file"], ["Cesium preview", area.outputs.cesiumPreview, true, "file"],
["Compressed GLB", area.outputs.compressedGlb, false, "file"],
["Compressed metadata", area.outputs.compressedMetadata, false, "file"],
["Compressed preview", area.outputs.compressedCesiumPreview, false, "file"],
]; ];
return entries.map(([label, file, expected, type]) => { return entries.map(([label, file, expected, type]) => {
const exists = fs.existsSync(file); const exists = fs.existsSync(file);
@@ -374,6 +371,7 @@ function metadataSummary(file, warnings) {
} }
function stageManifestStatus(area, configPath = null) { function stageManifestStatus(area, configPath = null) {
const compressionComplete = fs.existsSync(stageManifestPath(area, "compress"));
const reimportManifest = stageManifestPath(area, "reimport"); const reimportManifest = stageManifestPath(area, "reimport");
const hasReimportManifest = fs.existsSync(reimportManifest); const hasReimportManifest = fs.existsSync(reimportManifest);
const derivedConfig = path.join(area.outputs.pipelineDir, "osm2streets-qgis.config.json"); const derivedConfig = path.join(area.outputs.pipelineDir, "osm2streets-qgis.config.json");
@@ -446,7 +444,7 @@ function stageManifestStatus(area, configPath = null) {
inputs: { inputs: {
blend: area.outputs.blend, blend: area.outputs.blend,
}, },
outputs: { outputs: compressionComplete ? {} : {
glb: area.outputs.glb, glb: area.outputs.glb,
metadata: area.outputs.metadata, metadata: area.outputs.metadata,
}, },
@@ -457,15 +455,20 @@ function stageManifestStatus(area, configPath = null) {
inputs: { inputs: {
...(configPath ? { config: configPath } : {}), ...(configPath ? { config: configPath } : {}),
osm: area.input, osm: area.input,
...(compressionComplete ? {} : {
glb: area.outputs.glb, glb: area.outputs.glb,
metadata: area.outputs.metadata, metadata: area.outputs.metadata,
}),
lanePolygons: path.join(area.outputs.geojsonDir, "lane_polygons.geojson"), lanePolygons: path.join(area.outputs.geojsonDir, "lane_polygons.geojson"),
network: path.join(area.outputs.geojsonDir, "network.json"), network: path.join(area.outputs.geojsonDir, "network.json"),
intersectionSurface: path.join(area.outputs.geojsonDir, "intersection_surface.geojson"), intersectionSurface: path.join(area.outputs.geojsonDir, "intersection_surface.geojson"),
previewCss: path.join(path.resolve(__dirname, ".."), "lib", "cesium-preview.css"), previewCss: path.join(path.resolve(__dirname, ".."), "lib", "cesium-preview.css"),
previewJs: path.join(path.resolve(__dirname, ".."), "lib", "cesium-preview.js"), previewJs: path.join(path.resolve(__dirname, ".."), "lib", "cesium-preview.js"),
}, },
outputs: { outputs: compressionComplete ? {
vehicleRoute: area.outputs.vehicleRoute,
vehicleModel: area.outputs.vehicleModel,
} : {
cesiumPreview: area.outputs.cesiumPreview, cesiumPreview: area.outputs.cesiumPreview,
vehicleRoute: area.outputs.vehicleRoute, vehicleRoute: area.outputs.vehicleRoute,
vehicleModel: area.outputs.vehicleModel, vehicleModel: area.outputs.vehicleModel,
@@ -473,17 +476,13 @@ function stageManifestStatus(area, configPath = null) {
}, },
{ {
stage: "compress", stage: "compress",
expected: fs.existsSync(area.outputs.compressedGlb), expected: fs.existsSync(area.outputs.glb),
inputs: { inputs: {},
outputs: {
glb: area.outputs.glb, glb: area.outputs.glb,
metadata: area.outputs.metadata, metadata: area.outputs.metadata,
cesiumPreview: area.outputs.cesiumPreview, cesiumPreview: area.outputs.cesiumPreview,
}, },
outputs: {
compressedGlb: area.outputs.compressedGlb,
compressedMetadata: area.outputs.compressedMetadata,
compressedCesiumPreview: area.outputs.compressedCesiumPreview,
},
}, },
]; ];
return stages.map((entry) => { return stages.map((entry) => {

View File

@@ -6,11 +6,11 @@ const STAGES = [
{ id: "blender", label: "Blender scene", description: "Generate the editable scene and render" }, { id: "blender", label: "Blender scene", description: "Generate the editable scene and render" },
{ id: "cesium", label: "Cesium export", description: "Export GLB, metadata, and preview" }, { id: "cesium", label: "Cesium export", description: "Export GLB, metadata, and preview" },
{ id: "preview", label: "Preview refresh", description: "Regenerate preview HTML and vehicles" }, { id: "preview", label: "Preview refresh", description: "Regenerate preview HTML and vehicles" },
{ id: "compress", label: "Compressed preview", description: "Create optional compressed GLB artifacts" }, { id: "compress", label: "Compress delivery assets", description: "Replace standard GLB, metadata, and preview with compressed deliverables" },
]; ];
const ALIASES = { const ALIASES = {
all: ["intermediates", "blender", "cesium"], all: ["intermediates", "blender", "cesium", "compress"],
qgis: ["intermediates"], osm2streets: ["intermediates"], geojson: ["intermediates"], intermediate: ["intermediates"], qgis: ["intermediates"], osm2streets: ["intermediates"], geojson: ["intermediates"], intermediate: ["intermediates"],
intermediates: ["intermediates"], reimport: ["reimport"], gpkg: ["reimport"], blender: ["blender"], scene: ["blender"], intermediates: ["intermediates"], reimport: ["reimport"], gpkg: ["reimport"], blender: ["blender"], scene: ["blender"],
cesium: ["cesium"], glb: ["cesium"], preview: ["preview"], html: ["preview"], cesiumPreview: ["preview"], cesium: ["cesium"], glb: ["cesium"], preview: ["preview"], html: ["preview"], cesiumPreview: ["preview"],

View File

@@ -71,6 +71,10 @@ assert.equal(
const configPath = path.join(tempDir, "area.json"); const configPath = path.join(tempDir, "area.json");
fs.writeFileSync(configPath, `${JSON.stringify(base)}\n`); fs.writeFileSync(configPath, `${JSON.stringify(base)}\n`);
const area = normalizeAreaConfig(base); const area = normalizeAreaConfig(base);
assert.equal(area.stages.compress, true);
assert.equal("compressedGlb" in area.outputs, false);
assert.equal("compressedMetadata" in area.outputs, false);
assert.equal("compressedCesiumPreview" in area.outputs, false);
fs.mkdirSync(area.outputs.geojsonDir, { recursive: true }); fs.mkdirSync(area.outputs.geojsonDir, { recursive: true });
for (const file of [area.outputs.glb, area.outputs.metadata, area.outputs.cesiumPreview, area.outputs.vehicleRoute, area.outputs.vehicleModel]) { for (const file of [area.outputs.glb, area.outputs.metadata, area.outputs.cesiumPreview, area.outputs.vehicleRoute, area.outputs.vehicleModel]) {
fs.writeFileSync(file, "fixture\n"); fs.writeFileSync(file, "fixture\n");

View File

@@ -3,6 +3,7 @@
const assert = require("assert"); const assert = require("assert");
const { resolveStages, canonicalStages } = require("./lib/build-stages"); const { resolveStages, canonicalStages } = require("./lib/build-stages");
assert.deepEqual(canonicalStages(resolveStages({}, ["compress", "blender", "preview"])), ["blender", "preview", "compress"]); assert.deepEqual(canonicalStages(resolveStages({}, ["compress", "blender", "preview"])), ["blender", "preview", "compress"]);
assert.deepEqual(canonicalStages(resolveStages({}, ["all"])), ["intermediates", "blender", "cesium"]); assert.deepEqual(canonicalStages(resolveStages({}, ["all"])), ["intermediates", "blender", "cesium", "compress"]);
assert.deepEqual(canonicalStages(resolveStages({ intermediates: true, blender: true, cesium: true, compress: true })), ["intermediates", "blender", "cesium", "compress"]);
assert.throws(() => resolveStages({}, ["intermediates", "reimport"]), /mutually exclusive/); assert.throws(() => resolveStages({}, ["intermediates", "reimport"]), /mutually exclusive/);
console.log("Build stage tests passed."); console.log("Build stage tests passed.");