Refactor pipeline around area asset builds

This commit is contained in:
2026-07-27 12:56:13 +08:00
parent 59a22ea697
commit 61e4820a4c
10 changed files with 446 additions and 139 deletions

186
README.md
View File

@@ -1,14 +1,25 @@
# OSM GIS Pipeline
# OSM Asset Pipeline
把 OSM XML 转为 QGIS 工程osm2streets 风格道路)和可选的 Blender 3D 场景
单个园区/片区 OSM XML 转为可消费的 Blender 场景和 Cesium GLB。osm2streets GeoJSON、GeoPackage、QGIS 工程和预览图都是中间资产,用来提供道路几何、调试标线效果,以及给 Blender/Cesium 生成提供输入
## 目标产物
每个区域默认输出到 `outputs/<area-id>/`
- `<area-id>.blend`Blender 场景,包含道路、建筑、水体、植被等
- `<area-id>.png`Blender 预览渲染
- `<area-id>.glb`Cesium 可加载的 3D 模型
- `<area-id>.json`Cesium 放置元数据和示例代码
- `osm2streets_web_out/`osm2streets GeoJSON 中间层
- `<area-id>.gpkg` / `<area-id>.qgz` / `<area-id>-preview.png`QGIS 调试资产
## 环境
需要:
- macOS QGIS默认 `/Applications/QGIS.app`
- Blender默认 `/Applications/Blender.app`
- Node.js / npm
- Blender可选用于 3D 场景生成)
首次使用:
@@ -17,93 +28,94 @@ cd /Users/que01/osm2streets-qgis-workflow
npm install
```
## QGIS 管线(强依赖 osm2streets
## 主流程
使用默认配置
默认构建南台子湖创新谷样例
```bash
cd /Users/que01/osm2streets-qgis-workflow
npm run build
```
使用指定配置:
指定区域配置:
```bash
node scripts/build-osm2streets-qgis.js --config /path/to/config.json
npm run build:area -- --config config/areas/hanyang-block.json
```
命令行参数可以覆盖配置文件
只跑部分阶段
```bash
node scripts/build-osm2streets-qgis.js \
--input /path/to/osm.xml \
--out-dir /path/to/out \
--gpkg /path/to/output.gpkg \
--project /path/to/output.qgz \
--preview /path/to/output_preview.png
npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages intermediates
npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages blender
npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages cesium
```
创建新区域配置可以从模板复制:
`intermediates` 会生成 osm2streets GeoJSON、GeoPackage、QGIS 工程和 QGIS 预览图。`blender` 使用 OSM 和 osm2streets GeoJSON 生成 `.blend`/`.png``cesium``.blend` 导出 `.glb`/`.json`
## 区域配置
新区域从模板复制:
```bash
cp config/examples/template.json config/my-area.json
cp config/examples/template.json config/areas/my-area.json
```
### 配置项
核心配置:
```json
{
"qgisApp": "/Applications/QGIS.app",
"id": "my-area",
"input": "/absolute/path/to/input.osm",
"outDir": "/absolute/path/to/osm2streets_web_out",
"gpkg": "/absolute/path/to/output.gpkg",
"project": "/absolute/path/to/output.qgz",
"preview": "/absolute/path/to/output_preview.png",
"arrowScale": 0.8,
"clipPad": 0.002,
"canvasPad": 0.001,
"previewPad": 0.0007,
"canvasExtent": null,
"previewExtent": null,
"layerPrefix": "osm2streets",
"osm2streets": {
"debug_each_step": false,
"dual_carriageway_experiment": false,
"sidepath_zipping_experiment": false,
"inferred_sidewalks": true,
"osm2lanes": true
"outputRoot": "/absolute/path/to/outputs",
"qgisApp": "/Applications/QGIS.app",
"blenderApp": "/Applications/Blender.app",
"stages": {
"intermediates": true,
"blender": true,
"cesium": true
},
"qgis": {
"arrowScale": 0.8,
"clipPad": 0.002,
"canvasPad": 0.001,
"previewPad": 0.0007,
"canvasExtent": null,
"previewExtent": null,
"layerPrefix": "osm2streets"
},
"blender": {
"treeStyle": "natural",
"officeOverrides": ""
}
}
```
- `input`OSM XML 输入文件路径。
- `outDir`osm2streets GeoJSON 中间产物输出目录。
- `gpkg` / `project` / `preview`:最终 GeoPackage、QGIS 工程、预览 PNG 路径。
- `layerPrefix`QGIS 图层显示名称前缀(如 `"osm2streets"``"osm2streets road surface"`)。
- `arrowScale`:方向箭头几何缩放。
- `osm2streets`osm2streets 引擎选项。
`scripts/build-area.js` 会按 `id` 自动推导默认输出路径。确实需要定制时,可以增加 `outputs` 覆盖:
### 输出图层
```json
{
"outputs": {
"areaDir": "/absolute/path/to/custom-area",
"blend": "/absolute/path/to/custom.blend",
"glb": "/absolute/path/to/custom.glb"
}
}
```
GeoPackage 内会生成:
## 已沉淀区域
- `road_surface`
- `intersection_surface`
- `sidewalks`
- `sidewalk_corners`
- `lane_separators`
- `center_lines`
- `vehicle_stop_lines`
- `lane_arrows_webscale`
- `crosswalks`
- `config/areas/nantaizi-lake-innovation-valley.json`
- `config/areas/hanyang-block.json`
QGIS 工程绘制顺序:路面在最底,车道线、斑马线、停止线和箭头在上。
## 低层命令
## Blender 3D 场景(可选)
通常优先使用 `npm run build:area`。如果只想调试旧 QGIS/osm2streets 阶段,可以直接运行:
osm2streets 道路几何为可选输入;不提供时回退到 OSM highway 折线。
```bash
npm run build:qgis -- --config config/hanyang-block.json
```
生成场景
Blender 低层命令
```bash
/Applications/Blender.app/Contents/MacOS/Blender \
@@ -112,70 +124,24 @@ osm2streets 道路几何为可选输入;不提供时回退到 OSM highway 折
--osm "/path/to/input.osm" \
--geojson "/path/to/osm2streets_web_out" \
--output "/path/to/scene.blend" \
--render "/path/to/preview.png"
--render "/path/to/preview.png" \
--tree-style natural
```
`--geojson` 为可选参数。使用 `--office-overrides` 指定应渲染为办公楼的 OSM way ID逗号分隔
导出为 Cesium GLB
Cesium GLB 低层导出:
```bash
/Applications/Blender.app/Contents/MacOS/Blender \
--background \
--python blender/export_cesium.py -- \
--blend "/path/to/scene.blend" \
--glb "/path/to/scene_cesium.glb" \
--metadata "/path/to/scene_cesium.json"
--glb "/path/to/scene.glb" \
--metadata "/path/to/scene.json"
```
详见 [blender/README.md](blender/README.md)。
## 运行示例
以下命令已在本机验证通过。
QGIS 管线(使用汉阳区区块 OSM 数据):
```bash
node scripts/build-osm2streets-qgis.js \
--config config/hanyang-block.json
```
Blender 场景(含 osm2streets 道路几何):
```bash
/Applications/Blender.app/Contents/MacOS/Blender \
--background --factory-startup \
--python blender/generate_scene.py -- \
--osm "/Users/que01/Desktop/南台子湖创新谷OSM.osm" \
--geojson "/Users/que01/osm2streets-qgis-workflow/outputs/nantaizi-lake-innovation-valley/osm2streets_web_out" \
--output "/tmp/scene.blend" \
--render "/tmp/preview.png"
```
Blender 场景(纯 OSM无 osm2streets 道路):
```bash
/Applications/Blender.app/Contents/MacOS/Blender \
--background --factory-startup \
--python blender/generate_scene.py -- \
--osm "/Users/que01/Desktop/南台子湖创新谷OSM.osm" \
--output "/tmp/scene-nogeojson.blend" \
--render "/tmp/preview-nogeojson.png"
```
Cesium GLB 导出:
```bash
/Applications/Blender.app/Contents/MacOS/Blender \
--background \
--python blender/export_cesium.py -- \
--blend "/tmp/scene.blend" \
--glb "/tmp/scene.glb" \
--metadata "/tmp/scene.json"
```
## 文档
- [docs/input-cases.md](docs/input-cases.md) 已验证的 OSM 输入案例
- [docs/changelog.md](docs/changelog.md) 变更记录
- [docs/input-cases.md](docs/input-cases.md) - 已验证的 OSM 输入案例
- [docs/changelog.md](docs/changelog.md) - 变更记录

View File

@@ -2,6 +2,14 @@
该生成器将 OSM 建筑、水体、植被标签,以及可选的 osm2streets GeoJSON 道路图层,组合为 Blender 3D 场景。
项目主入口是根目录的区域管线:
```bash
npm run build:area -- --config config/areas/nantaizi-lake-innovation-valley.json --stages blender,cesium
```
下面的命令适合单独调试 Blender 脚本。
## 生成场景
```bash
@@ -15,7 +23,7 @@
--tree-style natural
```
`--geojson` 为可选参数。不提供时,道路使用简单的 OSM highway 折线,而非详细 osm2streets 几何。
`--geojson` 为可选参数。不提供时,道路使用简单的 OSM highway 折线,而非详细 osm2streets 几何。实际资产生成建议先跑 `intermediates` 阶段,为 Blender 提供 osm2streets 道路面、标线、斑马线和箭头。
`--tree-style` 可选 `natural``procedural`。默认 `natural` 使用低面数树干和多层不规则树冠,形态比球形程序化树冠更自然,渲染也比外部 OBJ 树模型轻;`procedural` 最轻但更卡通。
使用 `--office-overrides` 指定一组 OSM way ID逗号分隔这些建筑将渲染为办公楼风格即使其 OSM 标签为 `building=industrial`

View File

@@ -42,7 +42,7 @@ def cli_args():
i = 0
while i < len(argv):
if argv[i].startswith("--") and i + 1 < len(argv):
values[argv[i][2:]] = argv[i + 1]
values[argv[i][2:].replace("-", "_")] = argv[i + 1]
i += 2
else:
i += 1

View File

@@ -0,0 +1,32 @@
{
"id": "hanyang-block",
"input": "/Users/que01/Desktop/汉阳区区块.osm",
"outputRoot": "/Users/que01/osm2streets-qgis-workflow/outputs",
"qgisApp": "/Applications/QGIS.app",
"blenderApp": "/Applications/Blender.app",
"stages": {
"intermediates": true,
"blender": true,
"cesium": true
},
"qgis": {
"arrowScale": 0.8,
"clipPad": 0.002,
"canvasPad": 0.001,
"previewPad": 0.0007,
"canvasExtent": null,
"previewExtent": null,
"layerPrefix": "osm2streets"
},
"osm2streets": {
"debug_each_step": false,
"dual_carriageway_experiment": false,
"sidepath_zipping_experiment": false,
"inferred_sidewalks": true,
"osm2lanes": true
},
"blender": {
"treeStyle": "natural",
"officeOverrides": ""
}
}

View File

@@ -0,0 +1,32 @@
{
"id": "nantaizi-lake-innovation-valley",
"input": "/Users/que01/Desktop/南台子湖创新谷OSM.osm",
"outputRoot": "/Users/que01/osm2streets-qgis-workflow/outputs",
"qgisApp": "/Applications/QGIS.app",
"blenderApp": "/Applications/Blender.app",
"stages": {
"intermediates": true,
"blender": true,
"cesium": true
},
"qgis": {
"arrowScale": 0.8,
"clipPad": 0.002,
"canvasPad": 0.001,
"previewPad": 0.0007,
"canvasExtent": null,
"previewExtent": null,
"layerPrefix": "osm2streets"
},
"osm2streets": {
"debug_each_step": false,
"dual_carriageway_experiment": false,
"sidepath_zipping_experiment": false,
"inferred_sidewalks": true,
"osm2lanes": true
},
"blender": {
"treeStyle": "natural",
"officeOverrides": ""
}
}

View File

@@ -1,17 +1,23 @@
{
"qgisApp": "/Applications/QGIS.app",
"id": "my-area",
"input": "/absolute/path/to/input.osm",
"outDir": "/absolute/path/to/osm2streets_web_out",
"gpkg": "/absolute/path/to/output.gpkg",
"project": "/absolute/path/to/output.qgz",
"preview": "/absolute/path/to/output_preview.png",
"arrowScale": 0.8,
"clipPad": 0.002,
"canvasPad": 0.001,
"previewPad": 0.0007,
"canvasExtent": null,
"previewExtent": null,
"layerPrefix": "osm2streets",
"outputRoot": "/absolute/path/to/outputs",
"qgisApp": "/Applications/QGIS.app",
"blenderApp": "/Applications/Blender.app",
"stages": {
"intermediates": true,
"blender": true,
"cesium": true
},
"qgis": {
"arrowScale": 0.8,
"clipPad": 0.002,
"canvasPad": 0.001,
"previewPad": 0.0007,
"canvasExtent": null,
"previewExtent": null,
"layerPrefix": "osm2streets"
},
"osm2streets": {
"debug_each_step": false,
"dual_carriageway_experiment": false,
@@ -20,10 +26,7 @@
"osm2lanes": true
},
"blender": {
"scene": "/absolute/path/to/scene.blend",
"render": "/absolute/path/to/preview.png",
"glb": "/absolute/path/to/cesium.glb",
"metadata": "/absolute/path/to/cesium.json",
"office_overrides": ""
"treeStyle": "natural",
"officeOverrides": ""
}
}
}

View File

@@ -1,5 +1,12 @@
# Changelog
## 2026-07-27
- 将项目主入口重构为区域资产管线:`scripts/build-area.js`
- 新增 `config/areas/nantaizi-lake-innovation-valley.json``config/areas/hanyang-block.json`,支持按 OSM 输入生成独立输出目录。
- `npm run build` 现在默认走区域资产管线;旧 QGIS 管线保留为 `npm run build:qgis`
- 修复 Blender 脚本对 `--tree-style``--office-overrides` 这类连字符参数的解析。
## 2026-07-24
- Blender 场景生成器通用化:`generate_nantaizi.py``generate_scene.py`
@@ -20,4 +27,4 @@
- Added [config/hanyang-block.json](/Users/que01/osm2streets-qgis-workflow/config/hanyang-block.json) for `/Users/que01/Desktop/汉阳区区块.osm`.
- Added [config/examples/template.json](/Users/que01/osm2streets-qgis-workflow/config/examples/template.json) for new areas.
- Updated OSM node coordinate parsing to support both single-quoted and double-quoted XML attributes.
- Verified QGIS outputs for both a smaller Overpass-style XML input and a larger JOSM-generated `.osm` input.
- Verified QGIS outputs for both a smaller Overpass-style XML input and a larger JOSM-generated `.osm` input.

8
package-lock.json generated
View File

@@ -1,12 +1,12 @@
{
"name": "osm2streets-qgis-workflow",
"version": "0.1.0",
"name": "osm-asset-pipeline",
"version": "0.3.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "osm2streets-qgis-workflow",
"version": "0.1.0",
"name": "osm-asset-pipeline",
"version": "0.3.0",
"dependencies": {
"osm2streets-js-node": "0.1.4"
}

View File

@@ -1,12 +1,14 @@
{
"name": "osm-gis-pipeline",
"version": "0.2.0",
"name": "osm-asset-pipeline",
"version": "0.3.0",
"private": true,
"type": "commonjs",
"scripts": {
"build": "node scripts/build-osm2streets-qgis.js"
"build": "node scripts/build-area.js",
"build:area": "node scripts/build-area.js",
"build:qgis": "node scripts/build-osm2streets-qgis.js"
},
"dependencies": {
"osm2streets-js-node": "0.1.4"
}
}
}

257
scripts/build-area.js Executable file
View File

@@ -0,0 +1,257 @@
#!/usr/bin/env node
const fs = require("fs");
const path = require("path");
const { spawnSync } = require("child_process");
const repoRoot = path.resolve(__dirname, "..");
const args = parseArgs(process.argv.slice(2));
const configPath = path.resolve(
args.config || path.join(repoRoot, "config", "areas", "nantaizi-lake-innovation-valley.json"),
);
const area = normalizeAreaConfig(readJson(configPath));
const requestedStages = args.stages
? splitList(args.stages)
: null;
const stages = resolveStages(area.stages, requestedStages);
console.log(`Area: ${area.id}`);
console.log(`Config: ${configPath}`);
console.log(`Output: ${area.outputs.areaDir}`);
if (stages.intermediates) {
buildIntermediates(area);
}
if (stages.blender) {
buildBlenderScene(area);
}
if (stages.cesium) {
exportCesium(area);
}
console.log("Done.");
function parseArgs(argv) {
const out = {};
for (let i = 0; i < argv.length; i += 1) {
const arg = argv[i];
if (!arg.startsWith("--")) continue;
const key = arg.slice(2).replace(/-([a-z])/g, (_, c) => c.toUpperCase());
const next = argv[i + 1];
if (!next || next.startsWith("--")) {
out[key] = "true";
} else {
out[key] = next;
i += 1;
}
}
return out;
}
function readJson(file) {
if (!fs.existsSync(file)) {
throw new Error(`Config file not found: ${file}`);
}
return JSON.parse(fs.readFileSync(file, "utf8"));
}
function normalizeAreaConfig(raw) {
const id = requireText(raw.id, "id");
const input = path.resolve(requireText(raw.input, "input"));
if (!fs.existsSync(input)) {
throw new Error(`Input OSM XML not found: ${input}`);
}
const outputRoot = path.resolve(raw.outputRoot || path.join(repoRoot, "outputs"));
const outputOverrides = raw.outputs || {};
const areaDir = path.resolve(outputOverrides.areaDir || path.join(outputRoot, id));
const fileStem = outputOverrides.fileStem || id;
const outputs = {
areaDir,
geojsonDir: path.resolve(outputOverrides.geojsonDir || path.join(areaDir, "osm2streets_web_out")),
gpkg: path.resolve(outputOverrides.gpkg || path.join(areaDir, `${fileStem}.gpkg`)),
qgisProject: path.resolve(outputOverrides.qgisProject || path.join(areaDir, `${fileStem}.qgz`)),
qgisPreview: path.resolve(outputOverrides.qgisPreview || path.join(areaDir, `${fileStem}-preview.png`)),
blend: path.resolve(outputOverrides.blend || path.join(areaDir, `${fileStem}.blend`)),
render: path.resolve(outputOverrides.render || path.join(areaDir, `${fileStem}.png`)),
glb: path.resolve(outputOverrides.glb || path.join(areaDir, `${fileStem}.glb`)),
metadata: path.resolve(outputOverrides.metadata || path.join(areaDir, `${fileStem}.json`)),
pipelineDir: path.resolve(outputOverrides.pipelineDir || path.join(areaDir, "_pipeline")),
};
return {
id,
input,
outputRoot,
qgisApp: raw.qgisApp || "/Applications/QGIS.app",
blenderApp: raw.blenderApp || "/Applications/Blender.app",
stages: {
intermediates: raw.stages?.intermediates ?? raw.stages?.qgis ?? true,
blender: raw.stages?.blender ?? true,
cesium: raw.stages?.cesium ?? true,
},
qgis: {
arrowScale: raw.qgis?.arrowScale ?? raw.arrowScale ?? 0.8,
clipPad: raw.qgis?.clipPad ?? raw.clipPad ?? 0.002,
canvasPad: raw.qgis?.canvasPad ?? raw.canvasPad ?? 0.001,
previewPad: raw.qgis?.previewPad ?? raw.previewPad ?? 0.0007,
canvasExtent: raw.qgis?.canvasExtent ?? raw.canvasExtent ?? null,
previewExtent: raw.qgis?.previewExtent ?? raw.previewExtent ?? null,
layerPrefix: raw.qgis?.layerPrefix ?? raw.layerPrefix ?? "osm2streets",
},
osm2streets: raw.osm2streets || {
debug_each_step: false,
dual_carriageway_experiment: false,
sidepath_zipping_experiment: false,
inferred_sidewalks: true,
osm2lanes: true,
},
blender: {
treeStyle: raw.blender?.treeStyle || "natural",
officeOverrides: raw.blender?.officeOverrides || raw.blender?.office_overrides || "",
},
outputs,
};
}
function requireText(value, key) {
if (typeof value !== "string" || value.trim() === "") {
throw new Error(`Missing config key: ${key}`);
}
return value;
}
function splitList(value) {
return String(value)
.split(",")
.map((item) => item.trim())
.filter(Boolean);
}
function resolveStages(defaults, requested) {
if (!requested) return defaults;
const aliases = {
all: ["intermediates", "blender", "cesium"],
qgis: ["intermediates"],
osm2streets: ["intermediates"],
geojson: ["intermediates"],
intermediate: ["intermediates"],
intermediates: ["intermediates"],
blender: ["blender"],
scene: ["blender"],
cesium: ["cesium"],
glb: ["cesium"],
};
const out = { intermediates: false, blender: false, cesium: false };
for (const stage of requested) {
const mapped = aliases[stage];
if (!mapped) {
throw new Error(`Unknown stage '${stage}'. Use intermediates, blender, cesium, or all.`);
}
for (const key of mapped) out[key] = true;
}
return out;
}
function buildIntermediates(area) {
fs.mkdirSync(area.outputs.pipelineDir, { recursive: true });
const derivedConfig = {
qgisApp: area.qgisApp,
input: area.input,
outDir: area.outputs.geojsonDir,
gpkg: area.outputs.gpkg,
project: area.outputs.qgisProject,
preview: area.outputs.qgisPreview,
arrowScale: area.qgis.arrowScale,
clipPad: area.qgis.clipPad,
canvasPad: area.qgis.canvasPad,
previewPad: area.qgis.previewPad,
canvasExtent: area.qgis.canvasExtent,
previewExtent: area.qgis.previewExtent,
layerPrefix: area.qgis.layerPrefix,
osm2streets: area.osm2streets,
};
const derivedConfigPath = path.join(area.outputs.pipelineDir, "osm2streets-qgis.config.json");
fs.writeFileSync(derivedConfigPath, `${JSON.stringify(derivedConfig, null, 2)}\n`);
console.log("Stage: intermediates (osm2streets GeoJSON + QGIS)");
runCommand(process.execPath, [
path.join(repoRoot, "scripts", "build-osm2streets-qgis.js"),
"--config",
derivedConfigPath,
], "intermediates");
}
function buildBlenderScene(area) {
ensureFile(blenderExecutable(area), "Blender executable");
ensureFile(path.join(repoRoot, "blender", "generate_scene.py"), "Blender scene generator");
fs.mkdirSync(path.dirname(area.outputs.blend), { recursive: true });
fs.mkdirSync(path.dirname(area.outputs.render), { recursive: true });
const blenderArgs = [
"--background",
"--factory-startup",
"--python",
path.join(repoRoot, "blender", "generate_scene.py"),
"--",
"--osm",
area.input,
"--geojson",
area.outputs.geojsonDir,
"--output",
area.outputs.blend,
"--render",
area.outputs.render,
"--tree-style",
area.blender.treeStyle,
];
if (area.blender.officeOverrides) {
blenderArgs.push("--office-overrides", area.blender.officeOverrides);
}
console.log("Stage: blender");
runCommand(blenderExecutable(area), blenderArgs, "blender");
}
function exportCesium(area) {
ensureFile(blenderExecutable(area), "Blender executable");
ensureFile(area.outputs.blend, "Blend scene");
ensureFile(path.join(repoRoot, "blender", "export_cesium.py"), "Cesium exporter");
fs.mkdirSync(path.dirname(area.outputs.glb), { recursive: true });
fs.mkdirSync(path.dirname(area.outputs.metadata), { recursive: true });
console.log("Stage: cesium");
runCommand(blenderExecutable(area), [
"--background",
"--python",
path.join(repoRoot, "blender", "export_cesium.py"),
"--",
"--blend",
area.outputs.blend,
"--glb",
area.outputs.glb,
"--metadata",
area.outputs.metadata,
], "cesium");
}
function blenderExecutable(area) {
return path.join(area.blenderApp, "Contents", "MacOS", "Blender");
}
function ensureFile(file, label) {
if (!fs.existsSync(file)) {
throw new Error(`${label} not found: ${file}`);
}
}
function runCommand(command, commandArgs, stage) {
const result = spawnSync(command, commandArgs, { stdio: "inherit" });
if (result.error) {
throw result.error;
}
if (result.status !== 0) {
const signal = result.signal ? ` signal=${result.signal}` : "";
throw new Error(`Stage '${stage}' failed with status=${result.status}${signal}`);
}
}