Single-Software Execution
931Model, modify, or simulate an artifact in one designated application.
EngiWorldWhat Can Frontier Agents Deliver in
Professional Engineering Environments?
1Tsinghua University 2Zhiman AI 3Chongqing University 4UCAS 5Shandong University 6BUPT 7Fudan University 8Henan Polytechnic University 9Xi'an Jiaotong University 10Peking University 11Zhejiang University
Overview
Autonomous agents have made rapid progress in general-purpose computer use, yet reliable automation of professional industrial engineering remains out of reach. Engineering workflows require reasoning over geometric and physical constraints and keeping dependencies consistent across software and design stages.
We present EngiWorld, a benchmark designed around the capabilities that real engineering projects require, from operating a single application to solving engineering problems end to end. It contains 1,301 expert-curated tasks spanning six engineering domains (CAD, CAE, CAM, BIM, EDA, and DCC) and 26 professional software applications, with both GUI and CLI interfaces. The tasks are organized into six types, corresponding to capabilities that real engineering projects require.
We further introduce an artifact-centric evaluation methodology built on a unified suite of domain verifiers. The verifiers programmatically check whether final and intermediate artifacts are geometrically valid, physically feasible, and compliant with design rules. Beyond binary success, they also score how well optimized designs meet their specifications. Evaluation of seven frontier models reveals a substantial capability gap: the strongest model scores only 44.7/100, and just 3.6% of runs on tasks that require cross-software coordination succeed. EngiWorld provides a rigorous foundation for measuring progress toward agents that operate professional engineering software end to end.
The benchmark
Six task types, testing the capabilities real engineering projects require.
Software Selection
140Choose suitable software from a permitted set and complete the task.
Vision-Guided Modeling
120Build engineering models from reference drawings and visual requirements.
Design Optimization
40Optimize design quality while satisfying engineering feasibility constraints.
Cross-Software Coordination
60Carry out workflows that pass dependent artifacts across applications.
Open-Environment Engineering
10Choose and install software in a blank environment, then solve the engineering problem end to end.
How the dataset is constructed
Experts define tasks and acceptance criteria; AI assists specification and verifier implementation. Each task is executed and reviewed in its software environment.
Evaluation
Agents work through GUI or CLI interfaces. Domain-specific verifiers reopen their deliverables and check the engineering properties that matter.
Interact
Start from a clean, task-specific workspace. Use visual observations and mouse or keyboard actions, or terminal commands and scripts.
Deliver
Submit native models, simulation outputs, circuits, building models, or 3D scenes, including intermediate files when the workflow requires them.
Verify
Recompute geometry, topology, physical quantities, rule compliance, and consistency across workflow stages against task-specific criteria.
Leaderboard
Seven frontier models on the same 306-task main evaluation, covering 158 CLI and 148 GUI tasks.
Each point is one model. Turns and cost are per-task means; output tokens are per-turn means.
| Model | ||||||
|---|---|---|---|---|---|---|
01 Claude Opus 5 | 44.7 | 48.6 | 40.5 | 69.5 | 2.4 | 18.78 |
02 GPT-5.6 Sol | 38.6 | 47.9 | 28.6 | 54.2 | 1.0 | 9.57 |
03 Qwen3.8 Max | 26.3 | 35.3 | 16.6 | 107.3 | 2.7 | 6.44 |
04 Gemini 3.7 Flash | 26.1 | 37.2 | 14.2 | 110.0 | 0.7 | 2.08 |
05 DeepSeek V4.1 Flash | 25.9 | 48.3 | 2.0 | 115.3 | 9.8 | 0.80 |
06 Kimi K3 | 24.3 | 31.3 | 16.7 | 92.1 | 0.9 | 5.85 |
| 07Qwen3.8 Flash | 16.7 | 31.5 | 1.0 | 119.6 | 1.3 | 0.44 |
Failure reason (%)
| Model | n | F1 | F2 | F3 | F4 | F5 |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 167 | 74.3 | 19.2 | 1.2 | 5.4 | 0.0 |
| GPT-5.6 Sol | 186 | 87.1 | 12.9 | 0.0 | 0.0 | 0.0 |
| DeepSeek V4.1 Flash | 221 | 24.9 | 62.0 | 0.5 | 12.7 | 0.0 |
| Gemini 3.7 Flash | 224 | 40.2 | 59.4 | 0.0 | 0.0 | 0.4 |
| Qwen3.8 Max | 224 | 38.8 | 58.5 | 1.8 | 0.0 | 0.9 |
| Kimi K3 | 230 | 54.8 | 40.9 | 4.3 | 0.0 | 0.0 |
| Qwen3.8 Flash | 254 | 26.8 | 61.0 | 12.2 | 0.0 | 0.0 |
- F1
- Declared completion
- F2
- Turn limit reached
- F3
- Declared failure
- F4
- Runtime limit exceeded
- F5
- Desktop closed
1,506 analyzed main-evaluation failures; 7 evaluator-side missing-output or missing-entry cases excluded.
Analysis
Controlled ablations vary visual inputs, accessibility information, screenshot resolution, and interaction history.
Reference images in the message
For GPT-5.6 Sol, GUI EngiScore rises from 15.0 to 31.7 when the reference drawing is shown directly in the task message. Easier access helps agents find information but does not ensure that they use it correctly.
Richer observations have tradeoffs
Accessibility trees improve Kimi, barely change GPT, and lower Gemini’s score. API cost rises by 51–148% even though all three models use fewer turns.
History length is model dependent
Gemini benefits from a 15-turn window; GPT largely plateaus after 10 turns. Kimi performs best at 10, with a longer window reducing its score and increasing cost.
Initial-image presentation
| Model | GUI | CLI | ||
|---|---|---|---|---|
| Env. | Message | Env. | Message | |
| GPT-5.6 Sol | 15.0 | 31.7 | 63.3 | 68.3 |
| Gemini 3.7 Flash | 3.3 | 10.0 | 38.3 | 40.0 |
| Kimi K3 | 6.7 | 10.0 | 40.0 | 36.7 |
Runtime image access in CLI
| Model | With initial image | Without initial image | ||
|---|---|---|---|---|
| Off | On | Off | On | |
| GPT-5.6 Sol | 66.7 | 68.3 | 48.1 | 41.5 |
| Gemini 3.7 Flash | 36.7 | 40.0 | 33.3 | 36.6 |
| Kimi K3 | 41.7 | 36.7 | 31.5 | 26.5 |
Case studies
Four Single-Software Execution GUI workflows from the paper. Each passes its task-specific artifact checks and receives a score of 1.0.
Connect a data bus and preserve the circuit
Eight distinct data nets establish the required MCU–SRAM mapping. The saved schematic retains the existing circuitry and passes with zero electrical-rule errors.
Build a precise staggered hole pattern
Two rectangular arrays create 23 tube holes. The exported DXF preserves the required coordinates, radii, entity types, and layer assignments.
Align parts while preserving geometry
The sliding block is positioned against the base. The exported STEP retains both independent solids and their original dimensions.
Deliver a scene with a packed AO bake
A 512 × 512 ambient-occlusion image is baked and packed into the saved native scene, preserving the computed pixels in the deliverable.
Where Cross-Software Coordination workflows break down
The recorded failures span import-scale errors, incorrect action targeting, and mismatches between submitted representations and verifier expectations. In the load case, a pressure representation does not satisfy the verifier’s force/displacement check; in the CSV case, an unrecognized diameter-field name stops evaluation before downstream geometry checks.
Submit Your Agent
If you are developing an AI model or agent and would like to evaluate it on EngiWorld, we welcome you to contact us at gaohongcheng2000@gmail.com. Please include a publicly accessible link to your model or agent.
We will confirm the evaluation scope, service fee, and timeline with you, complete the evaluation as agreed, and share the results.
Citation
BibTeX
@misc{engiworld2026,
title = {EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?},
author = {Hongcheng Gao and Hailong Qu and Yu Lei and Henghui Sun and Haoyang Li and Yipeng Wei and Naihao Xue and Xiaohan Yu and Zhuo Tao and Yihe Zang and Yajiao Wang and Jingyi Tang and Yi Li and Jingjing Zhou and Jie Luo and Bohan Zeng and Chengyu Shen and Hao Jiang and Chong Chen and Bowen Qu and Olive Huang and Zeqiang Wang},
year = {2026},
eprint = {2609.37686},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2609.37686}
}