Building a Multi-Agent System for Engineering Drawing Parsing and Design Review
Reprint notice: This English translation is republished from Yige’s original Chinese article, “多Agent系统实现工程图解析及校核” (“Building a Multi-Agent System for Engineering Drawing Parsing and Design Review”).
We recently used DesireCore to run several rounds of parsing and review tests on a set of P&ID drawings. These drawings are large, densely populated, and full of cross-sheet relationships. Equipment, valves, pipelines, instruments, control loops, parameters, and design notes all appear on the same sheet, making them a useful test of whether multiple agents can handle complex engineering information.
Engineering drawing parsing is particularly vulnerable to a deceptive kind of failure: a model may recognize most of the content and produce a polished report while still getting a few characters, symbols, or connections wrong. A small error may not matter much in an ordinary image, but in an engineering drawing, one misread letter or incorrectly connected line can invalidate every downstream review result.
This article therefore does not discuss a specific project or show any original engineering drawings. Instead, it summarizes the general methods and problems we encountered, including drawing preprocessing, local recognition, topology reconstruction, design-reference comparison, independent verification, and human sign-off.
Why You Cannot Simply Give the Entire Drawing to a Model
When teams first attempt engineering drawing parsing, the simplest approach is to give an entire sheet to a multimodal model and ask it to identify equipment, instruments, and pipelines directly. That may be sufficient for a demonstration, but it is not suitable for engineering review.
When an A1 drawing is scaled down, small text and fine lines lose substantial detail. If the system crops only a local region, the text becomes clearer, but pipelines may enter from outside the crop and the model cannot determine the complete connection. A symbol viewed in isolation may look like an ordinary valve; only after considering the pipelines on both sides, the actuator, and the signal line can the system determine what kind of component it represents.
Two intersecting lines do not necessarily indicate a connection. They may cross without connecting, or a signal line may simply pass over a process line. Judging only from a local shape can easily produce a result that looks plausible but is wrong.
Legend sheets, process sheets, and composite-assembly sheets must also be read differently. Legend sheets define symbols; process sheets primarily express relationships between equipment and pipelines; composite sheets may contain many highly similar repeated units. These distinctions are intuitive to an engineer, but an agent must be given explicitly separated recognition tasks.
We ultimately divided the process into the following steps:
Freeze source drawings and versions
↓
Establish a whole-sheet pixel baseline
↓
Crop local views and component context
↓
Create coordinate-backed text, symbol, and parameter candidates
↓
Recover connections and cross-sheet relationships from component ports, segment by segment
↓
Freeze the design references used for comparison
↓
Review each item against defined rules
↓
Perform an independent review, followed by engineer sign-off
The source drawing is divided into local regions, converted into structured facts, and reconstructed as topology before entering independent review and human confirmation.
Every layer must preserve provenance. The system should always be able to reopen the exact PDF version, page, region, and local crop behind a conclusion, show the original text, and explain why human review is still required. Without that traceability, an engineer has no practical way to verify an agent's result.
Use Deterministic Programs for Preprocessing Wherever Possible
Models are well suited to symbol semantics, textual ambiguity, and contextual relationships. Rendering drawings, calculating hashes, recording page order, converting coordinates, cropping images, and validating fields do not need the model to improvise. We use deterministic programs for these steps wherever possible.
When a source file enters the system, its version and fingerprint are registered first, followed by a unified pixel coordinate system for each page. The full-sheet image is retained while overlapping local crops are generated. Downstream agents do not guess from a heavily compressed overview; each agent receives both the whole-sheet context and the local view assigned to its task.
This may sound like ordinary image preprocessing, but our tests uncovered several important failure modes. One sheet, for example, had a different orientation from the rest of the drawing set. If every sheet was processed with the same orientation, the system still generated the expected number of local images with apparently valid dimensions, but their contents were wrong. We therefore added orientation, page number, dimensions, parent drawing, and file hash to a validation gate. If any field is invalid, the workflow stops.
In another test, a component crop existed and was not blank, but the crop did not actually contain the object required by the task. Checking only that “the file exists” and “the image contains pixels” would have admitted it as valid evidence. A component agent must now confirm that its crop genuinely contains the target object before recognition can continue.
Stopping early in such cases is not a failure. The most dangerous outcome in an engineering system is not an error message; it is an incorrect input followed by a sequence of steps that all report success.
How the Agents Divide the Work
The complete workflow has a coordinating agent that registers inputs, schedules dependencies, checks execution receipts, and decides whether the next task may begin. It does not perform all drawing recognition itself.
Different agents handle specific drawing tasks. A page agent identifies legend areas, title blocks, process areas, and notes. A component agent locates objects but does not assign a definitive type without sufficient context. A text agent reads small labels while preserving the original string and any ambiguous characters. A topology agent traces connections segment by segment from equipment ports, recording crossings, junctions, cross-sheet exits, and open endpoints separately.
Design review is a separate group of tasks. It does not participate in recognition and reads only validated drawing facts and frozen design references. It checks whether tags, quantities, parameters, connections, control relationships, and repeated units are consistent.
An independent review agent runs at the end. Importantly, this reviewer does not read the earlier agents' free-form reasoning. It makes a fresh determination from the source images, structured facts, review rules, and evidence locations. If the reviewing agent sees the earlier reasoning first, it is likely to follow the existing conclusion, defeating the purpose of an independent review.
Page registration, component location, text recognition, and topology reconstruction run separately, after which an independent agent reviews the findings.
This division of work addresses three main problems:
- Each agent processes only the context relevant to its task. A text agent does not need to maintain the full topology while reading small labels, and a pipeline-tracing agent does not need to evaluate every design rule at the same time.
- Recognition and review are performed by different agents, preventing the original executor from approving its own result.
- When upstream evidence is incomplete, downstream tasks do not receive an input that merely pretends to be complete, and the coordinating agent cannot bypass a gate by changing “pending review” to “confirmed.”
More agents are not automatically better. If one agent can handle a simple single-sheet drawing reliably, there is no reason to split the task further. A multi-agent design adds real value only when the task has conflicting fields of view, meaningful professional specialization, or a requirement for independent verification.
Engineering Drawing Parsing Cannot Stop at Bounding Boxes
Many current engineering-drawing AI demonstrations draw detection boxes and count equipment, valves, and instruments. That can be useful for object detection, but it remains far from design review.
A component is not a bounding box. At minimum, it needs a tight local crop, a contextual crop that explains its connections, coordinates on the source page, possible ports, a tag, and a current status. Composite components must also be decomposable: what are the actuator, valve body, instrument-air supply, and signal connection individually, and what do they represent together?
In this work, we organized drawing objects into several layers. The lowest layer contains primitives such as lines, circles, text, and arrows. Above those are individual symbols. Symbols and connections form composite components, and combinations of composite components finally form process units that engineers can interpret directly.
Once these layers exist, the system does not have to infer everything from pixels each time. An agent can first retrieve similar examples from the current drawing set's legend and previously confirmed components, then judge the current object in context. OCR or a PDF text layer can provide supporting evidence, but neither should determine the final result by itself. When glyphs are similar, text has been converted to outlines, or a pipeline passes through a character, the original crop and its context must remain available for engineer review.
Topology reconstruction must also be incremental. A topology agent starts from confirmed ports and traces each connection segment by segment. When it encounters a crossing, valve, sheet-edge connector, or uncertain area, it saves a checkpoint. Continuously traceable segments become connections, while unclosed segments enter a review queue.
If a connection cannot be confirmed, the system does not delete it or fill the gap using engineering intuition. Instead, it creates a follow-up task that specifies which additional local crop is required and which page should be searched for the continuation.
Zero Discrepancies Does Not Mean the Drawing Is Correct
In one test, drawing preprocessing, local parsing, and structured facts were complete, and the review results contained no explicit discrepancies. Looking only at the statistics, it would have been easy to conclude that the drawing matched the design references.
At that point, however, the design references used for comparison had not been version-frozen, so the system did not have the conditions required for a formal comparison. A discrepancy count of zero therefore meant only that no valid comparison had been completed; it did not mean that the drawing and design references were consistent.
The review agent marked the result as insufficient input, the independent reviewer withheld approval, and the sign-off gate remained closed. The system did not generate a consistency conclusion or create an engineer's signature automatically.
When design references lack the required version and scope information, a zero-discrepancy result still cannot pass the review gate.
After this test, we completely separated raw observations, normalized facts, comparison results, and formal sign-off. Content read from a drawing is a raw observation. Converting different expressions into comparable fields is normalization. A rule result of consistent, conflicting, pending review, or insufficient evidence is a comparison result. Before anything can enter a formal report, the system must still verify that the version, scope, evidence, and human authorization are complete.
Every review finding must state what the design reference says, what the drawing says, where the evidence is located, the finding's current status, and what information is still missing. When an engineer opens a finding, the source page and local crop should appear before any model-generated explanation.
The Practical Role of a Trainable Agent
Engineering drawings often reflect strong project-specific and company-specific conventions. The same symbol may vary subtly between drawing sets. A particular character sequence may have a fixed meaning within one discipline. Which parameters must be preserved verbatim, which open endpoints are legitimate cross-sheet connections, and which findings must be escalated immediately are not facts a general-purpose model knows automatically.
If every new drawing set requires the team to explain the legends, numbering conventions, and cross-sheet rules again, the system will struggle to create lasting value.
In DesireCore, confirmed legend entries, component examples, port patterns, ambiguous characters, cropping methods, and review rules can be stored in AgentFS. When an engineer makes a correction, the system records not only the correct answer but also the cause of the error and the conditions that should trigger review in the future.
An engineer might confirm, for example, “This drawing convention does not represent two independent valves; it represents one composite object.” That correction can enter the component library and the agent's Skill. The next time a similar drawing appears, the component-localization agent retrieves the example first. If the new context differs, the result can still remain a candidate rather than being applied indiscriminately simply because the pattern has appeared once before.
Similarly, if a class of tags frequently causes confusion between letters and numbers, the system can record a specific handling method: create a tighter text crop while preserving the surrounding context, then submit both candidate characters for review instead of automatically changing the character to the one that looks more plausible.
These lessons are stored as files that can be reviewed, diffed, rolled back, and migrated. Even if the model changes later, the methods accumulated by the company remain reusable. To me, this is an important distinction when DesireCore is used in a professional setting: the model provides foundational capabilities, while the company trains its own agents.
A trainable agent should not become more willing to decide on its own simply because it has been used more often. Many of the most valuable lessons are constraints: when the workflow must stop, which evidence cannot be used alone, and which conclusions only an engineer may approve.
How to Evaluate Engineering Drawing Parsing
Counting how many objects were detected is not particularly meaningful. More bounding boxes do not prove that component semantics are correct. Coordinates for every component do not prove that the full topology has been closed. Producing a report does not prove that design review has been completed.
We currently focus more on the following measures:
- Can every fact be traced back to its source page and local evidence?
- Are misaligned, blank, and over-cropped images blocked before semantic analysis?
- Are uncertain characters preserved as observed rather than silently polished?
- Does every cross-sheet connection and open endpoint have an explicit disposition?
- When the design-reference version changes, do earlier conclusions become invalid automatically?
- Do the execution agent and review agent genuinely operate with independent context?
- When inputs are insufficient, can the system refuse to report “consistent”?
- Is the final conclusion still confirmed by a qualified and authorized person?
Testing can also deliberately remove a required input or introduce a misaligned crop to confirm that the system stops. Successful execution on normal samples alone does not establish that the system is fit for engineering use.
If the system can state why it stopped, where it stopped, and what evidence is needed next, it has a realistic path into an actual engineering workflow.
Where Multi-Agent Engineering Drawing Review Fits
After several rounds of validation, we found this approach well suited to high-volume, repetitive work where omissions are easy: organizing drawing sets, building evidence indexes, extracting tags and parameters, finding repeated units, tracing local connections, aligning design references item by item, and prioritizing the issues that require an engineer's attention.
Highly deterministic differences, such as versions, identifiers, quantities, and explicit parameters, can be surfaced early. For process suitability, abnormal operating conditions, safety adequacy, and maintainability, an agent can provide evidence and review suggestions, but the relevant engineering discipline must make the final judgment.
Professional sign-off involves qualifications, responsibilities, and site conditions and cannot be generated automatically by a model. DesireCore's role in this setting is to reduce the time engineers spend repeatedly searching across drawings and design documents, while organizing the recognition results, evidence, and issues clearly. Agents perform parsing and preliminary review; qualified professionals still confirm the design intent, classify the issue, and approve any proposed change.
Overall, the central question in multi-agent engineering drawing parsing is not how much content the system recognizes in one pass, but whether every result can be located, reviewed, and corrected. The first drawing set establishes the workflow; later sets can reuse confirmed legends, components, rules, and lessons from failures. Only by preserving this experience can an engineering drawing parsing system improve through real use.
