Can GPT-6 Astra Make Technical Drawings? Read the Benchmark Twice

The number doing the rounds since September 3 is 95.9%: GPT-6 Astra's BenchCAD score, up from 83.3% for the model before it. It is a real jump on a real task. It is also being read as “AI can do CAD now”, which is not what that benchmark measures. We went and read OpenAI's own chart instead of the headline, and the axis labels tell a more useful story: the score is a volume-overlap metric, measured with a Python tool in the loop, at the expensive end of a cost curve that starts near 67%. Then an independent tester found the failure that makes the point: a part at 96.1% similarity with the hole still in the wrong state.

The short answer

Astra can write real parametric CAD code, which earlier chat models did badly, and running that code produces a real solid model you can export as STEP. That is the genuine advance. It does not produce a dimensioned manufacturing drawing, it does not measure a physical object, and the headline score should be read as “most of the volume matched” rather than “the part was right”. Everything below is the detail behind those four sentences.

A 3D model is not a shop drawing

This trips up nearly every discussion of AI and CAD, so it is worth being blunt. Two completely different things get called “doing CAD”:

Astra's CAD result sits entirely in the first row. The benchmark hands it rendered views and scores the solid it builds. Nothing in that loop produces a dimensioned sheet, chooses which views to project, decides what to tolerance or fills in a title block, which is the work that makes a drawing manufacturing-ready. If the thing you need to send a supplier is a drawing, a strong 3D-reconstruction score does not get you there.

What the 95.9% actually measures

BenchCAD gives a model several rendered views of a part and asks it to write executable CAD code, in practice CadQuery, a Python framework that builds solids. The script runs, produces a solid model, and the result is scored against the reference shape. That much is in every write-up. Open OpenAI's own chart, though, and three labels change how the number should be read.

1. “With tools” means it got to run the code

The panel is titled BenchCAD (python tool)and OpenAI's caption says with tools. So the 95.9% is not a model looking at renders and emitting correct CadQuery in one shot. It is an agent loop: write code, execute it, look at what came out, fix, repeat. That is a fair way to run the benchmark, and it is also how you would use it in practice, but it means the score describes a workflow with a Python interpreter in it, not raw model output you can paste into a file.

2. “Mean voxel IoU” is a volume score

The vertical axis is not “parts correct”. It is mean voxel IoU: chop both shapes into a grid of little cubes, count the cubes the generated part and the reference part share, divide by the cubes either one occupies. It is intersection over union, applied to volume.

For anyone who makes parts, that has a specific and uncomfortable consequence. A feature costs you only its own volume.A missing Ø8 hole through a 3 mm plate removes about 150 mm³ from a bracket that might occupy 15,000 mm³, so filling it in costs roughly one percentage point of IoU, and the part still scores in the mid-nineties while being unusable. High voxel IoU means “the block is the right shape”. It does not mean “the holes are where the bolts go”.

3. 95.9% is the top of a cost curve, not a single number

The x-axis is API cost per task, and Astra is plotted as a curve rather than a point. At its cheapest setting it sits near 67%, climbing to the headline 95.9% as spend per part rises. So “Astra scores 95.9% on BenchCAD” is really “Astra reaches 95.9% when you let it work at its most expensive setting”. The efficiency story is genuinely strong, OpenAI puts estimated cost about 43% below Sol and 86% below Fable 5.1 at the configurations shown, but a cheap run is a different model of the same part.

One caveat OpenAI puts in a footnote and most coverage drops: the Claude numbers in that table reflect three modifications to the eval, detailed in Anthropic's own system card. Cross-vendor rows on a chart published by one of the vendors are a weaker comparison than they look.

Three things the score does not tell you at all:

• Whether the feature order is sane. Overlap is computed on the finished shape, not on how it was built.
• Whether it works from a photo. The inputs are clean renders. A phone photo brings perspective, shadows, reflections and a background, none of which mean anything geometrically and all of which read as edges.
• Whether the size is right in the real world.A render carries no scale. Reconstructing proportions is not the same as knowing the part is 82.5 mm across.

The catch an independent test found

The most useful published test so far is not the benchmark, it is a single part that failed in an instructive way. On a motor end cover, a central through-hole was supposed to be created. Several models cut the hole too early in the build order, then added a cylindrical boss afterwards, which partially closed the opening again. The geometry still came back 96.1% similar to the reference. The requested modification was simply not implemented correctly.

The practical instruction that falls out of it: if you do generate a model this way, review the operation order, not just the render. Open the feature tree, check that every cut happens after the material it is supposed to cut through, and check every hole is still a hole at the end.

The other route: asking it for a picture

The second thing people mean by “can it make a technical drawing” is the image route: ask for a blueprint, get a PNG. Astra does not render images itself, it takes image input and produces text, and calls a separate image model as a tool when the product gives it one. So that route is unchanged by Astra's reasoning improvements, because a different model is drawing the pixels.

We tested that route properly, with one prompt run twice, and got two drawings of the same bracket carrying different numbers. The write-up is in Can ChatGPT Make Technical Drawings? Not Ones You Can Cut. Nothing in Astra's release changes that conclusion, because the constraint was never the language model's intelligence: a raster image has no geometry in it, so its dimension text is decoration.

What we could not test, and why

Being straight about it: we did not reproduce the BenchCAD result or run Astra on our own part for this piece. Our ChatGPT account is on the Free plan, and Astra ships on Plus, Pro, Business and Enterprise. When we asked the model we do get whether it was Astra, it said no, which is documented with the screenshot in can GPT-6 Astra read CAD files, along with the two things we did verify hands-on: reading a DXF as text, and reading printed dimensions off a drawing image.

So treat the numbers in this article as what they are: OpenAI's published claim and one independent tester's finding, both cited at the bottom, neither of them ours. If you have Plus or Pro and want to check the operation-order failure yourself, the recipe is in the section above and takes about ten minutes.

What this means in practice

• Concept geometry, yes.If you want a parametric starting point you will edit anyway, generated CadQuery is a reasonable first draft, and reviewing code is easier than reverse-engineering someone's modelling history.
• Production parts, verify everything. Feature order, hole states, wall thickness. The similarity score will not catch what breaks.
• Drawings for a supplier, not yet. Nothing in this release produces a dimensioned, toleranced sheet.
• Anything starting from a real object, no. Renders in, model out. There is no measurement step anywhere in the loop.

When the part is in your hand

The common real situation behind all of these searches is not “reconstruct this render”. It is a part on the bench, no CAD file anywhere, and somebody asking for a DXF by Thursday. That is a measurement problem before it is a modelling problem, which is precisely the step benchmarks like BenchCAD skip by starting from clean views of a known object.

It is the step our AI technical drawing generator exists for: the photo plus one reference dimension you supply, and it rebuilds the part as a dimensioned 2D drawing exported straight to DWG, DXF, SVG or PDF. The walkthroughs are in from photo to manufacturing drawing and how to reverse-engineer a part from a photo, and if you want to check the output rather than trust it, open the exported file in our DXF viewer and measure it.