Text2CAD: Generating sequential cad designs from text prompts(sadilkhan.github.io) |
Text2CAD: Generating sequential cad designs from text prompts(sadilkhan.github.io) |
I was an engineer in a former life but still do a fair amount of printing. But when I design parts there's not even ways I __could__ know what I want before hand. As I build I realize I made wrong assumptions, but also not enough, that there are better ways to do things, that I can solve other problems, that I didn't think how things would interact together, that I could modify things to be better for the manufacturing process (this is such a big on in 3d printing and so many online files get this VERY wrong. But it is a hard skill to learn), and so many other things. In part this is because I've had more time to think, but there's more to it, when you see the thing "coming alive". This is much the same way I code, though I guess that's not common.
I'm not sure if they'll address these problems, but I think anyone working in this space should make sure they also spend a lot of time in CAD themselves. It isn't clear to me that any of the authors do (looking at their websites. The main author mentions interest in AI-CAD but only has work from this year. Get this man a 3D printer). It's quite possible that they do, but they look like they've been computer scientists their whole career and that is probably not enough to understand the the intricacies of the problems they're trying to solve. There's a classic problem in CS where people get that you can learn a lot of things quickly but it is missed that getting the nuance and mastery takes time, that you should talk to experts. The first part is useful because it gives you the language to talk to experts, not because it makes you a replacement for one.
Yep. I have two printed prototypes of different approaches to a mechanism on my desk that only exist because of months of staring at CAD in the evenings, learning new things, doing research.
They are not radical (they may be slightly novel in places; I have never seen 3D printed mechanisms like them).
I don't know if I could describe them in words at all, but if I could, it would only be because I worked through them in CAD in the first place.
For anything other than a trivial object I just can't see how you'd even come up with the words without having worked through the design -- what, on paper in 2D in pencil? After doing the maths? That's CAD in reverse.
I do wonder how much progress we could make on a problem like this with a 3D transformer architecture.
For a user of a UI like this, there comes a point where their time would have been better spent learning a CAD package.
Another layer is if you are modelling something that has to be machined or built in real life, you have to be keeping an eye on how it will physically exist throughout the entire process, stock it will be machined from or materials it will be built with. Thinking in terms of CAD workflows help with this greatly in my experience. The operations shown in the demo are not only easier to perform in a CAD package than describe in English to an LLM, but also the easiest part of it (except maybe if you are designing strictly for 3D printing)
I'm the lead author of Text2CAD, and I’d like to start by expressing my gratitude for all the support and constructive criticism on our paper. As an AI researcher, it's essential to engage with diverse perspectives, as these discussions spark new ideas for improvement and growth. I would like to answer a few questions raised in the discussion.
Final CAD Representation: Our CAD model representation is inspired by Onshape's Featurescript, a code-based CAD model generator. We opted for this since the available open-source datasets are from Onshape. The final CAD model is output in a sequence which can be transformed to .stl or .step format, using PythonOCC. These sequences are editable as it contains parameters for sketch and extrusion. In the future, as Text2CAD matures, we plan to support direct Onshape Featurescript generation as well. We're also planning to release a demo on HuggingFace soon, free for anyone to use. I would love to collaborate with the community for Onshape Featurescript generation, so if anyone is interested in contributing, feel free to reach out via email!
LLM Usage: I want to clear up any confusion around how we use large language models (LLMs). Right now, LLMs aren’t involved in generating the final CAD model. Instead, we use them to help create the dataset (for example, text annotations). The actual CAD generation is powered by a Transformer-based architecture that works in a sequence-based approach (sketch-and-extrude). That said, integrating LLMs into the CAD generation process could very well become a reality in the future, and we’re keeping a close eye on that evolution.
Simpler Shapes: Some people have pointed out that Text2CAD currently generates simpler shapes, and I agree. This is mainly due to the limitations of the datasets (not the method itself) we’re working with, which are restricted in complexity. We can currently only handle sketches with basic elements like lines, arcs, and circles, alongside extrusion operations. The creation of new CAD datasets is challenging due to IP restrictions, and as many of you know, great AI systems are built on strong datasets. So it will take time for open-source research in CAD to be fully usable for application. However, we’re hopeful that as more complex datasets become available, you’ll see far more advanced possibilities from Text2CAD and its successors.
Theory vs Practical (What is Text2CAD's contribution ??): As an AI researcher, it’s crucial for us to balance theoretical contributions with practical applications. Translating geometric and parametric information from text into 3D models is a significant open problem in NLP. With Text2CAD, we’ve taken a step toward addressing this by providing a dataset that others can leverage for further research. I understand the sentiment that "a picture is worth a thousand words"—the future of CAD is undoubtedly multi-modal, much like GPT-4, where both language and images will coexist. For this to happen, AI agents must deeply understand both modalities. Ultimately, it comes down to personal preference: whether a user finds working with text or images more intuitive. This question is just as relevant for other text-to-3D projects as well.
Assistance vs Replacement: In my website, I use the term “AI-Assisted CAD Modeling” to emphasize that the goal of our research is to support CAD designers, not replace them. The primary aim of Text2CAD is to streamline and speed up the prototyping process, rather than creating complete models from scratch. I recognize that not everyone may enjoy specifying numerous parameters, so we've also included the option to work with shape concepts without parameters. As with many tools, it comes down to personal preference, whether one prefers a more hands-on approach or quicker, conceptual modeling.
Future Goal: Looking ahead, I’d love to hear from you —especially from CAD designers— about how you'd like to see AI assist you in your CAD designing process. Your opinions are important for AI researchers like myself, as they help us design tools that are truly useful and aligned with real-world needs. Reach out to me if you are interested.
Cheers.
this is usually before i start adding intricacies such as shells, fillets, and other features, which do take a lot of effort, but making those by hand is more of the 'art' side of the process anyway
anything that gets me through the first 98% is welcome :)
Yes, for me that's usually around the point I've got a huge and complex assembly with all the motion wired up. Right-click -> Duplicate -> Rename "MyProject V2".
I would save a huge amount of time also this way.
Only if the result of the LLM is a good, well-architected parametric CAD model you can adjust, right?
as long as the output is something my manufacturer can understand (downloadable mesh: STL/STEP/etc (they dont take links)), the tool did its job for me
i would probably start the final model from scratch no matter what the output was, so i can match my chosen manufacturer's tolerances/design rules/optimizations, and to give breathing room for my quirks/workflow (i like to design subtractively, some people design additively)
It's still an interesting applications that might have a better future concentrating toward simulations (Salomé MECA, CodeASTER etc) since for these systems it's more tedious designing the simulation by hand than just sketch a 3D part with a modern parametric CAD.
presumably a big benefit here is that all it really takes English and geometry knowledge.
seriously, I think people overestimate how hard basic CAD work is
A beginner using FreeCAD or Fusion 360 can start learning today and will know how to design all those parts by the end of the week.
> Is there a blenderGPT-like tool trained on build123d Python models?
> ai-game-development tools lists a few CAD LLM apps like blenderGPT and blender-GPT: https://github.com/Yuan-ManX/ai-game-development-tools#3d-mo...
Tinkercad. https://www.tinkercad.com/
I teach children how to use it. It takes them about 15 minutes to pick it up. Are you going to design a car with it? No. But if you look at all the Text2Cad programs so far, they won't either. If you need to design something for 3d printing, Tinkercad, in terms of ease of use/simplicity, is hard to beat. And it's free to use.
I've played with other Text2Cad projects, and I've yet to be able to get anything out of them other than the most simple shapes, which frankly, what's the point if it can't make an object that's useful or meets my requirements? It takes way too much time having to write out paragraphs to even get the most basic of brackets built. I remember on the Zoo app, it very much likes typical CAD language which means you're going to need to be able to speak "CAD" anyways when trying to create a successful part (think: constraints in typical CAD)
What these LLM programs really need to focus on (imo) is writing Openscad (and similar "programmable CAD") code, which is going to at least give the user a better starting place, as well as the ability to edit their parts. I think one of the biggest constraints for this is the lack of millions of lines of code and documentation unlike most popular programming languages for the LLM.
I think the research is neat, but for now, it seems like a solution in search of a problem.
I can see this being incorporated into existing software as an alternate workflow path.
What caught my curiosity here is the "sequential" qualifier. One massive weakness of all the AI content generation schemes is the lack of editability -- you get what you get, and attempts to refine the initial results are middling at best. This seems like it allows the user to build a more complex scene from multiple prompts -- likely meaning you can go back and edit some of the prompts to tweak the building blocks, and edit the overall scene. Interesting!
I think with a long preamble in the prompt (you are an expert at designing injection molded parts, and so on) you could get something pretty useful.
If I could feed the manufacturers spec sheet along with maybe some pictures of the items and marketing copy then get a 3D model out it would save me a ton of money.
But "Make me a teapot" seems to be aiming for teapot-looking object, which cannot hold tea. But this is not failure perse.
Totally functional tea-pot production solution already. All you need to do is to automatize the display function and STL-production and any granny can print her own teapot without understanding nothing about computers and 3D-printers.
If you can tweak the model in a CAD package, you can quite possibly make it in CAD in the first place more quickly than you can manage several rounds of descriptions with an LLM.
It's like the whole "I'd like to get an LLM to write a song and then I'd adjust it" thing. No musician needs this. If you can truly fine-tune a song, writing one is easy. And if you aren't a musician, you won't get good results fine-tuning a song.
for some users, they will think of few enough designs in a lifetime to make learning any specific software worthwhile. For these users, the LLM's inefficiencies are worth the trade-off.
This whole idea seems contingent on imagining a situation where a non-CAD user has an idea for a truly novel physical object, has extensive geometry skills, and can describe that object in some magical level of detail that doesn't involve any terms of art from the CAD domain.
It's not a very likely scenario. And the energy put into tools to support this scenario would be better spent improving searchability of the data that is going to go into the training set, and simple tools to allow objects in the training set to be combined (such as those offered by TinkerCad or Microsoft's 3D builder).
It's also prone to the risk that the LLM gets something wrong: makes a part that is prone to failure, or will actually destroy a CNC, or be unprintable by a 3D printer, etc.
More importantly, creating a 3D model without understanding mechanical properties is a meaningless exercise. Go ahead, ask anybody who has built things about their first attempts - and this approach means they will always be first attempts.
It's like saying I only want 2 sculptures on my garden so we should make a thing that sculpts like Michelangelo because I don't want to learn to sculpt for only two statues.
This is why we have civilization, trade. We can each specialize, master, one thing, then share our surplus for others'.
Why wouldn't you hire someone who can do it on an hour?
This is what I think, and yet I am the longest time away from being an expert.
It's easier because CAD tools unlock CAD thinking and empower your brain to actually do design, as well as helping you see problems you didn't anticipate.
This whole area -- LLM to CAD -- is one of the most misguided applications for generative AI (beyond "generative design" as it was understood in the pre-LLM/pre-GAN era, where it was usually used for FEM/topology optimisation)
There are already enormous libraries of freely available basic CAD models for real-world objects; any beginner would be much better off simply learning how to merge them. And any tool aimed at beginners would be better off assisting that process (TinkerCad does, for example)
And if a beginner has a truly novel object to make, an LLM is not going to have the training set data to make it. Nor is the beginner likely to have the CAD knowledge (words, expressions) to describe it.
For this to be of use to a beginner you do have to imagine quite a niche kind of beginner: one who is expert in descriptive language and advanced geometry. Those people would be better off learning some sort of CAD environment; indeed they are the niche that is least likely to be driven insane by the limitations of OpenSCAD.
But this is actually like a lot of LLM applications. For example: it's not terribly hard to learn to write a song; you don't even need to know an instrument. The songs LLMs write aren't better than novice songwriters. The difficult bit is turning a basic song into a great song, and novices who've skipped writing the song itself won't be better at it.
> It's also prone to the risk that the LLM gets something wrong: makes a part that is prone to failure, or will actually destroy a CNC, or be unprintable by a 3D printer, etc.
It's a real "too clever by a quarter" thinking
Even if you could do what you're suggesting with an LLM (I have my doubts) this result would be a mesh or 3D pixel grid or something, yes?
This is terrible for interoperability and it's the opposite of what mainstream CAD packages do.
I think this is one of those things that programmers overestimate worse than non-programmers, too. To the point that they reject CAD UIs too early and get themselves stuck in often rather limiting code-CAD environments, because they never get to learn how parametric GUI CAD works.
This belief that only code can be intuitively parametric is obviously not something that non-programmers suffer from.
I think code-CAD has many benefits (though the idea of the various LLM-to-OpenSCAD tools out there makes me shudder; this is the worst possible combination of obscure knowledge-bases).
But just a trivial amount of time learning even FreeCAD (the least-intuitive CAD package, pretty unambiguously) unlocks so much potential.
For folks that don't belive us, check out "speed modeling"
I'm still really quite green at CAD; I've come to it late in life and I only make relatively simple things, perhaps; only simple mechanisms.
But when I look at people getting stuck and asking for help in CAD groups it so often comes down to the knock-on effects of very early mistakes, like squandering the benefits of the base planes by choosing the wrong initial orientation, muddling through with primitives when an extrusion of a sketch would do the job, or making a series of complex circular pockets when a single revolve could have done it better.
Basic familiarity with a few principles and their expression in CAD, and a little study of existing objects gets you a long way.
It interests me that programmers are willing to learn the expressive nuances of individual languages or libraries or methodologies, but as soon as it comes to GUI CAD they dismiss the whole thing as too hard or too obscure. The excitement around LLM text-to-CAD seems emblematic of this; as if the wrong conclusions about GUI interfaces have been drawn from bad experiences of dev GUIs.
Asking me or folks that do what I do to create a hardware product that way would be like asking a sculptor to sit down and write NC code for a marble router to cut out the sculpture they see in their minds eye.
When we're using these tools, Solidworks in my case, we're not just clicking with the mouse. We're typing complex commands, creating and using variables, creating scripts and macros, we often use gaming mice with many buttons mapped to complex hockey's, etc.
The graphics interface is just part of it, the part where it shows us 3d geometry so we can put it into our minds eye, and a way to tell the computer what geometry we are talking about before we execute work using what I described above.
Most people don't get it, if you do you do. You do.
There's definitely things I think I could describe in words, but without a doubt could be communicated faster by sketching. There more complicated things where I think it would just be faster to cad up the damn thing. It's like math (or code). The language(s) are precise and annoying because of that precision, but they're still the easiest way to do the things we want to do, which is why we use them. Natural language's flexibility is great for abstraction and big ideas but not so great when it comes to precision. Things get very wordy very fast when you get into the details. And I'm sure everyone knows the value of arguing with your friend or coworker over those tiny things, even if it doesn't seem important. If you don't, you probably need to work on teams more often or make more friends lol
Not to mention that in this case, this disagreement over the meaning of ultra-fine detail will be happening with an LLM, which does not really understand the words.
I can't be the only one, right? I feel like I'm being gaslit lol
Equal to inject-molded ABS cutlery.