Where should visual programming go?(tonsky.me) |
Where should visual programming go?(tonsky.me) |
For a little background, I’ve maintained that visual programming goes absolutely nowhere until we have visual diffs that work (work as in workflow). I’ve been saying that since before the UML Trough of Disillusionment kicked into high gear. Without diffs, without analysis, we are going nowhere fast. Almost every other link of the lifecycle is still intact with visual code but focusing on an editor without diffs breaks the chain. There’s no producing and maintaining commercial grade software without it. It’s either toy programs, or absolutely heroic effort, and what else could you have done with that much energy?
The epiphany was this: motion detection has been a feature of digital video since at least the MPEG days. Someone with a background in video compression needs to tackle the problem of doffing visual code. Figuring out how the code should look (mechanical sympathy) to facilitate this.
The victory of plaintext coding over visual programming is another example of worse is better. Semantic, language-aware diffs exist, but I see them much less frequently than dumb plaintext diffs. Intelligent code search exists, but in many cases grep is just as good. IDEs come with very advanced refactoring tools, but I still prefer to refactor using vim macros.
When we write code in plaintext, we're expressing our code in a lower level of abstraction with really great tooling. Doesn't matter if it's java or yaml or something custom—plaintext tools work on all of them. Visual programming languages can't do this. There's no language-agnostic vim or git for visual programming. Even if we wanted to invent sort of unified language-agnostic visual programming abstraction, the ecosystem isn't there, and there's no guarantee it'd get adopted.
So I think plaintext will remain king for the foreseeable future. Visual tools have to use human-readable, human-editable plaintext as their source of truth, if they want to succeed.
The pictures are easier to learn and use but they are not as expressive and precise as the more abstract text. Text is a marvelous innovation. The fact that most people on the planet can communicate complex ideas and emotions as well as program machines with it (in many different dialects) is mind blowing.
Using pictures to program feels like a regression.
I have an idea that I think would be suitable for visual programming. Every time I think about it, I come to the conclusion that first you need a DSL, or at least a JSON Schema or something.
If you have a DSL/schema, then you also have a way to diff.
Then, you implement your visual programming environment, and visual diffs turn into effectively communicating those differences visually. Faded color to indicate a deleted node and connector, bold parameter and value to indicate a change, etc.
In other words, not only does understanding these diagrams assume a ton of knowledge in the problem domain, but the crude tools at hand for creating them (lines, words, symbols, shapes, colors) can be repurposed in so many ways by different people and different projects that I don't see how generic motion detection will be able to detect high level changes. Sure, detecting changed nodes and descriptions might be easy, and possibly useful, but I think there will never be a substitute for human comments like "removed XYZ because the boss said so" or "renamed foo to bar because of IRS regulation so-and-so."
I suspect that the fundamental problem with visual languages is that you have to reify _something_ as the objects/symbols of the language. The most widely used text languages tend to be multi-paradigm languages which have significant flexibility in developing and integrating new abstractions over the lifetime of projects and library ecosystems.
It's not clear to me how this can be overcome in visual languages without losing the advantages, and instead ending up with a text language that is just more spread out.
It is a big problem if you are trying to develop a genric visual programming language. But much less of a problem if you restrict the problem domain to something like data processing, signal processing or image processing.
Anything else breaks the ability to use Git, or requires adding plugins which you might have to compile yourself or something, and I'm a big fan of just using Git and having VCS be a solved problem.
If this then that rules map nicely to LISP-like expressions in JSON or YAML.
Freely positionable elements are a pretty bad idea for a lot of things. They make extra noise in the diffs and can't be automatically formatted in a fully satisfying way, short of making an AI for it, plus they can't be edited easily on a mobile device, and they can easily become a literal picture of some spaghetti.
I honestly never use it because I only use unreal alone, so can't say how good or bad it is in real world project.
I get what you are trying to convey but that usage of the term is not helpful in the general case as it confuses what mechanical sympathy actually means.
Mechanical sympathy is simply this: abstractions that acknowledge the underlying embodying stratum of a dynamic construct. In code, that translates to software that attempts at harmonizing the software abstraction with the operational characteristics of the underlying computing device.
As long as there's some structured record of changes made to permissions and structure, those can be visualized.
I don't know of any non-proprietary tools, but at least the tech is out there.
Plenty of screenshots, a demo video, and more at https://github.com/williamcotton/guish
I'd love feedback, positive or negative!
A solution arrived for me in the form of a number of visual arduino IDEs. Flprog, Outseal, and Open PLC. They all allow programming in ladder logic, and Flprog and Open PLC allows for function block programming. Ladder logic is useful for many tasks involving discrete I/O (Ex. A button latches a contact which turns on a relay which turns off a light and turns on a motor.) Function block programming has enabled me to perform more complicated functions like creating an HMI using a menu system displayed on an LCD connected via I2C. These are tasks that I wouldn’t have a chance of accomplishing without the complex arduino code being contained in graphical functions blocks which I can wire together and configure with a gui.
Prior to compilation, Flprog displays the raw arduino code in the arduino IDE, which gives me a chance to exam the actual code produced by my function block program. I find it to be very educational.
As a non-programmer, I have no idea if this embedded programming use case can be generalized to more complex programming tasks, but for my limited use case it’s been transformational.
- building good UIs rapidly (e.g. improving & simplifying layout options, look & feel, etc.)
- make debugging trivial & powerful
- make deployment trivial
Visual Basic was on the right track in almost every way except as a language. Hypercard maybe had the language thing figured out: use english as a base?
Hooking up loops and conditionals visually is a waste of type, people can type it out with a little instruction with a good IDE & code completion that caters to non-coders. Focus on the tooling around the code & the things visual development has proven good for (laying out UIs, debugging) and then the stuff that people don't want to think about (deploy)
I just interviewed Daniel Kallin, the maker of the language/tool Nomnoml (https://www.nomnoml.com/), for making beautiful diagrams, which is one such place where it makes sense [0].
He had an expression I liked, wanting to make a tool/lang that felt like "drawing with ascii".
I think this is the way for visual programming. You must do nice DSLs side by side with the visual representation. The text must be source of truth. It _has_ to be that way! That's how computers work!
---
Don't let the hard to pronounce name throw you off, Nomnoml is a super fast and handy tool for making beautiful visualizations, hand crafted over 10 years and with no ads or login and fully open source.
[0] interview: https://pldb.io/blog/danielKallin.html
It has a real place among novice programmers. We even have some experts who use it as a fun alternative to writing text. I don't see visual systems as an effective way to replace everything us experts are doing but they've gotten a ton of mileage in the jr. and learning domain. Scratch (https://scratch.mit.edu/) being another obvious example.
See the email in my profile if you are interested in roles and I'll see if we can find something that fits.
Languages like Lisp are all about planning out the whole program structure (the "Abstract Syntax Tree") upfront, kind of like a blueprint. But modern tools and visual programming often take a more piece-by-piece approach. You build smaller parts first and then connect them together, gradually building up the whole thing.
This bottom-up method feels more natural to how we solve problems in real life. Visual tools make it even easier because you can just drag and drop components instead of worrying about complex code structures.
I wonder if this kind of visual, incremental approach could make programming more accessible to everyone. It might help bridge the gap between our everyday thinking and the formal world of code.
Languages in the Lisp family are good for prototyping and exploratory, interactive programming in situations when you don't know the solution upfront.
Bottom up from small pieces is ancient; it is not a modern technique.
Lisp seems like a poor example because the emphasis there is building incrementally in an interactive REPL session. I think this style of programming would be very amenable to visual programming.
((if (< 1 0) + *) 42 100)
More extreme example would be the various business process etc things, like KIE/Drools/jBPM, or BPEL/BPMN, that also allow various ways of mixing various visual and traditional practices.
I think the common thing with all of these is that in the end no developer actually liked them.
Now thinking of this, one of the problems is that there is no suitable generic graphics format that could be used as a base for these sort of visual programming ideas. So there isn't any text-editor analog for visual programming currently, instead all visual programming environments I've seen use some specialized custom tools and formats.
We need visual programming. No, not like that - https://news.ycombinator.com/item?id=40937119 - July 2024 (494 comments)
I didn’t follow that project for a while. It seems that their latest work is called Glamorous Toolkit, and the presentation is worth watching and relevant to this thread:
Visual programming should start in the debugger - https://news.ycombinator.com/item?id=40968215 - July 2024 (56 comments)
* Unit is a general purpose visual programming system* - https://news.ycombinator.com/item?id=40900029 - July 2024 (80 comments)
As a matter of fact, we _do_ have something that is more 'visual' or 'symbolic' than written speech, and it has a critical role in certain abstract forms of writing. It's mathematical notation (or, sometimes, symbolic logic) and it's already great.
Thus, any sufficiently advanced visual programming environment is indistinguishable from rendered LaTeX.
Not sure how much it is still being used at Itemis, at least is is still maintained.
http://www.codersnotes.com/notes/a-constructive-look-at-temp...
Jump to the section titled "HyperText" (DolDoc)
There’s a huge advantage to see programming for non programmers. Do you know how hard it is to explain hypervisors, VMMs and OverlayFS to non programmers? I gave them a simple example they need to make a simple text update to a Photoshop file. It costs a renewed font license for the latest but essentially an application library they can open the legal version they bought on windows see it look sort of okay looking on osx make changes to the file and be done with it.
One thing ChatGPT taught me was people don’t understand code. SQL was supposed to be ChatGPT of the day. Executives had no concept the “power” of essentially an NLP on top of structured data.
And for developers a high level AutoCad like view of what’s talking to what, etc. is valuable. I don’t need to visualize a merge sort but maybe a realistic abstract demo non programmers and programmers can work against.
I got tired of corrupt paste programmers mocking up design in Figma. I want to show you acquired Product X on SAP and it’s a different version. Using token data I want to be like as of right now you see realistic data and users, zoom in, show it merge in a datalake and produce an auditable report.
And that it runs strategies so algorithms (probably AI) can show aggregated date and where things might be failing. Zoom out see two icons from two company’s on different software and a generic data lake, zoom in and see and update data. See a message in Skack/email/etc. indicating whatever. Show elementary that consultant leads products like Twilio run off AWS and telemetry shows they don’t use but one feature and we can swap it out. All ok a running system without awkwardly switching windows. It helps executives get the big picture,’itnactualu works and you drill down enough you see type forwarding and an IDL for the programmers.
When was the last time you used merge sort not in std library or visualization would have helped you solve it? And yes I’m not talking just about corporate software.
Just the other day I was wanting to show an overlayfs/squashfs. I mean I’ll go down to C or god help me ASM so it’s not technical ability. It’s usually not even big vs little indian that trips people up but the big picture.
I wish so bad things like Salesforce would release their code, it is very rare someone would “steal” it and companies I know would benefit greatly and still pay full cost if I could rip out things and see their documentation is wrong and not have to go to COBOL or disassemble to me that’s visual programming. “INT32 actually a mutable type oh yeah they also made up their own types. That’s in red. In fact here’s a point map of all types that don’t match specs. First question why (in 1994 a driver didn’t work right so we did some weird type forwarding) ok… but in huge systems I’m not sure visual programming would help. Maybe weird things like btree in rust is better until you make an roc call outside of rust for some reason?
Maybe it won't happen during the next decades. Most likely it won't happen until all currently-alive maintainers of Linux "retire", and most programmers alive are from generations that grew up being pressured to use either Rust or React.
So I can easily imagine a future where there's two main camps: low level code in Rust (where "low level" now means "anything up to, and including, the web browser"); and high level code in some graphical thingy that compiles down to WASM.
And the path I see where we'll reach that point, is by the Rust community continuing to do their thing, and by SaaS companies continuing to do their thing (VM->Docker->"Serverless"[1]->"Codeless"[2]->"Textless"[3]).
With all that, I think it's possible that visual programming might become the dominant way of doing general programming. Not the only way (after all COBOL is still a thing today, so it would be like that), but I can see how what's "normal" is shifted up one level of abstraction higher, so that "code" in that future is seen the same way as "assembly" today; or "assembly" in that future is seen the same way as "punch cards" today.
Those old timers think their Rust language is good enough with their memory safety and stuff, and prefer to ignore decades of progress on programming. Separating the code in text files? Yeah no wonder they keep having incidents like the DeseCRATE or Cargottem supply chain attacks (years 2038 and 2060 respectively), if they install dependencies willy-nilly without even looking at the code they're bringing in (nobody wants to look at dependencies with that primitive tooling). If they used modern tooling instead, they would be able to easily tell at a glance which "nodes"[4] are trying to do suspicious stuff just by their location or relationships with other nodes; or even restrict network or filesystem access to a region of the canvas. They say "well, just use capabilities", but then the code looks like an unreadable mess and way too error prone, when modern tooling lets you just draw a square on a canvas and "any node[4] inside the square can read filesystem, everything outside it can't", without needing to modify any of the nodes[4] themselves. It eliminates whole types of vulnerabilities and whole types of human errors.
(/s, mostly)
[1]: Still needs servers, but you don't worry about it because servers are too low level and you only want to focus on the important stuff (just code).
[2]: Still needs code, but you don't worry about it because code is too low level and you only want to focus on the important stuff (just English).
[3]: Still needs text, but you don't worry about it because text is too low level and you only want to focus on the important stuff (just visual concepts).
[4]: Module, function, macro, etc.
It looks like you've got things sorted. But another option would be to write your flowchart down on paper ... get your idea out of your head in a visual medium, and then turn that into code without needing to hold the whole thing in your head at once.
If you're willing to pay a little extra for hardware you could go with Allen Bradley and their Connected Components workbench software.
I wrote some stuff in applescript, which is similar / descended from HyperTalk. I would say it's the worst language I've used; it's so hard to use because of pretending to be natural. It doesn't help when Apple forgot how to write documentation somewhere around 2008, so there's no authoritative docs, and it's unsearchable because it looks like english words.
Literally anything else would be preferable, but applescript is the best way to automate MacOS. Thankfully, I only had to use it once.
i don't think an english-derived scripting language would be a bad thing (well, I wouldn't, since I created https://hyperscript.org, now would i?) if the tooling was good, I don't see a huge difference between someone dragging and dropping a for loop and an if statement and just typing
for thing in whatever
if the thing's field is false
do somethingAllowing too large a subset of English ends up allowing more ambiguous statements and more user surprise. Also, more complex grammar increases the chances of mistakes in implementation.
My current experience with LLM hallucination makes it clear that at present, we can't just throw an LLM at the parsing and semantic analysis side of programming language implementation.
the old hypertalk (and, later, appletalk) writing environments were terrible, that's where technology could really help
If I were to build a poor man's effigy to visual basic today, I'd use Go as the underlying language, and Fyne for the visual kit.
- Go has strong concurrency that is easy to understand
- Go has easy syntax that makes errors very obvious and debuggable
- Go is multi platform, truly, from OS to hardware, right out of the box
- Go with Fyne can produce a single binary .exe just like visual basic.
- Fyne retains the same look and feel across different environments
Ever tried Delphi (or nowadays, Free Pascal / Lazarus)?
I first encountered Visual Basic, and it felt... Almost there. Then i encountered Delphi, and it was love at first sight.
Basic demo https://www.youtube.com/watch?v=XYZsvouytVQ
Make the abstractions purely high level, and let people write Python plugins for new blocks with an app store to share them.
Visual can easily provide enough flexibility for gluing blocks together.
IFTT, Excel, and lots of others do it perfectly well.
The issue is programmers like to program. Mostly they like "programming in circles", making little apps to help make more other little apps to explore ideas.
They see it mathematically, as a human centered activity all about understanding, while users see machines as a box you put stuff in to make it not your problem anymore.
They're always talking about empowering users to create their own fully custom ways of using computers... But.... I like apps that have a canned, consistent workflow that I don't have to fuss with or maintain or reinstall.
Software like Excel has the appropriate amount of power for working on a project that uses computers but isn't really related to programming or done by developers.
In Common Lisp one would need to write
(funcall (if (< 1 0) (function +) (function *)) 42 100)Also, the above can be further shortened to:
(funcall (if (< 1 0) #'+ #'*) 42 100)