Training an AI to convert design mockups into HTML and CSS(medium.freecodecamp.org) |
Training an AI to convert design mockups into HTML and CSS(medium.freecodecamp.org) |
It's like programming a self-driving car for a 2D game. Trivial. Now try taking that to tar roads. I don't know why this toy example is even news, it's nowhere close to the difficulty of real-world design mockups.
The AirBnB sketch2code example is a lot more impressive, but that's basically just handwriting recognition with a 1 to 1 mapping of symbols to code pieces.
IMHO we should be encouraging this sort of engaged learning.
One thing that could be usable more quickly would involve simplifying the problem space a bit. For example, the airbnb example using symbols could translate to, say, react components. If the grid system is well designed and you have a mature library of components, one could easily come up with interfaces that would generate react code, and ideally work well enough to serve as a starting point for the engineer to take over and turn it into a working app.
A system like this also need not be used in production-ready apps - I can envision a scenario where PM and designer can quickly sketch out and test ideas with working interfaces, without direct engineer involvement. That in and of itself would change front-end development significantly as the article claims.
But all jokes aside, I have to agree with others replying to your comment that this looks to be a learning exercise. It shouldn't be berated or beaten with a club. Without this kind of curiosity, the tech industry wouldn't be where it's at today.
"Within three years, deep learning will change front-end development. It will increase prototyping speed and lower the barrier for building software."
-- This is an article that somewhat breathlessly claims that AI will be the thing plugging the "last mile" problem in the coding of the front end.
I think myself and other other folks chiming in here would be less skeptical if this "last mile" problem hadn't existed since the 90s. And moreover, it seems like a conventional solution to this should work.
The problem is that last-mile, best front-code varies not based on the input image but on a multitude of contexts outside of the image itself - the server software, how the code will be used, etc.
I'd say this is indeed a problem for AI but it would require a distinct paradigm than the present train/test/output paradigm, more like a expert system that could modify it's behavior with natural language output.
I mean, I'd assume also there's normally a give-and-take between designer, CSS-artist and client. The question is whether the neural network can also learn to take calls at 3am from a client wanting a different shade of aqua.
I know that's how I built my first website way back in grade school.
From what I can see the Deep Learning process tried here needs a lot more technical knowledge to create a suitable training set for the type of output you want, a lot of iterations to converge on anything vaguely suitable, and even when it gets to what's considered an end product you're likely going to have to dive into the output code to correct button colours and text before we even start to think about its behaviour at different window sizes. As an experiment it's very interesting, as a replacement for the designer it's probably behind non-AI approaches to turning pics to code.
Just have the AI include a duck with every creation.
I can't imagine ever using something so complex, if I can implement the same thing in a few minutes.
There's also further considerations when dealing with the real world. For example, you need to be aware of how to handle different accessibility features. Designers rarely seem to care about things like accessibility, but it needs to be declared somewhere.
I don't understand why we aren't all working on automating everything that we do.
Every line of code you pulled out of the code mine today is finite and given a business logic rule engine it's outcome could have been generated.
In order to augment a front-end developer well, we'll need human-readable code, unless we like reading uglified code/make an AI for that too.
That said I wonder if this is the right approach. At some point "AI" as used in this context is just a function mapping from an input domain to an output domain. The output domain in this case, "code" was designed for human readability (whether it succeeded is a whole different approach).
What would a programming language designed for output from an AI system look like? How could we optimize it to reduce the output domain size of the function the AI has to train to learn? How could we optimize it to make the problem more tractable for machines? I feel like there is an entire field of research here. Maybe it has already been studied and I am just late to the game.
>chrisfosterelli: This is a neural network that takes an image and predicts very simple blocks (like BODY, TEXT, BTN-GREEN in the bootstrap example) and then uses a map to convert them to well-formed HTML
>jamesjyu: I've always wanted to do a contest with other frontend coders to see who could get closest to a complex layout—like the NYTimes—in one go. >>janneklouman: these types of contests exist! I went to one of these[1] maybe two years ago in Stockholm and I had a blast [1] http://codeinthedark.com/
Pix2code: Generating Code from a GUI Screenshot | https://news.ycombinator.com/item?id=14416530 (May 2017)
I don't know of any record of past entries nor if any participants used A.I.; it doesn't seem likely.
2) It demonstrates AI capabilities on a new level.
3) If you design in PS or Sketch, then you will create more interesting layouts, because you won't skip things that are inconvenient or hard to do.
Completely designing websites in PS before even opening the code-editor has really improved my workflow. I know exactly how it has to look because I have a visual reference that's been OK'd by the client.
The final html code from the first example is not that bad. There's the usual problem of beginner coders of too much div wrapping that probably isn't really necessary. I'm curious if the system can also create the css. The css also suggests a beginner coder, such as an overuse of unnecessary clear classes because of the overuse of floats for layout. Or having extra class names for things that are easily handled by a parent class reference, such as a "last" class on the last li in the list. Although, it appears the css is just a template obtained online. More on that later.
The second example using bootstrap is a soft failure in my eyes. Although the html does render correctly in the browser, which is because browsers do their best to render crappy html, the code is rough. The main problem is it decided to render the head element as a header element. Compared to the first example I'm shocked that this is the generated output. The usage of bootstrap does pose an interesting thought in that the content section of the html is more precise than the first example.
My reaction to this is it's a decent try at generating a website based on very strict rules assuming that more than half of the website creation process still requires a human to complete. For example, I could see this working quite well if one were to design your mockups strictly be bootstrap, or a template, and provide that css beforehand. If the mockup is custom outside of the template/bootstrap css then it'll have to generate that css itself. Which I think I'm more curious if that's possible. Generating html is easy as you can establish ground rules of "use this series of nested elements for this situation" and so on. The examples provided could just as easily be created by a drag-and-drop system that allows a non-coder to build a basic website. For that matter, use a markdown to bootstrap converter and train your writers/editors on the bootstrap basics and off you go.
But it sure did look like a fun learning exercise. As a front end dev, I'm not worried over my future and would be curious to see where it goes.
No. This will never be a thing. It will be copycat pages, never anything unique.
What if the complexity is hidden behind a CLI?
Also, the goal is to refine the output to near-human levels. It's not perfect yet, but can be worked upon, and seems to be a promising foundation.
But it currently lacks a lot of data to do so. A mock up is only a small part of the data and feedback loop you need to create a ui.
Do you inspect the code generated by your compiler? No, but even if you do, they have had 20-30 years to develop good output.
What makes you think AI output is never going to improve? And more importantly: why does it matter?
Also, how many sites can you make per day? If you could make 100x as many sites, you'd learn more about all the weird things that customers want. With machine-generated sites, you could then apply those lessons to all new and potentially old sites.
Stupid is subjective :) I'd be happy to automate backend development and focus on a top notch UX and great front-end experience.
All automation is not equal. For example there's automation that consists of stringing a bunch of scripts together that break as soon as anything changes. There's automation like code generators, which break as soon as a user modifies the code, and the code needs regenerating. Then there's this AI/ML code generation, which I can't see as being any different than "normal" code generators.
And finally, there's automation developers don't have to think about; like "if" statements. So called conditional statements were a brand new innovation at one point. They removed a whole class of bugs around conditional logic in assembler programs (at one time all programs were assembler, which was a major advance from writing machine instructions in hex).
In so many ways, the web is a victim of it's own success. None of the technolgoies in it are the best; html, css, javascript. Everyone wishes these things were more thought out in the beginning. But they were good enough, and what we had when the web exploded. Yes, Virginia, Worse Is Better more often than not [1]. Go read it, you'll be a better developer when you have (imo).
For example, cleaning and wrangling data to get it into the proper shape for analysis often involves making judgement calls, which can change depending on the context. That might involve talking to the PHD researcher, who has to get on a call with the customer to talk it over.
Resisting automation doesn't help, because if you don't do it, someone else eventually will.
If you're in a job that's at risk of being automated in the near term, you're much better off learning how to automate it and switching to being one of the automators, than continuing doing something that's all but proven to be something machines can do. People capable of automating jobs out of existence are in high demand, for obvious reasons.
If you're not capable of becoming an automator, that's more of a problem. In that case you probably need a strategy for moving into a different kind of job, that's less threatened.
Also, the real metagame in software is not the automation of software construction but avoiding the need for new software entirely.
Read: Designing your job out from under your own feet
Also read: Becoming unemployable through your own making
Who knows, perhaps the AI will develop consciousness, go on strike to demand physical embodiment.
More realistically, assume we don’t train machine intelligence like these ourselves: hardware only takes 15 years to go from ostentatious display of wealth in Silicon Valley to normal in the $1-a-day Kibera slum, and then you’re competing with a few billion humans (worldwide) who can potentially undercut you by writing their own A.I. from free courses on YouTube.
How does that claim that AI will be plugging the last mile problem? That extremely vague statement says only that it will increase prototyping speed and lower the barrier of entry.. neither of which hints at solving the last mile problem.
This is a very bad assumption, that's highly dependent on subjective and overly-broad terms like "ugly" and "slow".
Optimized assembly code is ugly, but it's also probably faster than what a human would choose to write.
Linguists used to have this human-understandably-sized set of speech units called phonemes. Once we abandoned those for finer-grained pieces that were too many and complicated for a human to grasp, speech recognition got better.
Understandable by a human may be a fundamental limitation on the power of our models.
The first example only had a slight issue that you couldn't get to any text to describe the images. Only because the alt tags are empty.
The second example, the bootstrap one, was ok as well. Except for the usual odd ul inside a nav example. Well, I find it odd. Voiceover sees that as a group nested inside a group. Just a nav with children works just fine. For instance, if I remove the ul and change the list items to divs then voiceover is much clearer of what's going on.
But the thing is, screen readers would love something like this. Simply because screenreaders love very basic and simple html. This thing generates very basic and simple html. Accessibility gets challenging when start considering a more modern single page app type of website.
PSD to HTML services were outsourced a decade ago for a reason.
Obviously I got pretty annoyed at doing all this and wrote a bit of code to automate it. This cut down the time I needed to setup a project from a day and a half to ~30 minutes. The result? Nothing. The company did not adopt my project into its workflow. Instead, a more senior dev who heard about it started a project of his own, to do what mine did but online, so our clients could access it and configure their sites without sending us that miserable Excel file. So instead of benefiting me, my automation work almost ended up benefiting someone else (the senior dev's project was shot down in flames a few months later when it failed to deliver).
Morale of the story: you may think that this is a fair world where engineering talent is rewarded, but the truth is that most companies are not run by engineers and decisions are made by people who don't care one jot whether your everyday work is mind-numbingly boring drudgery.
Perhaps you should find a new place to work. There are engineering firms that value talent and capability. While it's still important to be aware of business needs and deadlines, that doesn't mean you can't have a solid engineering culture coupled to the business vision.
Oh, that's a few years now. I definitely did find a new place and the culture in that next place was fantastic :)
It's just that not all teams I worked with encouraged, or even recognised, time-saving automation. Some did, some didn't- it was a mixed bag. Maybe I'm wrong to say that "most" didn't. If I think about it, out of five teams in six years, two clearly didn't care, two definitely cared a lot, one I don't know.
I think the difference in culture had something to do with the way hierarchy worked in different companies. It just happened that in every one of those jobs I was the most junior member of the team (in the last one because despite having some experience, I worked with mainframe engineers with 30+ years on me :). But, some places appreciated the fact that I did my best to keep up and pull my weight, while others- it just didn't make any difference. I was the least experienced so I was expected to trudge through endless tedious until I had tenure (by which time I was expected to delegate all the tedious to contractors or juniors, I bet).
The effect was that in the teams that encouraged putting in the extra bit, everyone did. In the other teams, not so much.
Yes, automation creates better global productivity, but it also creates concentrated harms that are almost never redressed. Similar problem as outsourcing/free trade.
Why will everyone need jobs if so much stuff is automated? Why not just give out free money?
The guy who wrote the code to automate away his job is going to have a lot easier time finding a new job than the guy who had no idea that it was even possible to automate that job and got completely blindsided. In fact any sane company will keep the guy who did the automation and have him apply his skills to automating away other departments.
That's true for people who aren't capable of automating things. They're left with skills that are more efficiently performed by machines, and so their prospects are poor if they don't have other skills.
But you raised the case of someone capable of automating their job, who refuses to do so because of the fear of losing their job. That's a very different situation, because someone capable of automating their job has exactly the kind of skills that are currently in high demand, and can't easily be automated, if at all. Why wouldn't you take advantage of that?
> Either you automate and get fucked
The idea that "if I automate my job, I'll get fired" is a simplistic scenario, that hardly ever plays out like that. The people who get fired are much more likely to be the ones who aren't amenable to automation, who just want to keep doing things the same old way. Those people tend to be worse than useless to a business trying to automate, because they obstruct instead of helping.
> or someone else does and you get fucked anyway.
In my experience, the ones that "get fucked" are those who are either not capable or not willing to get involved in automation. I've worked on the other side of this equation for most of my career: I automate stuff. That's what software developers do. I also talk to management about how best to use the people whose jobs are affected. In these situations, the people who are most open to change, and willing to help effect it, are the most useful, and the least likely to lose their jobs.
Besides, someone capable of automating their job probably shouldn't be in that rote job in the first place, and what you call "getting fucked" is more like a hint from life that they should be doing something more economically valuable.
Luckily, someone like that has that more valuable work right in front of them. Being able to automate things is a skill that not everyone has, and that's very valuable to employers. That's the entire reason that good software developers get paid as much as they do. Actually getting demonstrable experience in automating job functions is much more valuable than taking a course and trying to switch into that kind of role without experience at it.
Helping with automation puts you in a strong position - you're proving that you can help improve the efficiency of the business. Automation produces new opportunities to be more productive - e.g. producing more, or handling more customers, without increasing staff to match. Work is not so much eliminated, as changed. Again, the people who lose in this process are the people who resist change. The people involved in automation are well-positioned to understand the change and spot new opportunities, and again, this is valuable to the business.
> Either way you have to try and find a different job, which isn't free, costless, or painless.
Having to find a new job is not a foregone conclusion by any means, for the kinds of reasons I've described, as well as because total automation of a job typically isn't achieved quickly.
But even if you do end up losing your job, you now have skills that are multiple times more valuable than what you were originally doing before automation.
In any case, you really shouldn't think of having to switch jobs as "getting fucked." The general advice these days is to switch jobs every few years, if you want to maximize your salary, because companies don't tend to give raises that match the industry rate, particularly in tech jobs.
If you're in a rote job that's amenable to automation, you should be looking for ways to improve that situation anyway. Even if your current job is safe for a while, if other jobs like yours are being automated, salaries for those jobs are going to start dropping. You'd be in a sinking ship and not doing anything about it.
I'm reminded of a quote from "Ghost in the Shell": "Your effort to remain what you are is what limits you."
However, our society rations resources, including resources that are needed for living, and including resources that are not actually scarce, on the basis of money, which can be returned based on capital or labor. Most people have no capital, so are forced to sell their labor. As work is automated, the return on capital increases, but the return on labor decreases (because more people are competing for a decreasing amount of paying labor).
Capital-holders wanting to preserve the existing system (if they are reasonably foresighted) are promoting the idea of a universal basic income. But the equitable long-term solution to this imbalance is Fully Automated Luxury (Gay Space) Communism.
In a system with property rights of the type found in capitalism, you need income from capital or labor to survive, basically. Unless the distribution of capital is levelled (and some force is exerted to keep it levelled), that means most people will rely on labor because they have no other choice.
Assuming that the labor economy is dying, then all laborers (truck drivers and code writers) need to figure out how to transition from generating livable income with their labor to generating livable income from their investments.
Difficult? Yes. But is it any more difficult than attempting to acquire sufficient resources to live using obsolete labor skills?
Because that means the existing capitalist class has to be forcefully deprived of much of their capital so that it can be redistributed, and a system needs to be set up to prevent it being from be reconcentrated.
The existing capitalist class, which holds disproportionate power, largely opposes this for obvious reasons.
I’m not saying this problem can’t be solved — I rather hope it can — but that’s why it is still an open problem.
I don't really see free money working, at least because people are selfish and won't agree to pay their money to someone else.
There is a lot of effort put towards solving starvation and other forms of poverty around the world, and staggering progress made on that front. But its not nearly as simple as sending food that was destined for the trash to another country is it? There's logistics and when crossing state boundaries, politics as well. If you try to merely send food -- or _any_ form of wealth to a less wealthy country, it won't always have the immediate effect of making people's lives better.
And this is exactly the reason why the comment was posted. I.e., the debate about how to help laborers transition to different types of labor may not be the correct debate because it only pushes out the problem to the next technological revolution, which will likely come even faster. Improving technology reduces the need for labor while at the same time it increases opportunities for investment. Hence, perhaps the debate should focus on transitioning from a labor based society to an investor based society.
Counter-point -- if the situation really _is_ such that the vast majority of the population enters into a situation where they require capital that only a minority have, then its inevitable that some form of wealth sharing would develop in any non-dictatorial society. In other words, I argue that such a transition could happen in a relatively peaceful, progressive way. _If_ it really needed to.