Ask HN: Is deep learning obsession in college ill founded? Background: I'm a CS junior (about to become a senior) and in our last year, people choose their capstone project that they work on for the entire year. For some years (say since 2017), deep learning projects completely dominate the other projects in term of number and the awards that they go on to win. I understand the principles behind it, even find it cool, but the whole inscrutable nature of it is problematic to me. |
But that isn't really a problem. In most cases, the projects you do as an undergrad don't affect your professional life in any way after you get your first job. Very, very few undergrad projects turn into real projects that anyone uses after the student graduates.
So don't worry too much about it. Ten years ago, every senior project was an app. Twenty years ago, every senior project was a website. It's just a sign of the times and doesn't matter in the long run.
Spot on.
On top of that we have 'AI' models getting fooled over adversarial attacks which just involve a single pixel change. As long as these issues are not tackled or not researched well enough, then we'll be pretty much be heading into another AI Winter and the hype cycle will go through its trough of disillusionment phase. Being unable to inspect the black-box nature of such deep-learning systems is why highly regulated industries involving danger to life such as healthcare or other safety critical industries label deep-learning solutions as unsafe for them.
Sure, all you see right now are other students and startups 'applying' deep learning everywhere, but they are hardly advancing the field unlike DeepMind and OpenAI are. In terms of learning, it's something good to learn as a student at college, but creating a AI startup now requires using Google, Amazon or Microsofts data center's for training which is clearly not sustainable anyway.
Security related projects and research are always where it's at.
It's not all or nothing as you present it. ML models can be useful even if they are imperfect - and we should not forget humans aren't perfect either. For example, a model could reduce 50% of the time necessary to enter an invoice into the database. It's imperfect, yet useful.
A model need not run alone without any safety. It can have plain old programming rules to validate its outputs, or use human in the loop.
> Sure, all you see right now are other students and startups 'applying' deep learning everywhere, but they are hardly advancing the field unlike DeepMind and OpenAI are.
On the contrary, I would say that what DeepMind and OpenAI are doing is largely irrelevant for industry. There is a huge number of domains where no ML model has been created, and that is because there are so few people who can make them. The low hanging fruit hasn't been picked yet. It's like electricity at the beginning of the 20th century. The work these students and startups are doing is the good, useful work. You don't need DeepMind grade models to solve most real problems.
> creating a AI startup now requires using Google, Amazon or Microsofts data center's for training
You can train most useful models on a single machine today. Some, like Logistic Regression, train in seconds or minutes. Others take an hour, or a day. Some heavy ones take a week to train. If you don't do hyper-parameter search or cutting edge research you only need a few runs to get a working model. It's data tagging that usually takes months or years.
To understand why neural networks work, you will have to understand how a whole host of smaller, simpler ML models work in excruciating detail. Multiple linear regression, logistic regression, etc. What they mean, how they work, what's really going on "inside", what the underlying probabilistic model represents, etc.
Neural networks are great because it takes basically all of those smaller ideas and concatenates them into a super flexible statistical machine. It's really cool to see the "in->out" but it's even cooler once you have a good grasp on what's going on in the intermediate steps.
In my experience, almost everyone working with neural networks don't have those details down. This goes 100-fold for non-research roles. They learned the Keras API and are happy stacking layers, and as long as the output looks nice they push to production. For most cases empirical validation is probably enough, because NNs usually can achieve some incremental improvement just by virtue of the fact they have so many damn degrees of freedom. But to get a well-performing, well-founded model, you need to know the ins and outs.
Imo deep learning is so popular because it "works". For a classification problem, if you try a linear baseline and a deep learning model, and you do a reasonable job of hyperparameter tuning and experimental design, it's likely you will outperform a simpler model. This holds true across many problem spaces.
I think the issue is that modern DL frameworks make it a little too easy to get pretty good performance on new problems. Other techniques generally require more background knowledge to make reasonable modeling assumptions, and still frequently perform worse than a naively applied DL approach.
I think DL will remain, in practice and education, a very popular tool. But it is essential to learn traditional statistical inference and other background to appropriately contextualize DL models so it isn't just some form of black magic.
It's easy to beat a naive logistic regression model with a good neural network, but the gap often closes once you start trying to tune the logistic model too. (And it's not like the neural networks aren't tuned either--architecture search, data augmentation, etc).
Recent review on medical data: https://www.sciencedirect.com/science/article/abs/pii/S08954...
So, pick something in "AI" that is the hotness of the moment, learn what you can, do your best, and then get on with life and career.
It's not ideal - but if it wasn't DL it would be another topic / application.
It's not completely inscrutable in why it works, follow the thread on deep learning theory research by starting from the names here: http://www.vision.jhu.edu/tutorials/CVPR17-Tutorial-Math-Dee...
Given all that you've said, a capstone that tries to dent the "inscrutable nature" of deep learning might be an interesting choice.
I will point out that the real win is with new data sources, and simple linear regressions may still work there.
Make a Rust front-end for the GNU Compiler Collection.
Emulate something.
Write a hypervisor.
The vast majority of us are users. We massage the data to be in a certain shape, then feed it through a machine that someone else created. We can change the parameters. We can change the data. But few of us are going to look in to the code of a random forest function.
I've switched tracks and started doing web development. Playing with the hyper parameters in machine learning is no different than changing the feel of a drop down by changing the colors, fonts and other things to fit a certain aesthetic.
I could be wrong, but I have yet to meet anyone that has done anything besides use packages created by others to call themselves data scientists. I think that opens it up to becoming just another tool no different than Excel.
I've been working in this space for a long time and recently started reading up on a particular ML technique which gained a lot of popularity over the past five years. What strikes me about 95% of the material available is how over-hyped and uninformative it is, to the point of just being wrong.
Either we really disagree about deep learning, or you vastly underestimate the influence of the other technologies that you've listed.
It's a problem whereever reliable operation is required, or analytic tractability (explanation) is required, or where resources available for data labeling are limited.
Its niche appears quite small, unless and until solid mathematical foundations are developed for it.
Where it becomes problematic - and where DL isn't actually very well-suited anyway - is making "real decisions"; things that would normally be backed by rigid logic.
Neural networks performance on a problem is a benchmark of its real difficulty. It gives us insight, a new perspective.
In millennia of deliberations what have philosophers have discovered about the nature of intelligence? And then .. a neural net beats us at all board games, another can solve differential equations, another can translate, another can see, and so on. Have we really not learned anything by these inventions?
Another advantage of DL is that it frames the problem of intelligence in mathematical concepts and rigorous evaluation.
I, for one, have reconsidered all my spiritual beliefs after learning about the agent-environment-reward model of reinforcement learning. A new way of framing the agent and life, so parsimonious and powerful. And it does not require a soul, or a god, or anything outside out real environment, and yet can explain so much.
The whole machine learning paradigm is another powerful concept through which we can understand how we might function. Previously you might wonder how emotion, thought, sensation, imagination and will relate to each other. Now we can understand how they might be implemented and wired together, and what principles support their function.
I would still argue no, we haven't learned anything about intelligence from these. They are impressive achievements, but strictly in the sense of "We found a way to use computers in a way we were not using them before".
1.) Neural nets + MCTS beat us at all board games--board games that humans invented and can achieve mastery at. If a human Go player was born who could beat that version of AlphaZero, we would not say that person has solved intelligence.
2.) Differential equations: also invented/discovered by humans, can also be solved by humans
3.) Translate and see: See above, with the additional caveat that humans actually are better at translating and seeing than deep learning systems are.
In addition, these were all achieved individually by systems with different architectures and massive amounts of training data that would amount to several human lifetimes. An 18 year-old human can play board games, drive a car, do differential equations, and learn multiple languages, with a single brain using a generalized structure and a fraction of the "training data" afforded to DL. This indicates to me that ML as a whole is still very far off the mark of General Intelligence.
> Previously you might wonder how emotion, thought, sensation, imagination and will relate to each other. Now we can understand how they might be implemented and wired together, and what principles support their function.
Previously? This is still an unanswered question. Show me where deep learning research has even come close to producing a system that can learn and adapt like a human mind does.
Not really, and that is the problem. We can create something that sort of works - but we don't understand it. The only thing we have learned is that we don't need to understand intelligence to build something that works on some tasks.
In many papers, the "baseline" logistic regression model is very stripped down: y~logit(.) but the neural network has had its expressiveness optimized in various ways. People aren't comparing against a 3 layer feedfoward network; there's augmentation and pre-training, architecture search and special learning schemes.
My claim is that if you want to claim that a problem needs the expressivity that (only) a neural network provides, you ought to be devoting a great deal of effort to the logistic regression model too. Make it a steelman, rather than a strawman, if you will.
As for deep learning specifically...meh.