I would guess that this means DawnBench is basically working. You'll get some "overfit" training schedule optimizations, but hopefully amongst those you'll end up with some improvements you can take to other models.

We also seem to be moving more towards a world where big problem-specific models are shared (BERT, GPT), so that the base time to train doesn't matter much unless you're doing model architecture research. For most end-use cases in language and perception, you'll end up picking up a 99%-trained model, and fine tuning on your particular version of the problem.

> Comparing it to Moore's Law doesn't make any sense to me, though.

I assume it's meant as a qualitative comparison rather than a meaningful quantitative one. Sort of a (sub-)cultural touchstone to illustrate a point about which phase of development we're in.

With CPUs, during the phase of consistent year after year exponential growth, there were ripple effects on software. For example, for a while it was cost-prohibitive to run HTTPS for everything, then CPUs got faster and it wasn't anymore. So during that phase, you expected all kinds of things to keep changing.

If deep learning is in a similar phase, then whatever the numbers are, we can expect other things to keep changing as a result.

Like many things, Moore’s law is garbled when adopted by analogy outside its domain.

What does “more transistors” mean? To you, it means just what Gordon Moore means when he said it: opportunity for more function in same space/cost.

The laypersons, marketing grabbed the term and said it would imply “faster”. Which then was absurdly conflated with CPU clock speed (itself an important input, though hardly the only one, determining the actual speed of A system).

The use here is of the “garbled analogy” sort which surely is the dominant use today.

Agreed, but Moore's Law has morphed to refer to both xtors and performance despite his original phrasing.

The biggest innovation I've seen is in the cloud: backplane I/O and memory is essential and up until a few years ago there weren't many cloud configurations suitable for massive amount of I/O.

Ok, but achieving Moore's law has required combining an enormous number of conceptually distinct technical insights. Both training costs and transistor density seem like well-defined single parameters that incorporate many small complicated effects.

Implicit in moore's law is that cost does not increase in the same way, if not prices of chips would also be doubling. Something more analogously sounding would be the decrease of cost per transistor.

The number of transistors is also not dependent on a single thing, it can be argued many macro events contributed since the 80s, the VC model for chipmakers in SV, the rise of the internet, going fabless, rise of mobile, innovations in fabrication technology.

One thing I've wondered is if Moore's Law is good or bad, in the sense of how fast should we have been able to improve IC technology. Was progress limited by business decisions or is this as fast as improvements could take place?

A thought experiment: suppose we meet aliens who are remarkably similar to ourselves and have an IC industry. Would they be impressed by our Moore's law progress, or wonder why we took so long?

Last I saw, they don’t even support ROCm on their recent Navi cards, so I’d be hesitant.

Indeed nonexistent

> abnormality detection (for instance: in medicine), agriculture (lots of movement there right now), parts inspection, assembly inspection, sorting and so on

none of these is anything someone can run from their bedroom because they have very high quality and regulatory requirements and require constant work outside of the actual AI training.

This is actually reflected in the margins of "AI" companies, which are significantly lower than traditional SAAS businesses and require significantly more manpower to deal with the long tailed problems, which is where the AI fails but it's what actually matters.

anything that's even remotely profitable is already taken

This is a space I worked on during the crypto boom of 2017/2018.

The opportunity is present for a decentralized network that allows for training of models to be done from training sets at facilities.

Think of all the data sitting in silos from clinical trials. There is of course the painful process of authenticating researchers for access to data like that but it can be done. There just needs to be an economic reason to make that kind of effort.

I got pulled into a direction of using ML to predict costs of care in insurance so didn’t go further down the rabbit hole but I did author a patent for a novel approach to have a decentralized identity exchange data.

If any of this sounds exciting to you feel free to email me. hn (at) strapr (dot) com

You are missing TPU and spot/preemptible pricing, which need to be considered when we are talking about training cost. The big one to me is the ability to consistently train on V100s with spot pricing, which was not possible a couple of years ago (there wasn't enough spare capacity). Also, the improvement in cloud bandwidth for DL-type instances has helped distributed training a lot.

Nothing really has changed in the last two years in terms of training cost. I think the author is making unreasonable extrapolations based on changes in performance on the Dawn benchmarks. A lot of the results are fast but require a lot more compute / search time to find the best parameters and training regimen that lead to those fast convergence times. (Learning rate schedule, batch size, image size schedules, etc.) The point being that once the juice is squeezed out you aren’t going to continue to see training convergence time improvements on the same hardware.

Also, because you cited our GPU benchmarks, I also wanted to throw in a mention our GPU instances which have some of the lowest training costs on the Stanford Dawn Benchmarks discussed in the article.

https://lambdalabs.com/service/gpu-cloud

Another data point:

"For example, we recently internally benchmarked an Inferentia instance (inf1.2xlarge) against a GPU instance with an almost identical spot price (g4dn.xlarge) and found that, when serving the same ResNet50 model on Cortex, the Inferentia instance offered a more than 4x speedup."

https://towardsdatascience.com/why-every-company-will-have-m...

I don't really know if those hardware breakthroughs that the article refers to already reflects in Cloud GPU performance, but software reflects nonetheless. So even though pricing has fluctuated marginally since 2018, it is just plain faster to train a neural network today because of software advances, from what I understood.