What Extropic is building(extropic.ai) |
What Extropic is building(extropic.ai) |
My prediction is that they will raise a nine-figure sum over the next decade, and never release a product that comes close to the performance of an NVIDIA card today.
* The stochastic/random nature of processors is already used in cryptography for physically uncloneable functions. Dunno if this has any practical uses in industry, and it is crypto, so it is probably also BS, but it is the same phenomena you get if you log in into your BIOS and turn off ECC of your RAM.
* The very first computer capable of MCMC was designed by von Neumann himself and used uranium as a source of randomness as part of the Manhattan project.
Anyway semiconductors have never been my strong suit, but I guess this is more of a IP play then a consumer product business. Now let me get back to writing unit tests.
>Extropic is also building semiconductor devices that operate at room temperature to extend our reach to a larger market.
funny stuff
Key Points
The demand for computing power in AI is increasing exponentially, but Moore's Law is slowing down due to fundamental physical limitations of transistors at the atomic scale.
Biology hosts more efficient computing circuitry than current human-made devices by leveraging intrinsic randomness in chemical reaction networks.
Energy-Based Models (EBMs) are a potential solution, as they are optimal for modeling probability distributions and require minimal data. However, sampling from EBMs is difficult on digital hardware.
Extropic is implementing EBMs directly as parameterized stochastic analog circuits, which can achieve orders of magnitude improvement in runtime and energy efficiency compared to digital computers.
Extropic's first processors are nano-fabricated from aluminum and run at low temperatures where they are superconducting, using Josephson junctions for nonlinearity.
Extropic is also developing semiconductor devices that operate at room temperature, sacrificing some energy efficiency for scalability and accessibility.
A software layer is being built to compile abstract specifications of EBMs to the relevant hardware control language, enabling Extropic accelerators to run large programs.
---
Is this real or just theoretical?
They’re not building “a better rng”- they’re building a way to bake probabilistic models into hardware and then run inference on them using random fluctuations. Theoretically this means much faster inference for things like PGMs.
See here for similar things: https://arxiv.org/abs/2108.09836
There’s a company called Normal Computing that did something similar: https://blog.normalcomputing.ai/posts/2023-11-09-thermodynam...
I do think that a "better rng" can be interesting and useful in and of itself.
Thanks for the Normal Computing post, it felt more substantial.
We experimented with doing ML training with it, but it's not clear that it trains any better than a non-broken PRNG. It might be fun to feed the output into stable diffusion and see how cool the pictures are, though.
I’m unfortunately not familiar enough with hardware to weigh in.
What you want is low precision with stochastic rounding. Graphcore's IPUs have that and it's a really great feature. It lets you use really low precision number formats but effectively "dithers" the error. Same thing as dithering images or noise shaping audio.
Is there any evidence that such a probabilistic model can run better than a state of the art model?
Or alternatively what would it take to convert an existing model (let's say, an easy one like llama2-7b) into an extropic model?
No, but they got 15M seed funding anyway.
"We're taking a new approach to building chips for AI because transistors can't get any smaller."
I really don't know what they gain by convoluting the point and it's pretty hard to follow what the CEO is talking about half the time.
Deep learning doesn't seem to need that much numerical precision. People started with 32-bit floats, then 16-bit floats, now sometimes 8-bit floats, and recently there are people talking up 2-bit trinary. The number of levels needed may not be too much for analog. If you have a regenerator once in a while to slot values back to the allowed discrete levels, you can clean up the noise. That's an analog to digital to analog conversion, of course.
That's not what these guys are talking about, as far as I can tell.
“Create a website for a new company that is building the next generation of computing hardware to power AI software. Make sure it sounds science-y but don’t be too specific.”
Pragmatically, it doesn't make much sense given that it would take years for this approach to have any real work use cases in a best case scenario. It seems way more likey that efficiency gains in digital chips will happen first making these chips less economically valuable.
So much so I wonder what the hell they're doing with this company. Is he a prolific poster and an engineering genius? Or is he just another poster
Never would have guessed the guy was an actual physicist
This whole pitch sounds like the usual quantum computing babble.
Also, the fact that they're using ultra-cold superconductors makes me wonder how much noise helps and how much it hurts. If your system is all about leveraging noise well, but you can only use super special well-behaved noise, then "bad noise" could easily ruin the quality of your generated solutions.
It's cool to see something so wacky out there, though!
[1] https://en.wikipedia.org/wiki/Effective_accelerationism
[2] https://twitter.com/BasedBeffJezos
[3] https://knowyourmeme.com/memes/cultures/eacc-effective-accel...
Like, it's hard to take someone seriously when they spend tons of time shitposting on Twitter, it's even harder when it's revealed that they're behind one of the most popular shitposting accounts within a niche, almost cult-like community.
But at least it is not the 5000th so-called AI-powered SaaS company that is using OpenAI API that has raised $20M+ to VCs and burning hundreds of thousands every month with little to no plan to generate revenue.
Will be watching this one closely, but highly skeptical of this company.
At best they advance the field massively, at worst the backers lose their money but the tech/knowledge finds a home elsewhere and the knowledge in the field is nudged forward.
- It is written in a way that sacrifices legibility for supposed precision but because the terms used can't really be applied precisely, it's equivalent to spurious digits in a scientific calculation. The usual reason this occurs is to obfuscate or to overawe the audience.
- It is hard to overstate the difficulty of beating semiconductor with a wholly new branch of technology. They're so insanely good. People have been trying to beat them for decades and there's not even a solid theoretical thesis as to how to do so. Even the theoretical advantage of quantum computing is predicated on error correction being scalable which is a totally open question even theoretically.
Very bad vibes. Hire someone who can communicate, and demonstrate what you're building.
Can anyone name a company which used such absurd language to describe themselves and then actually delivered something valuable? There must be one.
And also, are you the real Eliezer?
"Extropy" is a term that was previously coined by a group of fairly nice people to describe themselves, and so far as I know is being stolen here without permission.
https://knowm.org/thermodynamic-computing/
It's a random, unassuming 7-year-old blog post from a DARPA-funded and defense-involved inventor. They happen to work in neuromorphic computing. Their other posts talk about some of that work. A cynical take is that it can seem like just hand-wavey garbage, but then again, it's been quietly getting tons of defense contractor money.
I came across it years ago, and it has greatly accelerated my worldview, and has made me feel ahead of the curve in understanding what is going on in the universe. It's informed my community organizing. It's informed how I understand AI and consciousness and language, and the intersection of all these things.
I'm inclined to believe that the people in this area are clued into something very substantial about how the universe works.
EDIT: oops, shared the wrong link. This one is about thermodynamic evolution
Quantum computing has been exploring an entirely new model of computation for which it's hard to even articulate the problems it can solve. Whereas using analog computers in place of digital is already well defined.
> Extropic is also building semiconductor devices that operate at room temperature to extend our reach to a larger market. These devices trade the Josephson junction for the transistor. Doing so sacrifices some energy efficiency compared to superconducting devices. In exchange, it allows one to build them using standard manufacturing processes and supply chains, unlocking massive scale.
So, their mass-market device is going to be based on transistors.
The actual article read like a weird mesh of techno-babble and startup-evangelism to me. I can't judge if what they are suggesting is vaporware or hyperbole. This is one of those cases where they are either way ahead of my own thinking or they are trying to bamboozle me with jargon.
I personally find it hard to categorize a lot of AI hype into "worth actually looking into" vs. "total waste of time". The best I can do in this case is suspend my judgement and if they come up again with something more substantive than a rambling post then I can always readjust.
Am I the only one who thought the article was clear, lucid, and reasonably concise?
The company's success or failure will depend on execution, but the value proposition is quite sound. Maybe I've just spent too much time in the intersection between information theory, thermodynamics, and signal processing...
"Don't splurge on high SNR ('digital') hardware just to re-introduce noise later." == "Don't dig a hole and fill it in again. You waste energy twice!"
In most applications superconductivity does not actually yield better energy efficiency at system level, since it turns out cooling stuff to negative several hundred degrees is quite energy demanding.
It even makes me think that they don't understand what they're talking about which is why they're using complicated terminology to mask it but I'm hopeful I'm wrong and this is an engineering innovation that benefits everyone.
Since they're building a special-purpose accelerator for a certain class of models, what I'd like to see is some evidence that those models can achieve competitive performance (once the hardware is mature). Namely, simulate these models on conventional hardware to determine how effective they are, then estimate what the cost would be to run the same model on Extropic's future hardware.
This interview makes me much more excited and less skeptic than Verdon's usual mumbo-jumbo jargon. He should try using simpler, and more humble language more often.
edit: btw the bottleneck in AI algos is matrix multiply and memory bandwith.
So, their solution is to embrace the stochastic operation of smaller chip geometries where transistors become unreliable, and double down on it by running the chips at low power where the stochasticity is even worse. They are using an analog chip design/architecture of some sort (presumably some sort of matmul equivalent?) and using a "full-stack" design whereby they have custom software to run neural nets on their chips, taking advantage of the fact the neural nets can tolerate, and utilize, randomness.
https://m.youtube.com/watch?v=8fEEbKJoNbU&pp=ygUVbGV4IGZyaWR...
If it is a fraud, how do people like this get funded?? (And how can I be creepier so that my real ideas get funded)
* competent and curious engineers
* entrepreneurs, who live on a continuum where one end is...
* ...hucksters and snake-oil purveyors, of which there are plenty, and
* (because this is the Internet) conspiracy theorists and other such loons
and recently
* political provocateurs
You can make a thread work (for that group of people) if it self-selects who reads it. Unfortunately, AI is catnip to all five of these groups, so the average thread quality is exceptionally low – it serves all five groups badly.
Whether some of these people _should_ be served well is a separate question.
now, it is hard to tell who put effort in at all. read or write.
would you consider your own response to be optimistic or high effort?
Right. A better word is confabulation.
I.e. pseudomemories, a replacement of a gap in information with false information that is not recognized as such.
> We are very excited to finally share more about what Extropic is building: a full-stack hardware platform to harness matter's natural fluctuations as a computational resource for Generative AI.
This is New Age, dressed up with the latest fashion.
AFAIK there are other efforts to develop analog neural network ASICs. Since neural networks are noise-tolerant this could work and could allow faster computations than conventional must-be-perfect digital circuits. IBM, Intel, and others have experimented with this.
I wouldn't believe there's anything particularly novel here unless a lot more detail or test hardware is given.
I'm not 100% sure this is true but I've heard that this fellow was involved with the NFT craze and made money there, and that sets off alarm bells. I've suspected for a while that e/acc is a marketing thing since it's just repackaging old extropian stuff from the 1990s.
"I want to believe" but have seen enough to be skeptical of extreme claims without hard evidence.
As far as the feasibility and impact on AI in general, I have no idea.
Someone should tell them about MCMC and alike.
Or if they want to accelerate MCMC for a particular problem, they can build a classical ASIC and scale it.
It might fail for the reasons many startups fail, but it's not prima facie fantasy.
Then again, I should probably also ask what you mean by "analog computing", and why you think quantum error correction would not allow qc to be classified as analog.
For context, I did research on crafting a high-fidelity (error-correcting) quantum gate (successfully).
Big challenge would be whether any gains could compete with the economy of scale from NVidia.
Seems like it was quite a time to be online. I mostly know of it through this version of events. Not sure how accurate you'd find it:
https://aiascendant.substack.com/p/extropias-children-chapte...
The word "extropy" itself seems to go back several decades before the mailing list, if I'm reading correctly here: https://en.wikipedia.org/wiki/Extropianism. Still, I wouldn't be surprised if many/most of the original mailing list members found this usage a corruption.
binoct's comment is not "pointing out BS" - that requires that you, y'know, actually point out things. binoct's comment is sneering, without any content or thought. Pure fluff.
This site doesn’t get everything right. It tends to miss things that succeed in the consumer space because this is a pro audience not a mainstream audience. But it usually gets hard science right.
They are very energy efficient (measured in pJ/bit), but non-cryptographic PRNGs, which are typical for ML, are far more efficient.
It's not obviously wrong to think that AI algorithms will pick up bias from "overfitting" to their PRNGs used during training, but I'm not expecting the benefits to be very large.
However, if you're thinking about research grants, no. That's the point of research grants.
Today, it's a very different situation. Now we have Elon Musk, who is really good at what he does, and also tweets a lot...
There may or may not be something there, but the article is mostly buzzword-slinging. They wrote "This will allow us to put an Extropic accelerator in every home, enabling everyone to partake in the thermodynamic AI acceleration." Huh?
If they said something like "We are trying to cut the cost of stable diffusion by a factor of 100", that would sort of make sense. But then people would want to see a demo.
Still, you are correct that Extropic is looking at the computation rather than data. But, I wanted to chime in so as the discussion here would not leave the impression that we are still in the days of pure unsupervised scaling.
However, the idea of using analog matrix multiply is reasonable, and has already been done by at least one company:
(before that you were given the chance to object to the estimate, but not to change it.)
It's a prohibitively expensive way to go, and depending on how you built the quantum computer, it may be more susceptible to interference and non-quantum noise than using good circuits and custom systems.
