I think the ASIC we discussed yesterday[1] from Phantafield is a hint of what is to come. Instead of constantly moving weights from HBM to MACs, they have local dynamic ram areas that hold the weights locally, using a novel architecture that is more efficient at using space than the traditional SRAM cell, especially in a mostly-read environment.

Instead of doing MAC using 16 bit wide hardware, they do it all bit serial, so that they can fit far more computing into the same chip area, trading a bit of the speed they get from all that local RAM into efficiencies in usage. It also makes it easy to do FP8 support, or any other bit size required.

I expect another few orders of magnitude improvement in the cost of compute over the next decade. Moore's law isn't dead yet.

[1] https://news.ycombinator.com/item?id=48713686

not so much ASICs but there are some very promising advances in photonic systems and NDRAM. Stay tuned.

I believe so