Given the impact the model might have on the world, and that $600k is roughly equal to the annual salary of 1~3 top notch engineers... it's probably intended to mean relatively cheap.

For similar context, millions of VC money have been thrown around for much less impressive results.

#firstworldproblems

Spot instance pricing for p4d.large is about a third the on-demand cost, though it's not clear what they're using to calculate market rate. An A100 card is roughly $10k retail, so it's not like "owning a GPU cloud" is that cheap a proposition either.

Probably not for a few years, you need a (maybe few) A100(s) to be able to backprop a model that big with float32.