Ask HN: Heavy computation setup for Python?

2 points by gavribirnbaum 4 years ago | 6 comments

I'm running some DFT calculations at home, and they take way too long to run on my computer, I was looking for a pleasant and cheap way to develop these models while also speeding up the computation speed. Open to getting servers, etc, but would love some direction. Thanks!

indoorskier 3 years ago |

I am obviously biased as one of the maintainers, but this is exactly the kind of thing we built Meadowrun for!

See https://meadowrun.io

It takes your local code and environment (we support pip, poetry and conda) and runs python code on AWS or Azure (though Azure might have some rough edges still). You can specify what kind of machine you want based on cpu/memory etc, no need to deal with weird instance names. You can get cheap spot instances etc.

fastneutron 4 years ago |

I support some users running DFT, and I have a few questions: What’s your current setup? How big are the models? Which DFT code are you running; VASP, QE, something else? How long are your runs currently taking vs. how long do you want them to be?

(Edited for an autocorrect hiccup)

gavribirnbaum 4 years ago | |

I'm currently running things locally on my MacBook Pro. The models aren't too big, just a few atoms for now as I'm still testing things. But the optimization codes runs for a couple of hours, and I wish it went down to less than a half hour. I am running GPAW.

fastneutron 4 years ago | | |

Ok, so each instance takes a a "reasonable" amount of time and the whole structure optimization is taking hours? If you can get a setup with MPI and/or CUDA you may be able to cut the instance runtimes, but if the optimization is just slow to converge it won't help much. If it's the former then a higher end workstation or a small cluster could get you there. Cloud could be an option here too if you can aggregate the compute resources.

I'm not as familiar with GPAW but if you're able to share a minimal example I could try to take a look.

mattbillenstein 4 years ago |

Have you tried spot instances on AWS? Fire up a box for a few hours, then spin it down when you aren't using it?