Hey, I've been designing a framework to improve the training of deep learning models on expensive numerical simulations. Instead of relying on limited, pre-generated dataset, the approach streams data directly from large-scale numerical simulations running simultaneously with training. Because it avoids I/O bottlenecks it makes training faster and cheaper.

It is implemented as an asynchronous iterable dataset using ZMQ for data communication. I would love your thoughts on how to extend this to a full-fledged event-driven application.

Although the framework target cluster environments, the link contains code you can try locally. Check it out and let me know what you think!