Embedded engineer here, there are a lot of concepts of "thread priorities" that are great to steal from RTOS, and connectivity features that are made to ensure you never ever break the minimal "software" that is required to at least make a reboot and change the disfunctional code. General guidelines : 1. make everything have priorites, so you can always keep alive the most important features, 2. for interfaces, make specific rules, every communication message that breaks a rule is rejected at emission time, 3. put as much watchdogs as you need, and remove them little by little as you test out for reliability