Shoot, hope ya feel better!
> However, I am also skeptical that most humans are terrible at understanding stats so any information based on statistical analysis must be dumbed down to ELI5 level so most report consumers understand.
For most regular people, I think stats is kinda like code: As soon as they see it, their eyes glaze over and they stop paying attention. I don't think they should even see the underlying math, just the abstracted conclusions. For real-time data, that might be a simple traffic light (green/yellow/red with colorblind-friendly symbology) showing when something is within expected ranges, turns yellow when something is 1 standard deviation away, red when it's 2+, blinking and screaming when it's 3+, etc. The underlying stats have to be carefully analyzed based on the actual use case with proper domain knowledge, but the UI can simplify that down to "everything is ok" vs "keep an eye on this, it's starting to look weird" vs "wake everyone up in the middle of the night to deal with this".
For time-series, maybe it's a colored background range showing the expected the highs/lows for those few months, under the actual data line that plots the real data. If the line is within the range it's fine, if it's far outside the expected for that time period, something might be off.
For deltas, sparklines or font bolding/coloring/sizing can give a visual indication of the magnitude of change, either relative to the previous time period (this metric is 2000% higher than yesterday) and/or relative to other metrics in the dataset (everything else changed +/- 10%, but this one was +50%).
The downside to this approach is that it requires actual domain knowledge, an understanding of each metric and their applicability to the business, a customization for each report viewer based on what their job can do (i.e. what actions can they actually take in response to which metrics), and a lot of filtering, analysis, testing, and further iterations. It's a far cry from a 1-click auto-generated dashboard based on some standard dataset (like web analytics).
In one of my jobs, we used https://lookerstudio.google.com/ quite a lot because it also allowed the report viewers to edit the dashboards on their own (for simpler things, or to change pagination and layout, etc.). But we'd have pipelines in the middle that would ingest raw data and produce statistics for the dashboard. But we approached it like any UX problem: not metrics-first, but user-first. We talked to them about why they wanted a dashboard, how they consume the data, how they triage the metrics, how they respond according to the metrics, how they like to be notified or not, etc. It's a very personalized approach that tries to mimic what a good assistant would do, e.g. "here's today's must-know summaries for you" vs the sysadmin approach ("here's how every CPU core and process is doing").
Fundamentally, it was about finding the very few signals among all the noise, showing as little as possible at a time upfront, but allowing drill-downs where needed. 90% of the time they wouldn't drill down – which to me was a good sign that we were able to customize the dashboard to their everyday needs.
-----
I should note that this is purely anecdotal, based on my experience as a frontend dev who also had to make several dashboards, not a data specialist/data scientist.