Build yourself a weather station(blog.kdubovikov.ml) |
Build yourself a weather station(blog.kdubovikov.ml) |
https://github.com/FlorinAndrei/WeatherStation
The Arduino Nano 33 BLE Sense has all the sensors I need. A bit of Python on a Pi Zero performs data collection and a bit of pre-processing.
I've been wanting to build a fully autonomous (solar panel + cellular network) for a long time...
The licensing requirement isn't steep in the US; if memory serves its 35 questions, all multiple choice, and the question pool is published. Readers of Hacker News should be able to pass the technician exam with about a week of light study and that gives you full privileges from 30 MHz and up (APRS runs between 144 and 146 MHz depending on which country you're in).
Moving to SQLite to ease analyse.
Other variables can be significantly in error due to the siting of the station: under a tree, not aspirated, in the wake of a building, on a rooftop, etc.
I would think there is still value there, at a sufficient scale.
Navigation inside the app doesn't feel intuitive at all. Maybe it's more similar to default experience on android? No idea. Map/Hourly/Daily buttons don't work as tabs. Daily view arrives with scrolling up animation, but when you scroll to close it there is a large ad banner (well, understandable, but awkward) and for some reason weather news (why would anyone want it there or at all?). I've noticed "close" button later, but it was in bottom right corner for some reason.
Plastering "RealFeel(r)" everywhere was horrible idea. It is corny, unnecessary and reads worse than "feels like".
Then I got my answer: they focused on the temperature forecast, while since I bike to work what I was interested in was "is it gonna rain today?"
I get that it's difficult to predict when and where rain might occur, but I'd much prefer a rough probability scale to nothing. On the bright side, they have a real-time, radar-based 90 minute rain forecast now, so at least if I check it in the afternoon I can jump on my bike and get home before I get wet.
As an example, they may forecast that 30% of the areas your city will report rain tomorrow. They may have an high level of confidence in this prediction (and verify it to be correct after the fact). It may still not be very useful to you, since the one outcome is so different to the other.
That said, I agree with your that the predictions should be presented in a more scientific manner. In fact, one irritating pattern is that rain probability and the predicted amount of rain sometimes get combined into a single percentage, which no longer represents neither the probability, nor the amount. It's some kind of weird combination of the two.
On RPi side I only had to get an RTL-SDR receiver and run https://github.com/merbanan/rtl_433 on the RPi itself to decode the data.
It was significantly cheaper than going with BLE or WiFi sensors.
Can you provide some links to temperature sensors you've tried that work well with an RTL-SDR receiver? I'm also interested in setting something up with a Raspberry Pi.
Just last Thursday they reported zero precipitation for the entire day, not a hint of rain. Yet we got 3 hours of medium-heavy showers in the afternoon... not a very uncommon combination. If I had known there was a 30% chance of showers, I would have packed my rain jacket.
[0] http://www.realclimate.org/index.php/archives/2020/06/sensit...
Rain is not easy predict, especially long in advance. Just taking things with a grain of salt here. If you truly believe that forecasting actual rain is not very hard, please start a new weather forecasting company and give it all you got! However i'll tell you, it's not that easy. Just because a computer model 'runs' does not mean it correctly modeled the atmosphere. All models are imperfect bud and those imperfections get amplified by 10x when trying to simulate earth+atmosphere systems
https://www.aliexpress.com/item/32810015137.html
(Hopefully the link still works.)
The sensors are kinda bulky, but still more reliable than anything I managed to build by myself :)
Jim Easterbrook (ex-BBC engineer) has a really good write up on the process, and also shares his python code:
http://www.jim-easterbrook.me.uk/weather/
I don't know Jim, but I have done the same following his instructions.
I unfortunately do not have a good site for a weather station, so I just use the NYS Mesonet: http://www.nysmesonet.org/
(They charge $450,000 a year for commercial customers but still don't have a TLS certificate. The weather stations are good, though.)
You can see some of the data here [4]
You can hook this thing up to the internet over wifi and it will send data to https://www.ecowitt.net https://www.wunderground.com https://weathercloud.net/ and/or post it to your custom endpoint. The gateway can also be queried for data over an API. This thing is quite new on the market but I am very happy with it. You can hook it up to already supported websites or you can use a raspberry pi to collect data yourself (WeeWx and Cumulus MX already support the API).
[1] http://www.ecowitt.com/weather_station/5.html
[2] http://www.ecowitt.com/wifi_weather/80.html
[3] https://www.froggit.de/product_info.php?language=en&info=p44...
[1] https://www.banggood.com/USB2_0-FM-DAB-DVB-T-RTL2832U-R820T2... [2] https://github.com/merbanan/rtl_433
https://partofthething.com/thoughts/weather-and-air-quality-...
Edit: http://www.weewx.com/
Unfortunately, I think this design also means it has a low ingress protection value. The unit I ended up with had the humidity sensor replaced and it still wasn't providing numbers. I think it may have spent a year or so out at sea on a buoy.
I plan to have multiple ESP32 nodes with sensors (luminosity, temperature, pressure, moisture, presence detector), ble scanning (in order to detect who is in the room), and microphones for voice control (picovice porcupine library for stop word detection, then google speech 2 text). For the microphones, I'm still waiting for my mems microphnes to directly include them on the sensor node, or I will use an additional board like the esp32-korvo which works very good (already tested). For audio playback in the rooms, I'm planning on using the up2stream amp devices with ceiling speakers. As for powering it, I'm still unsure on what to use. I first ordered some olimex POE esp32 isolated (so you can power them from both POE and USB at the same time). However the ethernet interface will go up/down in an endless loop after a few minutes of sending data over mqtt. Strangely, the non ISO version works though as it seems, I haven't run it though in a box (higher temperature). So I'm still unsure if I use POE or a 220V->5V converter in each case, or feed them directly with 5V from a central place (which would require more cable length planning)
I did some programming at first, but then quickly switched top https://esphome.io/ which allows you to include your sensors without writing any code. It has support for all kinds of sensors, integrated API, mqtt, webserver, etc... and you only have to configure one yaml file which will then automatically generate the C code and compile/upload your firmware. No need to reinvent the wheel!
It's also easy to include your own components in case you want to do anything custom. The only disadvantage is that it's still based on esp32 idf 3 (in case you are adding components, I would prefer to do it for the new version and not the old one), but the v4 was only released a few weeks ago.
Once you get a 1-wire master, you can chain devices on a single cable (e.g. the ubiquitous DS18B20 temperature sensor). Digikey, Adafruit and Sparkfun have plenty of breakout boards and it's pretty easy to find code either bare metal (bit banging) or linux I2C
There's a pretty good intro book by Tim Bitson: https://www.biblio.com/weather-toys-by-bitson-tim/work/56888...
And some interesting projects such as this 1-wire rain gauge: http://www.midondesign.com/Documents/1wire_raingauge.PDF
https://www.element14.com/community/groups/internet-of-thing...
So yeah, I have something similar myself, with a React frontend on the official Raspberry Pi display, a BME680 (never got the IAQ stuff to work though) and InfluxDB + Grafana. No remote/outdoor modules yes, because I'm undecided about the technology.
Screenshot: https://www.instagram.com/p/CBc5uWLDOgc/?utm_source=ig_web_c...
Considering the data they output must conform to aircraft avionics parsers that are sometimes decades old, they are unlikely to ever change or go away. Since the weather observations are required for IFR flight, the FAA is quick to repair them when they break, too.
Whatever you can build with hobbyist hardware it won't compare to the airport's equipment.
I use a few of them with a NUC server and some custom nodejs scripts.
There are lots of DBs that you can use, including postgres. the flexibility and configurability that you get is outstanding. With the alerts subsystem you can easily attach triggers for bots.
Also having really simple and configurable graphs that look good.
I resorted to adding a DS18B20 for temperature readings and only using the air pressure and humidity of the BME280. I could design a PCB that moves the BME280 further away from the D1 Mini with tiny traces to avoid heat transfer, but then again I didn't want to risk it and a quick research showed that the DS18B20 seems to be reasonably accurate as well.
When I brew my beer, I need to look at the environmental temperature. I can't stick the thermometer in the beer because it'll be contaminated. So I put a thermometer in a glass of water which I place next to the beer vat.
I'd expect you'd have to put that BME280 in a small metal oven, or submerge in a volume of oil or water. It'll screw up the barometer probably.
Incidentally, I tried doing searches to see if those images are just copied from somewhere else, and ended up on what looks like a news site with what looks like content of the article after multiple passes through google translate [2]. Anyone has an idea why would something like this exist? Somebody playing with AI/ML perhaps?
[2] https://newshimalaya.com/2020/06/15/earn-your-self-a-weather...
When I come home, my phone connnect to the router and opens the gate and my rpi zero tells me, "Welcome home." It's been working for a year now every day :D is a simple python script with openwrt script on router.
I wrote it on my blog http://sebastian.korotkiewicz.eu/techlog/homeautomation-dete...
it very simple to do!
IMO there is even more value if this is built with a central server in mind for each weather station constructed. A network of stations working together can begin to create their own hyperlocal weather network and could even be useful in small-scale, short-term predictions.
There are some startups that do a bigger version of this and for a long time I've been wanting to build a physical weather station network.
Super cool work.
I use barometers in phones for this kind of purpose. I think that a minimum of 1 quality reading per 4 sq km would represent sufficiently dense coverage to create an improvement in forecasts. But more is better, since quality can be iffy.
Some papers published by Cliff Mass et al from UW might have more info on density work. I'll try to find a link. [1] [2]
The work I do is presently just in the US on Android: https://play.google.com/store/apps/details?id=com.allclearwe... with future expansion planned to allow better data acquisition. Open source code I wrote for Android for this purpose (periodically measuring pressure) https://github.com/JacobSheehy/AllClearSensorLibrary
[1] https://journals.ametsoc.org/waf/article/33/5/1375/40806/Imp...
[2] https://journals.ametsoc.org/jtech/article/35/3/523/107377/S...
If you collect your own weather data at scale, you can write assimilation routines to get WRF to incorporate your data into the model runs along with NOAA/MADIS/etc-provided data.
I know that's more complicated than you are suggesting but it's worth mentioning.
There are a lot of simple ways to do home weather forecasting. You can write your own Dark Sky clone by downloading/scraping/APIing your local radar station and then write your own wind predictor a few hours into the future based on rain movement or simple statistics from recent local wind data.
I would suggest looking into simple predictions. One of the first ones you learn in school is just basically just repeat what happened today. So if today was sunny, tomorrow sunny. Stupid, simple but decently effective. Except during transition periods between air-masses (when weather is changing).
For showing the weather forecast I have this by my door. https://fluffyelephant.com/2015/12/reuse-my-ebook-reader/
A quick glance in the morning and I know if I should bring an umbrella or not.
Fully OSS, nicely integrated into Homematic (home-automation system, mostly used in Germany/Europe tho)...
Now I will say something negative. It's a nit, but I think it needs to be said.
I've seen a lot of these "In which I write a (perhaps series of) essays on building X using all kinds of cool tech you'd probably like to play with when you could get the same result with less than ten bucks, maybe an hour or two, and just a handful of code, if any"
I'm okay with and completely love "Let's do something cool and learn!" I'm not okay with "I have this problem, so let me by implication teach you how you should architect problems when you run across them" We need to clearly differentiate. You should not, and you should never, start with an important project and begin speculating on what cool tech you could play with.
Playing and learning are great. Yay! Adding structure and complexity where none is needed is an antipattern if you're doing anything of value for yourself or somebody else. This stack might end up being necessary. Absolutely. Make the problem "prove" that it's necessary. You should have a clear and unambiguous process for choosing a stack; do it by necessity. Never even accidentally teach that you do it by "hey, that's cool!" People end up doing that, and somebody ends up having to clean it up.
tl;dr Love fun projects like this. Keep 'em coming. Just please be sure you're clear to the reader that they're learning/fun projects.
* Perfectly fine to add a tech or two to a new project in order to learn-as-you-work, it should just be done with the upmost care and a close eye to how much team bandwidth it's using versus the value being provided and how you'd back it out if you had to.
http://madscientistlabs.blogspot.com/2012/03/first-you-get-s...
OP i hate to break it to you. but a weather station wont tell you the right clothes to wear for the day. IT will tell you if it's raining outside currently. Consider this, station says it's raining and cold. You wear lots of clothes, umbrella etc. The rain storm is out of area by 9am and rest of day is hot and sunny.
I'm not familiar with the industry. Are other providers doing this?
(I assume that the question of whether these sensors are at all useful is another matter entirely.)
[1]: https://www.climacell.co/company/
[2]: https://www.climacell.co/blog/virtual-sensing/
[3]: https://www.fastcompany.com/90411326/how-cities-are-using-hy...
https://github.com/Mrjohns42/WeatherDash/blob/master/README....
https://journals.lww.com/health-physics/Abstract/2018/11000/...
I'm going through ESPHome right now, but in any case, any recommendations on very low-key but reliable thermostats that are amenable to hacking with them would be super welcome!
At home, for example: more accurate irrigation control based on local rainfall, greenhouse control, microclimate variations (which can be significant if you live in a hilly or mountainous region), observatory control for astronomers etc. There are lots of reasons why your local conditions could vary considerably from the airport, but you do need to put some effort into enclosures to avoid biases from solar radiation. You can also add other sensors like buried temperature sensors to monitor the soil, cloud cover, etc.
Microclimate is significant. If you live in a city, you'll likely have urban heating effects. If you live up a hill, you will have significantly different measurements to an airport in a valley. Anyone who's been hiking knows that you absolutely need local observations to assess conditions (eg from a hut). Though this is more for illustration, if you're hiking there are normally special weather services for common routes (Austria has a nice site for this I think).
The Met Office runs an API where you can contribute your own measurements [1]. Sites are rated based on quality of instrumentation - note that the gold standard here could be a calibrated mercury thermometer in a Stephenson screen, nothing special. Calibration is key.
These simple weather stations can also be used indoors as cheap multi room thermometers. You can pair them with air quality, gas and particulate sensors. Also a nice exercise in mesh/multi sensor networking.
Our government since WW2 has subsidized any city which wants an airport so they can have one.
I just took a look at the Dallas, Texas airspace as an example and there are 14 weather stations within the ring defining the metro area's restricted airspace. It looks like the airspace around London-Heathrow has 3.
If you are interested in the wind direction or the height of the cloud bases, the aviation weather stations are OK. If you are interested in temperature data, you will have to look elsewhere. (Many states have observation sites designed for weather purposes. The Oklahoma Mesonet was the first, but other states have them too. Often these sites have very interesting information, like soil moisture and temperature, and temperatures at varying heights above the ground: http://www.nysmesonet.org/about/sensors http://www.mesonet.org/index.php/site/about/instruments)
The standard data output of an FAA weather station is in degrees Celsius, however, there is no accuracy more than that.
If you are gardening in LA, and you knew the hyper local wind conditions in your plot, the light level, and the ambient moisture, you could develop an automatic watering solution for your garden that is truly hands free in all operation and can be customized based on individual plants needs. Many existing auto watering systems use soil sensors that are known to corrode and need replacement, so if you can infer soil moisture content from other non corrosive means that would be much preferred. You'd probably want to use the soil sensors at least initially to get an understanding of the function between your weather data and the particular soil composition of a given plant, but then they wouldn't have to be continually replaced.
After the recent very hot and dry winds from the desert (normally wind comes west) killed off some of my more temperamental plants last week, I'm thinking about some sort of automatic solution like this.
Otherwise, your tax dollars are paying for those airport readings, use the data you're footing the bill for already, imo ;).
[1] https://en.wikipedia.org/wiki/Citizen_Weather_Observer_Progr...
But then you are 100% reliant upon them for their graphs and app displays, which are slow and often broken, and give you little choice in formatting. For example, the app doesn't display a wind history at all, but the web does. But it won't display on a variety of mobile devices very well.
I know they're free (at the advertising level anyway) but I would love to see some competition with better display options and broader browser / device support.
I currently "tee" my transmissions through another server, but only because I have a model of station (no longer made) that made this trivial. Now you have to intercept the RF which is much more of a pain.
It's my long passion to 'activate' all the billions of barometers and use them in weather forecasting. There shouldn't really be much need to manufacture so many weather stations when so much can be done using already deployed hardware (smartphones).
I know IBM is doing assimilation of barometer data into their new models from Weather Underground app installs, and I suspect Apple may finally glance at using their iPhone barometers for weather forecasting now that they bought Dark Sky, but I also don't trust either group to make the results and research public and for the greater societal good - I expect Apple and IBM will mostly use their barometer tech for their own internal, private weather forecast systems.
One of my original goals was to make the financial entry-barrier as low as possible. Another — to provide educational context for covering a full-stack IoT development from firmware to backend to frontend. This means using cheap hardware and open-source software. In fact, I spent around 9$ on everything that was needed for this project.
I wouldn't say that this project is over-engineered with cool tech. I tried to keep the setup as minimal as possible, sticking to industry-standard data exchange protocols and frameworks.
Of course, all of the software part could be written using ANSI C or C++, but from my point of view, it wouldn't make the project simpler, because there are more specialized tools that are better suited for the particular task, be it backend, frontend or firmware (Rust, HTML5+JS and C in case of this project). I have outlined rationalization behind this particular technology stack under the "Choosing the technology stack" heading.
You are right that in a commercial setting, there might be myriad more inputs that constrain the technology choice like team experience, your company's recommended technology stack and etc. Of course, we can't take into consideration all these hypothetical requirements, as they depend on the environment you are working in and the commercial goals of the project.
To make your comment complete, could you share your ideas on how to architect this alternatively to get the same result with less than ten bucks, maybe an hour or two, and just a handful of code, if any? If it is a viable alternative I'll consider adding it into the post as an alternative tech stack, so people reading it will see more alternative options on how to build this project.
If your goal is to play and learn about stuff, whatever you're doing is fine.
If your goal is to get basic weather information, buy a used weather station with usb access, plug it up to whatever hobby computer you already have (I have a pi, so I'd use that). Then write a cron to read from the station and write it out to some publicly-available spreadsheet or database. There are a bunch: Google, Airtable, and so on. Most all of these can be written by one POST.
Then you can figure out how you'd like to see the data. Add in a station ID for more fun and a distributed system. I'm seeing Ebay new weather stations as low as 24 bucks or so. That'd be the only cost I know of, and I think with some searching for used gear you could get it under $10.
There's a _huge_ difference between focusing on the problem and focusing on the tech. If you're preparing yourself for a world of corporate coding? Please keep learning about message queues and so forth! You'll need those skills. If, however, you just want to know the weather in various places with your own setup? Do that. Just don't do one of those things while pretending you're doing both. Each project has vastly-different parameters.
To drive this home, if I've decided I want a coder to come to my office and make a cool little way of telling the weather outside? I want the project that just does that. Code for an hour or two, then leave. If you come and start deploying a kubernetes cloud-based auto-scaling megasytem, you're coding for yourself instead of my problem. My only point was that since there is such a huge gap, when you teach people you should be clear about the hidden lessons of what you're teaching.
* Now traditionally programmers throw out a lot of "what if"s here. What if we needed split-second reports? What if we're dealing with an offline situation or a saturated net? All of those are great considerations. Also, all of those were not included in the spec I read back to you. Don't go inventing what-ifs where none existed. It's a good way to build unmaintable monsters. Hopefully you got the point. It was a minor point and not directed at you, just a comment on cool-tech articles in general. (Also, I'd love to demo actually building it, but it would probably be quite less dramatic and interesting than your version, for the reasons I explain above)
Keep up the good writing! Looking forward to reading you again.
https://www.metoffice.gov.uk/research/climate/maps-and-data/...
I'm not sure if the CAA and Met Office share stations though. Airports often have more specific instruments like ceilometers to give measurements of cloud layer height, while AWS might have more cliamtologic things like pyranometers for sunlight hours.
Thanks for pointing that out, that is definitely not the site we put up for that project (~2011-2015)
I just am skeptical about this all. cant trust users phones to provide input for mission critical weather forecast models (GFS). Peoples lives depend on this information!
No way. The data is quality-controlled and bias-corrected before assimilation. Any bad data is thrown out. The papers I linked above do a good job describing how this is done.
> What if joe forgets to maintain his station?
No maintenance required. Individual data points are irrelevant, this idea only works if you can throw out bad data without caring. So throw out any suspicious data.
> How do you assert quality control?
Sorry to keep directing you to the papers above, but a team of really smart people spent years on this topic and they do a better job of describing it than I could. Cliff Mass et al.
> I just am skeptical about this all.
AFAIK the Weather Channel has been providing your forecasts for some years now in Apple and Google products with all of this barometer data from phones and weather stations in backyards. If you didn't notice a strong quality decline since some years ago, then I would say it's fine. You've probably noticed a forecast quality increase during this time I'd guess.
> cant trust users phones to provide input for mission critical weather forecast models (GFS).
Why not though? What is wrong with any of this method? It is all scientifically tested, researched, peer-reviewed, published, analyzed, post-event-analysis, the whole deal.
> Peoples lives depend on this information!
Well yeah, that's why we need more data! We have so little data now that people are dying from preventable cases like a lack of storm warning or good communication about incoming weather events.
This all matters so much. Ignoring billions of deployed, internet-connected barometers is costing lives that could be saved if we used all the data available to us to make accurate weather predictions.
To answer your question, if you want to obtain raw sensor data our government (and some volunteers) maintains a network of thousands of weather stations. Many are automated, many are people doing manual obs. If you know how to write code you can programmatically obtain these data.
However you might be confused... climacell does forecasts and you asked about finding station data? They are completely different things. I have no insight into what climacell does but i am a knowledge person in industry. I can tell you that nobody has some groudnbreaking 'AI' machine learning algorithm that is amazingly great at forecasting weather. If such a thing did exist, that person would have zillions of dollars because they can accurately predict weather better then anyone else.
https://www.microsoft.com/en-us/research/publication/meeting...
Perhaps using 4G and GSM signal quality as an ad-hoc doppler or metric for precipitation. Is their landing page inaccurate? Are they no different from other companies (besides the good at sales and marketing bit)? They seem to have a lot of big names on their customer list.
I'll have to see if the newer-model Ambient here can be pointed to PWS.