Cloud-based management platforms should not oversee inverters directly.
Only deep inspection of the silicon and code can improve the situation.
Perhaps Western blocks could develop provably secure silicon IP and code, formally verified, and perform continuous random sampling on imported goods, including full multilayer silicon inspection; publish it for free and refuse to import products that don't cooperate.
Edit: did some research and apparently it varies - many modern inverters can be remotely controlled by manufacturers - if they’re setup to allow it and are internet connected.
The article is still sensationalist about the risks though
>In the increasingly interconnected global economy, the reliance on Cloud Services raises questions about the national security implications of data centers. As these critical economic infrastructure sites, often strategically located underground, underwater, or in remote-cold locales, play a pivotal role, considerations arise regarding the role of military forces in safeguarding their security. While physical security measures and location obscurity provide some protection, the integration of AI into various aspects of daily life and the pervasive influence of cloud-based technologies on devices, as evident in CES GPT-enabled products, further accentuates the importance of these infrastructure sites.
>Notably, instances such as the seizure of a college thesis mapping communication lines in the U.S. underscore the sensitivity of disclosing key communications infrastructure.
>Companies like AWS, running data centers for the Department of Defense (DoD) and Intelligence Community (IC), demonstrate close collaboration between private entities and defense agencies. The question remains: are major cloud service providers actively involved in a national security strategy to protect the private internet infrastructure that underpins the global economy, or does the responsibility solely rest with individual companies?
---
And then I posted this, based on an HNers post about mapping out Nuclear Power Plants:
https://news.ycombinator.com/item?id=41189056
[We can easily map the infrastructure of the cloud and AI -- and their supply chains - and these are increasingly of National Security Concern:]
((Not to mention the actual powerplants being built to exclusively provide datacenter power))
Now, if we add the layers of the SubmarinCableMap [0] DataCenterMap [1] - and we begin to track shipments
And
https://i.imgur.com/zO0yz6J.png -- Left is nuke, top = cables, bottom = datacenters. I went to ImportYeti to look into the NVIDIA shipments: https://i.imgur.com/k9018EC.png
And you look at the suppliers that are coming from Taiwan, such as the water-coolers and power cables to sus out where they may be shipping to, https://i.imgur.com/B5iWFQ1.png -- but instead, it would be better to find shipping lables for datacenters that are receiving containers from Taiwain, and the same suppliers as NVIDIA for things such as power cables. While the free data is out of date on ImportYeti - it gives a good supply line idea for NVIDIA... with the goal to find out which datacenters that are getting such shipments, you can begin to measure the footprint of AI as it grows, and which nuke plants they are likely powered from.
Then, looking into whatever reporting one may access for the consumption/util of the nuke's capacity in various regions, we can estimate the power footprint of growing Global Compute.
DataCenterNews and all sorts of datasets are available - and now the ability to create this crawler/tracker is likely full implementable
https://i.imgur.com/gsM75dz.png https://i.imgur.com/a7nGGKh.png
Nothing prevents the satellite to transmit the commands at night, if that feels more convincing to you.
Ask yourself what is the active area of a photodiode in your TV/... ? What is the active area of your light-bucket on a roof?
If we are discussing the solar panel that powers your outdoor garden light, I don't suggest to apply the precautionary principle, but we are talking about products that sum up to a significant fraction of grid power generation.
Since this didn't pass the smell test: the author is looking at nameplate capacity, which is a completely useless metric for variable electricity production sources (a solar panel in my sunless basement has the same nameplate capacity as the same panel installed in the Sahara desert).
Looking at actual yearly energy generation data, this is more like 1.5 times the generation of an average nuclear power plant (NL solar production in 2023: 21TWh, US nuclear production in 2021: 778TWh by 54 plants).
Which maybe puts more into perspective the actual risks involved here. I'm not saying there shouldn't be more regulations and significantly better security practices, but otoh you could likely drive a big truck into the right power poles and cause a similar sized outage.
At that moment, the attacker will not only blast the grid with the full output of the solar panels, but they will also put any attached batteries into full discharge mode as well, bypassing any safeties built into the firmware with new firmware. We must consider the worst case, which is that the attacker is trying to not only physically break the inverters, but the batteries, solar panels, blow fuses, and burn out substations. (Consider that if the inverters burn out and start fires, that's a feature for the attacker rather than a bug!)
So yes, not only is it 25 medium sized nuclear power plants, it's probably much higher than that! And worse, that number is growing exponentially with each year of the renewable transition.
This was probably the scariest security expose in a long time. It's much much worse than some zero-day for iphones.
A bad iPhone bug might kill a few people who can't call emergency services, and cause a couple billion of diffuse economic damage across the world. This set of bugs might kill tens of thousands by blowing up substations and causing outages at thousands to millions of homes, businesses, and factories during a heat wave. And the economic damage will not only be much higher, it will be concentrated.
The big risk is turning them all off at the same time, while under maximum load. That will cause a brown-out that no other power generator can pick up that quickly. If the grid frequency drops far enough big parts of the grid will disconnect and cause blackouts to industry or whole areas.
It will take a lot of time to recover from that situation. Especially if it's done to the neighbouring grids as well so they can't step in to pick up some of the load.
Power transformers have a loooooooot of thermal wiggle room before they fail in such a way and usually have non-computerized triggers for associated breakers, and (at least if done to code, which is not a given I'll admit) so do inverters and every other part. If you try to burn them out, the fuses will fail physically before they'll be a fire hazard.
Instead, they're going to get a few guys with guns and shoot some step of transformers and drive away.
The problem with infosec people is they tend to wildly overestimate cyber attack potential and wildly underestimate the equivalent of the 5 dollar wrench attack.
For solar panels, the nameplate capacity is usually also the power generated at the peak production time, which is the moment when an attacker turning off all inverters at the same time would have the most impact.
That is: for an attack (or any other failure), the most important metric is not the total power produced, but the instantaneous power production, which is the amount which has to be absorbed by the "spinning reserve" of other power plants when one power plant suddenly goes offline.
The peak theoretical power output of a solar panel depends on where it's installed, inclination, temperature, elevation, and so on. The actual peak power is going to take weather and dirty panels into account.
1kw nameplate in Ireland (or the Netherlands) is never going to give you an instantaneous 1kw output -- you're going to be lucky to see 60% of that.
So in addition to the other stuff people mentioned, you might be off by another factor of 2 there. They also said “medium sized” so let’s call it 3.
The only new (non-Russian) European design built in the past 15 years is the EPR at 1600 MW. The only new American design built in the past 15 years is the AP1000 which as the name suggests is 1000 MW (technically 1100). AP1000 uses a massively simplified design to try and be much safer than other designs (NRC calculations say something like an order of magnitude) but is not cost competitive against most other forms of power generation. Which is why after Vogtle 3 and 4 there are no plans for more of them in the US.
It's not that EPR is any better- they are actually doing worse in terms of money and time slippage than Vogtle did. Flamanville 3 had it's first concrete poured in 2007 and still hasn't generated a single net watt!
It turns out that the pause in building nuclear reactors in the west from about 1995-2005- both US (which actually was longer, from the early 1980's, after 3 Mile Island things still under construction were finished but nothing new was built) and Western Europe (after Chernobyl following a similar path) basically gutted the nuclear construction industries in both, and they haven't built back up. The Russians kept at it, and the South Koreans have moved in to the market (and China is building a huge number domestically, though I don't think they've built any internationally), but Western Europe and the US are far behind, and after Fukushima Daiishi I strongly suspect the Japanese are in the same boat. Without the trained workers you can't build these in any predictable way, and when you pause construction for a decade you lose all of the trained workers and it's really hard to build that workforce back up again.
Isn't latitude taken into account by grid operators for determining their expected peak output? The owners would otherwise be installing bigger (more expensive) converters than needed, so they'd know this value at least roughly. Even smarter would be to include the angle etc. but not sure what detail it goes into compared to latitude which is a very well-known and exceedingly easy to look up value for an area
I certainly see your point about it not being apples to apples, but on a cloudless summer day, the output afaik genuinely would be the stated figure (less degradation and capacity issues). The country is small enough that it's also not unlikely that we all have a cloudless day at the same time
One might well expect some sun in summer and put some of the used-in-winter gas works into maintenance or, in the future, count on summer production of hydrogen— although hacks are likely a transient issue so I wouldn't foresee significant problems there
The distinction is important, especially in the Netherlands, which has a capacity factor of only about 10%-15%, whereas most of the US will be at least 20%-25%, which is twice as high.
I'm not sure of the typical number of reactors in the Netherlands, but using the US average of 1.6/power plant may not be the most representative comparison.
Too much from production the frequency skyrocket, little production the frequency plunge.
Now classic grids are designed on large areas to average the load for big power plants, this way those plant see small instantaneous change in their output demand, let's say a 50MW power plant see 100-300kW instantaneous change, that's something they can handle quick enough. With massive p.v., eolic etc grid demand might change MUCH more for big power plant, like a 50MW P.P. need to scale back or power up of 10MW suddenly and that's way too much to sustain. When this happen if the demand is too much the frequency plunge, grid dispatcher operators have to cut off large areas to lower the demand (so called rolling blackouts), when the demand drop too quickly the frequency skyrocket and large PP can't scale back fast enough so they simply disconnect. Disconnecting the generation fall and the frequency stabilize, unfortunately most p.v. is grid tied, if a p.p. disconnect most p.v. inverters who have seen the frequency spike disconnect as well creating a cascading effect of quickly alternating too low and too high frequency causing vast area blackouts.
Long story short a potential attack is simply planting a command "at solar noon of 26 June stop injecting to the grid, keep not injecting till solar noon + 5'", with just "1 second or so" (due to eventual time sync issues) all inverters of a certain brand might stop injecting, making the generation fall, a bit of rolling blackouts and large pp compensate quickly. Than the 5' counter stop, all inverters restart injecting en-masse, while the large pp are full power as well, the frequency skyrocket, large pp disconnect causing most grid-tied inverter to follow them, there are large change an entire geographic segment of a grid fall. Interconnection operators in such little time do not know what to do and quickly the blackout might became even larger with almost all interconnection going down to protect active parts of the grid, causing more frequency instability and so more blackouts.
Such attack might led to some days without power.
I get your comparison and the following is probably obvious to most people, but I feel like it really needs to be said: this requires being there, having a truck and the willingness to risk almost certain jail time, whereas taking down all SolarEdge installations on the planet could probably be done by an anonymous teenager in a foreign country with nothing but a computer and TOR client.
(I mention SolarEdge because I just had to deal with one of their systems and it pissed me off, I don't actually know of any vulns)
When it is sunny in the netherlands, it is likely sunny everywhere in NL because of how small the country is.
This is the situation where having so much solar power capacity (kW) is dangerous.
The risk scales with energy output but it would not term nameplate capacity a "completely useless metric".
Every adult in Seattle eventually has to learn that if you have an activity planned on the other side of town, if you cancel it because it’s raining at your house you’re not going to get anything done. You have to phone a friend or just show up and then decide if you’re going to cancel due to weather.
Now to be fair, in the case of Seattle, there’s a mountain that multiplies this effect north versus south. NL doesn’t have that, but if you look at the weather satellite at the time of my writing, there are long narrow strips of precip over England that are taller but much narrower than NL.
Often friends of mine who live in my city report rain when I see none, or no rain when it's raining outside my window. That's to say nothing of a location 30km away, where basically anything can happen. Do we live on the same planet?
> It wasn’t necessary from a technical standpoint to let everything run through the manufacturer’s servers, but it was chosen to do it this way.
(emphasis from article)
I'm working on IoT cloud system. It was chosen to be done this way because netither consumers nor installers have any expertise whatsoever to setup their own network or any devices to be acessible from outside (and they want their panels to be accessible when they are outside their home). I can do it, most readers of HN could do it, but typical consumer or installer can't. Sad but true.
I'm thinking about screwing it open and desoldering the wifi module but honestly I'll replace it in the next couple of years so I'd rather not kill myself by making a mistake.
Be careful with this language, especially when you're involving politicians and the non-technical.
The current atrocity of criminally negligent IT infrastructure right now is mostly created and driven by people with diplomas, including from the most prestigious schools. (And a top HN story over the weekend was one of the most famous tech company execs, turned government advisor, advising students at Stanford to behave unethically, and then get enough money to pay lawyers to make the consequences go away.)
And most of the certificates we do have are are individual certifications that are largely nonsense vendor training and lock-in, and these same people are then assembling and operating systems from the criminally negligent vendors. And our IT practices certifications are largely inadequate compliance theatre, to let people off the hook for actual sufficient competence.
My best guess for how to start to fix this is to hold companies accountable. For example, CrowdStrike (not the worst offender, but recent example): treat it as negligence, hold them liable for all costs, which I'd guess might destroy the stock, and make C-suite and upper parts of the org chart fear prison time as a very serious investigation proceeds. I'd guess seeing that the game has changed would start to align investors and executives at other companies. What could follow next (with growing pains) is a big shakeup of the rest of the org chart and practices -- as companies figure out that they have to kill off all the culture of job-hopping, resume-driven-development, Leetcode fratbro culture, IT vendor shop fiefdoms, etc. I'd guess some companies will be wiped out as they flail around, since they'll still have too many people wired to play the old game, who will see no career option other than to try to fake it till they make it at the new, responsible game (ironically, and self-defeatingly, taking the company down with them).
This article shows that we inadvertently introduced new choke points. And of course the global security environment makes it more worrisome.
This focus on paper qualification to mitigate risk seems a very European approach. Not saying it is wrong - it is just not emphasized as strongly elsewhere. And while it seems like a good fit for a slow-moving industry with high expectations of safety, the solar/wind world is not a slow-moving industry.
From experience in this sector though, I think the real issue is a lack of technical awareness and competency with enough breadth to extend into the "digital" domain - often products like these are developed by people from the "power" domain (who don't necessarily recognise off the top of their head that 512-bit RSA is a #badthing and not enough to use to protect aggregated energy systems that are controllable from a single location).
Clearly formal diplomas/certificates are not needed for that - some practical hands-on knowledge and experience would help a lot there.
When a product gets a network interface on it, or runs programmable firmware, we should hear discussions about A/B boot, signatures, key revocation, crypto agility to enable post quantum cryptography algorithms, etc. Instead, the focus will be on low-cost development of a mobile app, controlled via the lowest-possible-cost vendor server back-end API that gets the product shipped to market quickly.
Let's not even go near the "embedded system" mindset of not patching and staying up to date - embedded systems are a good place to meet Linux 2.4 or 2.6, even today... Vendors ship whatever their CPU chipset vendor gives them as a board support package, generally as a "tossed over the wall" lump of code.
I doubt many of these issues (which seem to be commercial/price driven) will be resolved through paperwork, as you say.
How someone would wave a 20 year old piece of paper as evidence that they know how to use solar tech that was developed last year, I don’t know.
I do not allow any system into my environments (at home and at work) that requires a third party data connection function.
There are way too many incidents where a provider, cloud or otherwise which required connection failed for various reasons.
(e.g., Cisco Spark Board, Xerox ConnectKey, Google Cloud Print, WeWork's Connected devices, Lattice Egnines, MS Groove Music Pass, Shyp, Adobe Business Catalyst, Samsara, Zune, FuelBand, Anki Vector Robot, Google Stadia, Pebble)
Despite this, I am very leery of regulating solar power specifically.
Anyway, I agree that there should be a regulation that forbid remote management, and you can only consult data in a read only manner remotely (you could air gap the inverter with the internet gateway using a one way rs232 connection where the inverted just write continuously). And if grid operators need to be able to turn solar off, they should install relays controlled by their infrastructure.
As always, the vulnerability of enabling remote updates. When will people learn? Updates should only be possible if there's a physical switch (not a software switch) on the device. If it's "off", no updates are possible.
Isn't the most devastating attack vector remotely installing malware? With a hardware switch, none of that malware will survive a reboot of the device.
I remember when hard disk drives came with a write-enable jumper. Then, once you've made a backup, the jumper is removed. Then it is impossible to accidentally or maliciously write over your precious backup.
Seems to me that this is very similar to the situation with IoT only with higher stakes. I appreciate this article's presentation of inverter and grid trust.
Beyond trusting customer inverters to do the right thing, I wonder if there is a method for safing a grid at the hardware level. Naive question: could there be a grid provider device that prevents overcurrent or incorrectly clocked cycles?
https://github.com/veganmosfet/Balcony_in_the_cloud
https://github.com/veganmosfet/SolarFlareSec
https://github.com/veganmosfet/CyberEclipse
https://github.com/veganmosfet/SecureWatt
A big mess, but it's getting slowly better...
There is a more important part, while with p.v. we still can go offline, with car's we can't. My car is connected and I can't do NOTHING to manage it, it's managed by it OEM behind me and that's a much bigger threat since single cars can paralyze the nation if properly blocked in critical points of the road network.
At a largest scale that's the reason we can't have a national smart grid but only individual smart microgrid, meaning p.v. should be used only for self-consumption NOT grid-tied like in California.
This results in almost all of the situations threads here address.
In a better timeline, everyone has stable and secure OSs on all their devices, and the default is for everything to be locally networked, with optional monitoring from the outside via a data diode.
They measure the PV panels anyway for MPPT.
It would be interesting to use a solar panel as a sensor for moonbounce of an IR laser, though. You'd have to try at the almost new moon, with a window of a few hours/month ;-)
A data diode applies to a specific connection. It's easy to have a serial port that goes one way.
Preventing any possible input to an already compromised device is much harder. But if your device isn't already compromised then it won't be looking at the input light levels for commands.
This seems likely to be more fruitful than attempting to regulate 400 Chinese panel manufacturers.
What am I missing?
The "plants" in this context are homes and businesses. The junction points between plants and grid are the inverters sold by the panel manufacturers.
The inverter itself functions perfectly fine without an internet connection, and will display instantaneous power output on the screen. I could just be content with that and look at my monthly power bill to see how much I generated and how much I used each month and never connect it to the internet.
To get any kind of data logging & history from the inverter, it must be internet connected (wifi or ethernet). And all of that is through the manufacturer's website, which constantly nags me to "upgrade to pro" for some obscure feature that I'll never use.
Micro inverter for each panel would be very costly. In 1 MW plant you will have around 4000 panels, communicating with that amount electronic devices would be a headache.
Perhaps that's a component, but one really doesn't need to think about it too hard to identify better explanations for why this particular energy source is held to unusually high regulatory standards.
I don't have an opinion as to whether other large-scale sources of energy should be held to similar standards, but to suggest that solar energy's failure modes are comparable to nuclear energy seems intentionally misleading.
> In the Netherlands alone, these solar panels generate a power output equivalent to at least 25 medium sized nuclear power plants.
> Because everything runs through the manufacturer, they are able to turn all panels on and off. Or install software on the inverters so that the wrong current flows into the grid. Now, a manufacturer won’t do this intentionally, but it is easy enough to mess this up.
> As an interim step, we might need to demand that control panels stick to providing pretty graphs, and make it impossible to remotely switch panels/loaders/batteries on or off.
Basically, if a hacker were to make all batteries (or panels) suddenly switch between full discharge and full charge every second or so, it would tear down the electric grid. Voltage and frequency would swing rapidly, and whatever plants are riding load would struggle.
This could create a massive power outage; but there is a huge risk that this could damage power plants and other infrastructure.
> There are also people that claim that the many Chinese companies managing our power panels for us might intentionally want to harm us. Who knows.
Wait, seriously? The European power system relies on Chinese companies not messing it up remotely? And the debate is over whether the companies will stay nice? For heaven's sake, isn't it obvious that during a war the Chinese government can force them to just destroy the continent's power system remotely? How is this not seen as a extreme continental security risk?
On top of that PVs installed at homes are insignificant to cause such troubles. As the size of installation increases, you will have different connection agreement and certain requirements. You can't install 15 MW and connect through inverters that are used at home, which is 100 kW at most. Even 15 MW is insignificant change for a grid.
There have been plenty of botnets in the past. Some even in the millions of computers. If such a botnet decided to make every node's power draw fluctuate per above, wouldn't this cause the same type of problem? Is there a reason we've never seen this happen despite large enough networks of hacked machines existing?
And that presumes an attacker can switch on/off all 8.4million computers in a small timeframe. 100% of them would need to be on, online and hacked.
I don't think this is a realistic problem.
Tesla F-ing up an OTA update that suddenly switches all charging Tesla's off, is probably a theoretical worse scenario.
I found a figure on Wikipedia saying that the NL's 4.7GW worth of offshore wind capacity is 16% of their total electricity demand nationwide. 4.7/.16 = 30GW total, so this theorized computer load attack would represent about 10% of their grid's total capacity. Can their grid add and shed that much load that quickly? That's the part I doubt.
and sounds like something that could easily occur
The hardware is modular and the software is above any competition. Choose the responsible vendors.
This is because of the market for carbon credits. When you installed your PV panels, someone estimated how much electricity they would generate over the next 10-15 years. Tradable carbon credits were created based on that estimate and went into the marketplace. And for the next 10-15 years they have to verify that the electricity was actually generated, or else someone has to pay back some money. Did you read the fine print on your contract? It is probably you that has to pay it back. You didn't know that one of the "rebates" you got was actually a pre-payment for those credits?!? Should have read the fine print ;)
Oh yeah, BTW: that "rebate" was only your portion of the credits. The installer got some of it (and doesn't have to pay back anything), the person that filled out the paperwork you didn't know existed got some (and doesn't have to pay back anything)...
This also makes it simpler from a programming point of view - instead of having separate cloud sync & local control protocols, you just have one local protocol and you merely tunnel it through the (dumb) cloud if you can't connect directly.
> This also makes it simpler from a programming point of view - instead of having separate cloud sync & local control protocols, you just have one local protocol and you merely tunnel it through the (dumb) cloud if you can't connect directly.
Having only cloud protocol is even simpler, I've done all of the above (I do backend and our firmwares).
I have done a controller for a 40 foot container battery and it wasn't like we received any API from Hitachi (battery manufactor). We had to write everything ourselves.
However, for purpose of the security of the nation's power grid, I don't just need my inverter to be secure, I need pretty much everyone's inverter to be secure. If an attack bricks 95% of solar inverters, the fact the nerdiest 5% of users have their inverters airgapped won't stop the grid having a lot of problems.
That said, Apple Homekit integration is local network based, so products that do that and the typical manufacturer cloud system have done both paths.
Homekit is a pain to use without Apple hardware/software, but there you go. (There's a plugin for HomeAssistant, but I'm still classifying that as a pain)
I can understand people wanting to be able to see the metering live, but remote control of the panels just seems like a security incident waiting to happen. I'm quite glad I have a non-internet-connected inverter.
The standard go to a raz pi solution will up and die every few months. And half the time you'll need physical access to get it back. It takes a lot of work to develop an embedded system that has enough reliability.
Like you wouldn't believe.
My most memorable case of insecure IoT devices - wifi socket was sending wifi ssid and password of the network in cleartext in every ping packet to chinese servers.
Large manufacturers would like you to think this. It would provide them a convenient excuse for not even trying to differentiate the market along these lines.
> We (the society) don't even want to change the default password on most things.
Actually.. I just want to use my device _first_ and not go through some manufacturer controlled song and dance of dark patterns.
In my experience, if you don't pre load the user with this garbage, and then wait for them to have an actual _need_ that depends on the feature, they're FAR more compliant with following even lengthy instructions to get it done.
It's more a problem of aligned benefits and timing than anything else.
The sole remark I have against them (beside the not so good software quality it's the impossibility for individual owners to do offline updates, we can upgrade via VRM portal but not downloading fw and flash it locally even if the needed device is on sale, because they offer fw files only to registered vendors.
Fronius (to remain in the EU) have a local WebUI witch need a connection only for fw updates, even if differently from Victron it's not a Debian based system with sources available but a closed source one, they unfortunately offer only a very limited REST API and a very slow ModBUS but still anything con be do locally.
I'm not sure, since I haven't any myself by SMA (Germany) and Enphase (USA) seems to been able to operate offline as well.
Stated that, yes, you are damn very right in saying most installers have no competence, thankfully where I live self-installation is allowed (at least so far), but that's simply demand better UIs and training for them perhaps avoiding the current state of the industry with an immense amount of CRAP at OEM level, with most "state of art" systems not at all designed to be used in good ways (see below) and absurdly high prices to the customer at a level it's not interesting installing p.v... 4 years ago I paid my system 11.500€ for 5kWp/8kWh LFP, the smallest offer to have it designed and built by someone else was ~30.000€ the most expensive ~50.000€ and all the 6 offers I tried shows some unpleasant issues and incompetence.
About OEMs just observe how ABSURD is that there is no damn DC-to-DC direct car charger. Most EVs now have 400V batteries, the same of stationary batteries, with equal BMS comms. Why the hell not sell an MPPT-to-CSS combo direct solution? Ok, we do not ONLY charge from the Sun, than it's perfectly possible have a compo charging station with DC for p.v. and AC for the grid, switching from one to another as needed. It's ~30% energy lost in double conversion.
Why no DC-to-DC high power appliance who still run DC internally (A/C, hot-water heat-pump heaters etc)?
Why not a modern standard protocol for integration of anything instead of building walled gardens?
Long story short OEMs have choose the cloud model partially because most installers are electricians able to use desktop holding the mouse with one hand and clicking with the other, but also because they have no intention to made user-interesting solution in an open market...
Certainly this requires a lot of progress to secure the IOT space, but we can allow the enshitification of clouds to continue.
I call bullshit. They've been conditioned to think that they want it, because all product brochures have it.
What kind of tangible benefit could there be to know how bright the sun is at your home while you're not there? A cool party trick to virtue signal or a break between doomscrolling, I suppose, but it's not like you're gonna jump up and drive back to... what even could you do if you knew?
You gave reason WHY they want it. Maybe consumers were conditioned to want access, but they still want access. If you give them similar devices, they will chose the one which has application or webpage to see how their big investment is actually working. It's not about current state of device, it's all about historical data and month-by-month savings presented as a nice graph. They will check this maybe every week or month (later every several months), but buyers still want to know what their installation did for them.
It reminds me of <https://xkcd.com/627/>, but when you're launching a product that isn't good enough.
It's hard enough to open up a port even with uPNP (typically disabled) and other made-for-purpose tech. Torrent clients end up trying to poke holes and such. Service discovery might work via local UDP broadcast, or it might not. LAN clients might live at 10.* or 192.* or be isolated by default. It's easier to just go onto the public internet and contact some mysterious server. Botnet by design.
Governments should have done the same thing as with digital TV transition(s) : first ban selling devices that can't do IPv6, then ban selling (most) devices advertising they can do IPv4.
It may be sufficient to just disconnect the antennas from the WiFi module, that will help prevent any network connections.
Compromising an individual one by getting close-range physical access will be a local annoyance but wouldn't scale to a level where it can threaten the grid, so it limits the pool of potential attackers to local vandals (which can achieve their goals easier by just throwing rocks at your panels).
Option 1: Artificially sell the thing as an ongoing cost.
Option 2: Artificially make the depreciation cycle faster. Get consumers to regularly replace it anyway with upgrades or trend changes.
Option 3: Make ongoing money from the item via a side-channel (tvs are great at this one)
Option 4: Manufacture and sell a huge number of different goods across market segments and weather the slow depreciation cycle (Oxo does this).
Option 5: Sell some consumable good you can get recurring revenue from along side the item (Coffee pods, printer ink)
Option 6: Make up the money on maintenance, repairs, and financing. Become a bank.
Option 7: Make your money in some other sustainable profitable business and drop the product once you've gotten what you can for it.
All of these kinda suck and option 1 is easy to implement.
Punishment for mistakes is what led to the Chernobyl disaster.
But I'm concerned about the entire field of software, which doesn't have that sense of responsibility, and I don't see how it would get it. However, software industry -- both companies and workers -- are guided almost entirely by money. To the point that it's often hard to explain to many people in HN discussions on why it would be good to behave in any other way than complete mercenary self interest. So I don't see any way to get alignment other than to link money to it. If people see that as punishment, so be it.
we see competition in cloud/IaaS providers because they actually need to build datacenters and networks and so there's some price floor, but when it comes to "antivirus" CrowdStrike was able to corner the market basically, and downstream from them not a lot of organizations/clients/costumers can justify having actual independent hot-spare backups (or having special procedures for updating CS signatures by only allowing it to phone home on a test env first)
the cultural symptoms you describe in so much detail are basically the froth (the economic inefficiencies afforded) on top of all the actual economic activity that's sloshing around various cost-benefit optimum points.
and it's very hard to move away from this, because in general IT is standardized enough that any business that needs some kind of IT-as-a-service will be basically forced to pick based on cost, and will basically pick whatever others in their sector pick -- and even if there are multiple providers the will usually converge on the same technology (because it's software) -- thus this minimizes the financial risk for clients/customers/downstream, even if the actual global/systemic risk increases.
Knowledge without reasoning is how you get mired in bureaucracy.
BS gatekeeping rituals and compliance-for-sale theatre are arguably just symptoms -- of companies and individuals not being aligned with developing trustworthy systems.
Renewable and decentralized are different axes.
That's why instead of pushing self consumption and semi-autonomous systems we push grid-tied and cloud-ties crap, to be tied to someone else service, slave of that. It's the "in 2030 you'll own nothing" already a reality in modern cars, connected to the OEM with a much higher access than the formal owner, much modern IoT and cloud+mobile crap. People do not even understand they do now own, until it's too late.
Another simple example: in most of the world banks between them have open standard to automatic exchange transaction, in EU that's OpenBank APIs, with signed XML and JSON feeds. There is NO REASON to block customers for directly use such APIs from a personal desktop client. All banks I know block such usage. So you do not have all your transactions signed by the bank on your iron, you have NOTHING in hand. In case of "serious issues" you have nothing to prove what you have on your bank, what you have done with your money. In the past we have had paper stuff to prove, we now have signed XML/JSON witch is even better than paper being much harder to falsify, but no, we miss because 99% must own nothing.
We have connected cars with a SIM inside, but instead of having the car offering APIs and a client or perhaps even a WebUI, directly to their formal owner we have to pass through their OEM, the real substantial owner. And we can't even disconnect the car. In the EU it's even illegal for new car to be disconnected since the emergency e-call service must be active on all new cars.
And so on.
You can set up a local network that has no WAN connectivity on it. About anything else is difficult to verify even the most basic of security properties. Certifying is another step up (although you could argue certifying is just a third party saying something passed a finite list of tests) - the real challenge is defining a meaningful certification scheme.
There has been some good work towards consumer IoT device security (i.e. the 13 steps approach from the UK), that covers some of the lowest hanging fruit - https://www.gov.uk/government/publications/code-of-practice-...
The trouble is that these set out principles, but it's hard to validate those principles without having about the same amount of knowledge as required to build an equivalent system in the first place.
If you at least know the system is not connected to a WAN, you can limit the assurance required (look for WiFi funcitonality, new SSIDs, and attempts to connect to open networks), but at a certain point you need to be able to trust the vendor (else they could put a hard-coded "time bomb" into the code for the solutions they develop).
I don't see much value in the academic/theoretical approaches to verification (for a consumer or stakeholder concerned by issues like these), as they tend to operate on an unrealistic set of assumptions (i.e. source code or similar levels of unrealistic access) - the reality is it could take a few days for a good embedded device hacker to even get binary firmware extracted from a device, and source code is likely a dream for products built to the lowest price overseas and imported.
Are you suggesting using smart phones should count in "not allowing it in"? Then yes, I try to where possible. I do not depend on a smart phone. All functionality that are operationally necessary can be done elsewhere without major delays or impact.
There isn't really any practical way to prevent overvoltage though. So a rogue controller in charge of all the solar systems in a street might be able to do quite a lot of damage to consumer devices.
A problem from the utility point of view is that they can no longer guarantee that the 240 V side of the distribution system is safe to work on just by tripping a breaker on either side of the distribution transformer. So all work on the 240 V distribution system has to be done with the assumption that the system is live.
Eventually regulations will be updated, if necessary, to deal with large numbers of solar installations on domestic buildings.
To put it more simply: if the phase is wrong, the effect is the same as a short circuit, which fuses and circuit breakers protect against. If the frequency is wrong, the phase will become wrong after a number of cycles.
> There isn't really any practical way to prevent overvoltage though. So a rogue controller in charge of all the solar systems in a street might be able to do quite a lot of damage to consumer devices.
There is, it's called a surge protector or surge protective device (SPD). It converts any overvoltage above a certain level into a short to ground, which then trips the fuse or circuit breaker. It's often used as a protection against lightning-induced currents.
> A problem from the utility point of view is that they can no longer guarantee that the 240 V side of the distribution system is safe to work on just by tripping a breaker on either side of the distribution transformer. So all work on the 240 V distribution system has to be done with the assumption that the system is live.
From what I've seen, the utility workers usually ground the wiring when working on it (they have a special-purpose device for that). Once it's safely connected to ground, it's no longer live.
What harm on scale is the variable output especially from small p.v. utilities built out of incentives NOT personal power plants, the grid is sized with some large power plants serving a large set of customers, their absorption vary but if the grid is vast (and not too vast) enough variation tend to be slow on average, let's say 50MW PP experience 100-200kW demand variation in very short time. They can compensate easily keeping the grid frequency stable. With a significant amount of grid injecting p.v. variation might be MUCH bigger creating significant stability issues where injection goes up too quickly making the frequency skyrocketing and large PP can't decrease their output fast enough risking disconnection witch in turn might put large p.v. plants offline suddenly creating a cascading effect of large blackouts.
That's the real issue with grid-connected and tied renewables and another reason why we need to go toward self-consumption NOT injection.
Owners of small scale solar panel installations are payed a fixed price per kWh in many EU countries, regardless of the market price. The taxpayers pick up the tab I guess.
this incentivizes better capacity and availability forecasting for solar installations, and preserves the usual dynamics of the open energy market.
..
the problem is with these super small ones, where initially states just let people connect it, because it's green, yey. (but now DSOs started to make connecting waay harder. and regulators are investigating, eg. in Spain. [0])
of course the non-residential installations already usually need aFRR capability. (eg. this is the case in Hungary.)
and there's already a market for "reserves" in the EU. (but the interconnection rate is below the target 15% as far as I know. but still, there are intra-state markets, etc.) and we can see that when solar is high the reserve prices are surging. [1]
[0] https://caneurope.org/content/uploads/2024/04/Rooftop-Solar-...
[1] https://gemenergyanalytics.substack.com/p/european-power-res...
Idaho National Lab is one of those places that researches this. https://inl.gov - their domains are energy (primarily nuclear and integrated) and national security ... and securing the grid is the intersection of that.
And some time back... https://www.wired.com/story/how-30-lines-of-code-blew-up-27-... ( https://web.archive.org/web/20201101002448/https://www.wired... ) . The story is from 2020. The event is from 2007.
The test footage linked in the article is on YouTube - https://youtu.be/LM8kLaJ2NDU
The wikipedia article on the test: https://en.wikipedia.org/wiki/Aurora_Generator_Test
From the wired article the key part of how it broke:
> A protective relay attached to that generator was designed to prevent it from connecting to the rest of the power system without first syncing to that exact rhythm: 60 hertz. But Assante’s hacker in Idaho Falls had just reprogrammed that safeguard device, flipping its logic on its head.
> At 11:33 am and 23 seconds, the protective relay observed that the generator was perfectly synced. But then its corrupted brain did the opposite of what it was meant to do: It opened a circuit breaker to disconnect the machine.
> When the generator was detached from the larger circuit of Idaho National Laboratory’s electrical grid and relieved of the burden of sharing its energy with that vast system, it instantly began to accelerate, spinning faster, like a pack of horses that had been let loose from its carriage. As soon as the protective relay observed that the generator’s rotation had sped up to be fully out of sync with the rest of the grid, its maliciously flipped logic immediately reconnected it to the grid’s machinery.
I'm not sure everyone is really thinking clearly here.
Don't get me wrong, they should get rid of this practice of cloud monitoring. A consumer should be able to access monitoring over the internet without an intermediary. They should, of course, be allowed to contract with an intermediary if that is their desire.
But the security argument?
Yeah, that ship has sailed. Total war, means total war. Your power grid, your internet, your communications, and your fossil fuel deliveries will all see material disruption. I wouldn't count on being able to stop those disruptions by banning a few web sites. (And frankly, during total war, those disruptions would be the least of your problems in any case.)
Best bet for places like Europe, China, the US and Russia is, just don't do total war with each other. If you choose to do it anyway, then you can see what you can expect from that in the documents filed under "Play stupid games, win stupid prizes."
Germany was an enemy once as well
It can connect to standard cloud weather service providers and I can view my data there.
I can also just redirect that exact same protocol to any other host or IP I specify.
They built it once and just gave me the ability to control WHERE that data goes. It's honestly not that hard.
employees? they get fired once the market is saturated and demand flattens. ITYM shareholders.
When you look at it from that angle, another possible solution is:
- downsize manufacturing plant/capacity to make it match (or slightly exceed) replacement demand.
But that would mean a steady state profit margin (and not a cancerously growing one), so it will never fly.
Organizations based on other incentives don't work, or work very poorly.
This is just a way of pretending to give access while making it as hard as possible. We are talking about a device that is already connected to the network. The local path is not some rest services, but a serial port for which I need to fabricate some hardware? Don't piss on me and tell me it's raining.
1. Cloud management with their app.
2. Wifi management without the cloud (when you're on your home wifi).
3. Unplug the wifi dongle from the inverter for a fully offline system. You don't really need your inverter on the internet anyway.
4. Unplug the wifi dongle and DIY whatever you want, the dongle's just a serial-to-wifi converter.
That's not to say the security of any of this stuff is good, of course. In fact the security is pretty bad! But you can for sure get inverters with multiple options for non-cloud operation.
Nope, they want to have it working out of the box like with any other manufacturer out there. If you enable functions only after user wants it, they will comment "this sh*t doesn't work" on your app store page. Then you have to respons to each comment with "what doesn't work, could you specify please?" and then after several days that user has enabled the functionality, but in the mean time several another gave such comments.
What if I told you those 3% of people will say this no matter what you do. These comments and the reality of your product are entirely disconnected. We had a small userbase and added voice uploads into the app; unsurprisingly, about 3% of them are clearly impaired in some way when they leave a message. [x-files theme].
In any case, a simple "Fast / Slow Setup?" question to start is all you need, and a "Do More Setup?" after they finish one item has, again, in my experience, been entirely sufficient.
Reasonable people understand, "oh.. this needs the cloud.. and I didn't do that part yet.. so I'll go ahead and click the 'social media provider' button." If you also decide this is a good time to ask about a news letter, well, you got what you bargained for.
Of course a company that skimped to the point of total neglect on software security would never skimp anywhere else, right? Right?
:crossed-fingers: <- This is what we are relying on here.
And even if they did all the right things with their physical safety, the attackers can still brick the inverters with bad firmware and make them require a high skill firmware restore at a minimum and turn them into e-waste and require an re-install from a licensed electrician at a maximum.
At least in Europe, product safety organizations and regulatory agencies have taken up work to identify issues with stuff violating electrical codes (e.g. [1] [2]) and getting it recalled/pulled off the market.
Sadly there is no equivalent on the software side - it's easy enough to verify if a product meets electrical codes, but almost impossible to check firmware even if you have the full source code.
[1] https://www.bundesnetzagentur.de/SharedDocs/Pressemitteilung...
[2] https://www.t-online.de/heim-garten/aktuelles/id_100212010/s...
Well not even high skill - for "security" reasons and to prevent support issues as well as to skimp on testing needed informations are often only accessible to a chosen few.
Paradoxically the effect of thes "security" concerns often mean that there are plenty of easily exploited methods in devices like that. And the only people that have them are the ones that you need to worry about instead of some 16 year old teenager finding it and playing blinkenlights with his friends parents house causing trouble for him but getting the hard coded backdoor taken out after the media got wind of it.
If I was dictator of infrastructure I would ban any non-local two way communication and would mandate all small grid storage solutions run off a curve flattening model thats uniform and predictable. Basically they would store first and only be allowed to emit a fraction of their storage capacity to the grid afterwards. Maybe regulated by time of day.
Aside I'm not much an expert as well but, from DC to AC we are pretty efficient, around 98% of energy get converted. But the other way round it's pretty inefficient, meaning a loss around 30%. So while your p.v. it's pretty efficient in generating AC, your car it's pretty inefficient generating DC from AC to recharge. You heat pump would gain as well.
To be clear I do not advocate DC for any DC home appliance like washing machines or dishwashers, just for energy intensive appliances like big heat pumps and car chargers. Not more.
Try to imaging how well you could run if you have a small battery, as an expensive UPS for the home when you are not there (ventilation, homeservers, fridge(s)/freezer(s) etc) and when you are at home, thanks to a let's say 70kWh battery you can also run your heating in winter in case of a blackout...
In the end it's the very same power circuit, why not? 300€ extras in breakers to run my heat pumps more efficiently? No issue, they'll pay back countless more.
Your solar TWh comes from 25GW at ~15% capacity factor, and to get your nuclear numbers you're looking at 1.6GW for each of nuclear "plants" when each reactor is usually about 1GW or less. There are ~90 reactors in the US, at 54 plants. The article is assuming 1 reactor per plant for the Netherlands.
Small addition that isn't mentioned in the English version of the article, but only in the original Dutch version: the article talks specifically about the Borssele power station [0] (which has a power output of 485MW).
[0]: https://en.wikipedia.org/wiki/Borssele_Nuclear_Power_Station
Engineering grid and PV installations acknowledge that the generation may be lost, so you are having contingency plan by means of transferring or picking the load. You're going to lose power for short time if you didn't do this properly.
Actually due to the nature of PV generation, no sun no generation, it is reckless to just rely on PV. If sun is shining that is great, there'll be generation. However, daily peak consumption coincides with less day light. So during planning the target is the extreme cases (statistically estimating demand), in other words you do load flow studies for extreme cases. This helps to see your capacity limits.
In parallel to this you should consider electric grid as a layered system. PV generation at house is the lowest level, so less impact. So when it is lost, or neighborhood or town lose PV generation it will impact nearby station, which is couple MW if not kW. So when you lose PV generation and you planned your system for extreme case, higher level of generation or substation will take control of it.
Losing GW solar does not mean you're losing that amount power in small geographical region. You have to divide that into so many small parts. Also, PV generation at GW level is too high for small region.
Hope this explanation helps. It is because how power flows, governed by rules of physics. Bottom line, if hacker wants to affect a grid they should target higher level of grid, PV panels on the rooftops will not help their cause, they are end of line.
So a more realistic "attack" would be able to move demand, 0.5% of the total grid capacity. Switching on/off one smelter in an aluminium factory, is probably more than that. Hacking a major charging-station company and switching off their chargers is probably more than that even.
I understand the direction you think, and I agree that the combined power usage of "consumer devices" is big. But the larger power system is rather well protected by an attack on these devices through the diversity of these devices and the diversity of their setup (consumer firewalls, routers, individual protection, in-house fuses, local load killswitches etc).
The solar devises lacks this diversity, as the article mentions. There are few brands, and all of a brand need to connect to the one cloud service in the exact same way. So this does have a single point of attack. Whereas "switching on/off all personal computers in a country" is of an entirely different level.
https://worldpopulationreview.com/country-rankings/computers...
And people still have problems following that instructions. We gave a lot of instructions, but some people just don't read them. Or can't understand and follow them. Example from this week:
Yeah, main support guy for our client is on holidays this week and only a girl from sales is available and she doesn't know why our device doesn't work, she tried resetting her phone wifi but still can't pair our zigbee smart hub connected with usb modem stick (problem: no one told her she needs to message us to actually activate sim card before installing device for end client).
Yes, you can solve this one problem, but there are many more we didn't see yet. Consumer support does not scale and you can't write tests for something you don't know will be a problem.
Did you have that written down somewhere? In instructions, which was my point.
Sorry for the snark, but I think "no one knows how to do anything" coupled with "Oh, the idiot didn't know to discombulate the canooter valve before inserting into the tinklerater--it's so obvious we didn't write it down" supports the point I'm making.
Because the competitor that doesn't have this installation "hassle"¹ will sell more units. It ends back up with society choosing to behave this way in aggregate
Society is obviously good (wouldn't want to live without it) and capitalism held up for hundreds of years now (not sure it's the best solution but with the tweaks that are in place in many well-faring countries it seems to work okay for them), but I do believe "society" does have a tendency to go for easy and cheap more than complicated and thorough when the need is not self-evident and has not been tangibly pointed out in living memory
¹ Example of hassle that HN users may not think of as exceedingly difficult: iOS setup. My dad asked me to help him set up his new tablet. I've fallen for Apple's motto and was annoyed he doesn't just do it himself. So we sit there and... lo and behold, it's good that he asked me. I forgot the details by now but things like what the heck screen time is, whether the Find My network is something to enable, whether to enable automatic updates, what to do about Siri, whether Apple Pay is something he needs to use banking apps, etc. are not things he'd have any idea of and I can give the necessary context from my IT background even if I don't have iOS myself. However, we still needed to contact their support because we both couldn't figure out how to get an Apple account set up. The continue button was disabled. I tried scrolling: nothing. Tapping the button: nothing. No error message, nothing. Turns out, the flow was only tested in portrait mode but my dad wants landscape mode easier for typing and so we never tried portrait. A part of the screen was in fact scrollable (I've used Apple devices before and my dad used the old tablet weekly, it's not like we're not familiar with their design patterns or don't know how to use a touchscreen) and revealed a required input field that would have been visible by default in portrait. Next, even setting up a standard Microsoft 365 email server in Apple Mail wasn't self-evident, iirc because he had never heard of the word Exchange and the oauth flow worked across multiple devices with again different bugs in Apple's own UIs (one needed landscape instead of portrait, another required some trick with zooming beyond intended bounds iirc)
"Hassle" is already perceived in setting up what people proclaim the most user-friendly of systems. Units that don't need it will absolutely sell better, even if that leads to a potential national energy supply risk
Unless your device itself is handling the VPN, I have bad news for you if you trust the mobile network to not open your devices up to malicious attackers: https://berthub.eu/articles/posts/5g-elephant-in-the-room/
For a simple demonstration, https://www.buienradar.nl/nederland/zon-en-wolken/wolkenrada... has been showing cloudless hours pretty regularly in the last month. Someone meaning malice can certainly keep an eye on that for a few days to find a good moment
The problem is what percentage are generally in full sun, and how low that percentage goes. My comment was about assuming that all of the panels are not in sun at the same time. Not whether sunny days exist.
> When it is sunny in the netherlands, it is likely sunny everywhere in NL because of how small the country is.
These statements are not the same. They're quite different, actually. One talks about most sunny days, one talks about one sunny day.
What point are you trying to make here?
Do you know why microinverters exist? They exist because panels next to each other can see different light, and without the micro inverter the entire chain produces no power at all if one of the panels in series is generating no power. So we use microinverters to rescue power stranded by trees, debris, or partial cloud cover.
But this is precisely part of the threat model, the manufacturers are best positioned to execute supply chain attacks on foreign buyers.
And it's vastly harder to do an attack like that.
Now other light sources will also integrate with time, this is where the modulation scheme comes in. First consider the amount of time you'd have to integrate the noisy signal to raise it above the noise floor. Thats the on time you need. How do we remove background light variations from other sources? Consider a discrete time pre-agreed pseudorandom sequence, that has "0" periods as often as "1" periods. To remove a constant background you take calculate the sum of light intensities of all "1" periods and the sum of all measured intensities of "0" periods. Then you subtract the "0"-sum from the "1"-sum, a constant signal will remove itself, the satellite signal will be summed N times. since your pseudorandom sequence was kept secret, random variations in light (think bird passing by) will not conspire to selectively block light during the "1" periods, so such noise will be uncorrelated with your pseudorandom signal. adding N uncorrelated noises grows by sqrt(N), so the S/N-ratio grows as sqrt(N). These are widely understood methods, an engineer might call it lock-in amplification, a physicist might call it correlation. This is very basic engineering / science knowledge. It's baffling that people consider this "hard" to execute, sure if you're the milk-man in a village this is hard to execute.
It can both be true that weather is locally different on most days but coincides to be universally cloudless on a fair number of hours every late-summer month (easily within a reasonable waiting time for an attacker)
> When it is sunny in the netherlands, it is likely sunny everywhere in NL because of how small the country is.
Nowhere there is the qualifier "a fair number of hours every late summer month". If you add arbitrary conditions of course you can get a different meaning.
The IT world seems to be an endless swamp of special cases and things breaking deep in the stack that require very specific, arcane skills from unwritten textbooks.
I forgot to respond to this sentence in the sibling response.
Supply chain attacks can be executed by intermediaries of the supply chain, or by manufacturers themselves: develop the capability to deny a foreign nation its energy infrastructure. The manufacturer is not a hacker in a basement. Manufacturers can be pressured by their local gorvernments, militaries, 3 letter agencies, ...
A precautionary principle would induce potential target nations to surreptitiously catalogue the inverter boards, sort them by most-GW serving type, and consider which control traces to cut to control the internal energy transfers in its inductors, capacitors, ... from a trusted parasite board. Just develop and test a few parasite boards for the most common inverters, and preferably have critical stock ready.
The main value in inverters is the power switches, inductors, capacitors, ... it would be cheaper to reroute the control to a trusted controller in the event of a calamity. We would survive fine, but it will be a painful few days.
I was responding to:
> With a hardware switch, none of that malware will survive a reboot of the device.
A reboot of the inverter would not prevent a supply chain attack using MPPT measurement electronics for an optical backdoor channel.
The hardware backdoor channel is present anyway because MPPT needs it.
The software can abuse the measurements to listen for optically transmitted commands.
If you understand the principles of control systems and how an electrical grid works, this is broadly "just" a grid stability concern.
To some extent this feels like an issue of IoT-ification of things that we otherwise understood just fine! Maybe the real issue is how we blend cyber security knowledge into other sectors, and quantify and ensure it is present?
It's easy to imagine a scenario where something happens between China and Taiwan, Europe gets involved in a way that majorly pisses off China, and China decides to sabotage Europe's grid in response.
Nothing about that is "total war" with Europe, and it's not like Europe is going to escalate with nukes either because that would be wildly disproportionate.
But it's a major vulnerability that should be fixed as quickly as possible. It's negligent for that to even be an option for China, because it certainly doesn't seem like Europe can do anything similar to the grid in China.
Your idea that security vulnerabilities don't matter, that "that ship has sailed", is false and irresponsible.
No one advocated ignoring the vulnerability. I, myself, specifically stated that monitoring should be direct. Consumers should unilaterally decide where, when and how their assets are monitored.
The material point on security is that there are many, many methods of disrupting a power grid. Even when you are looking for plausible deniability, shutting down solar panels from cloud website doesn't make a list of your top 10 options. (In fact, it won't make the list in those scenarios precisely because you are looking for plausible deniability.)
Let's imagine a power grid as modern societies know them today, except all consumers monitor their solar panels themselves, and none of those consumers outsource this monitoring function to any third party foreign or domestic. Power grids can still be materially disrupted in this scenario. Especially in the case of total war. Obviously in the case of open war. And particularly in the case of cold war.
As I said, I advocate consumers disconnecting any power generation functions from networks. But if I'm in the seat coming up with post conflict, or even simply emergency recovery, operating assumptions, I'm not counting on those panels generating power. It's just irresponsible to do so. In total war EMP will knock most of that generation off line where you're luck enough not to have it eliminated entirely. In cold or open war, disruptions to distribution can and will render that generation useless. (Just ask Ukraine.)
Consumer cloud, or even personal, monitoring of solar panels does not enhance, nor does it degrade, your adversary's ability to disrupt your power grid when your adversary is at that super power level. If you believe it does, you're either not looking at the full spectrum of what you're calling "vulnerabilities" extant in the infrastructure of modern societies. Or you're underestimating the full spectrum of capabilities of modern military powers. Both, frankly, are fatal mistakes in the types of crises we're postulating.
> But the security argument? Yeah, that ship has sailed. Total war, means total war.
Those are your words.
I'm saying, focusing on total war is irresponsible and leads you to draw false conclusions. In the real world, limited conflicts are what we're dealing with 99.9+% of the time, thank goodness.
And now in your new comment, for some reason you're focusing on "plausible deniability" which is another red herring. If China wants to disrupt Europe's grid, it doesn't care about plausible deniability -- the entire point is to publicly retaliatiate. It just needs to do it, as easily as possible. The idea that relying on a cloud vulnerability "doesn't make a list of your top 10 options" doesn't make any sense at all. It might very well be the #1 option, or one of three tactics employed simultaneously.
I get that I'm a tiny minority, and that very few customers want what I want. But A) it seems like giving me what I want should be very cheap (i.e should not entail ongoing customer support costs beyond normal, and in fact would get rid of the small cloud hosting cost) and B) I'd be willing to pay a premium to get it.
I think you're too optimistic about costs though. Providing any support at all, even one-time during the install, is expensive and cloud-free IOT is going to require support due to home networks being broken.
So if my home network breaks, that is not their problem. And they don't need to set it up, they just need to make it possible for me to set up, including figuring out how to make it work with my potentially broken home network. If it requires a new router because mine doesn't provide some necessary functionality? Not their problem. Etc. Etc.
In the end, freedom goes away because we could not be arsed to ask for it at least, let alone fight.
It would be even more laborious and take more time to bring things back online if the attacker manages to damage or destroy equipment with an overload like the GP describes.
Turning everything off when the panels are at peak output? That lets frequency sag enough that plants start tripping offline to protect themselves and the grid and it'll cascade across the continent in just a few minutes. Then you have a black start which might take months.
There's an excellent video on how catastrophic a black start is. https://youtu.be/uOSnQM1Zu4w?si=x0dA7X7-19CJm6Kf
The grid stayed online. Likely thanks to grid batteries, see aforementioned link.
That's a far cry from 10GW.
Differentiating between people like you, who can take blame for misconfiguring device and 99% of other consumers is not viable for most companies. Also, if you bought our device and wanted to do that, I would probably make a firmware version for you that connects to your endpoint and give you some docs. But:
- Just talking and coordinating that possibility for one user would cost my company more than the final price of device, when considering time spent on this.
- You would have to spend a lot of time to implement a lot of functionality to glue our protocol to your desired endpoint.
The video has a lot of good info and seems compelling. During the Texas freeze many power company officials said the exact same thing, if the Texas grid went down it would have taken weeks to bring everything back online.
It's called black start (https://en.wikipedia.org/wiki/Black_start), and power companies plan for it, and the necessary components are regularly tested. It's not a fast process, it can take many hours to bring most of the grid back up. We had last year a large-scale blackout here in Brazil, and a area larger than Texas lost power; most of it was back in less than a day.
> if the Texas grid went down it would have taken weeks to bring everything back online.
The trick word here is "everything". Every time there's a large-scale blackout, there's some small parts of the grid which fail to come back and need repairs. What actually matters is how long it takes for most of the grid to come back online.
With or without cloud based monitoring, our power grids can be disrupted.
That's the commonly accepted meaning of "that ship has sailed" as a colloquialism with respect to cloud based monitoring.
Also, you, yourself, brought up the idea of cold war style confrontation. The basis of most actions against proxy supporters in cold war style conflicts is plausible deniability. It's not a red herring, it's a widely adhered to tenet of cold war style conflict planning when targeting said proxy supporters.
I tried to cover total war, open war, and cold war to address the full spectrum of likely super power on super power active confrontations. In each scenario, the existence, or non-existence, of cloud based monitoring of solar panels, has no effect on the ability or inability of your adversary to disrupt your power grid.
Which disruption was the central thesis of your assertion. I was simply explaining why it was false.
You are being willfully argumentative at this point. If you didn't want to address cold war scenarios, why did you bring them up? You have a nice day sir or ma'am.
No, I didn't.
I think it's clear that China shutting down Europe's power grid would not be a cold war scenario. That would be quite hot. But also clearly not total war either.
Since you don't want to converse any more I won't make any further points, but please don't claim I said things that I clearly didn't.
Here's a recent example where a 512-bit RSA signing key was being used to sign JWTs, allowing a "master" JWT to be signed and minted, giving control of every system on that vendor's control system.
I suspect most grids are extremely easy to hack(never tried, don't bite the hand that feed you etc).
Info sec is just a hobby of mine. I install high voltage switch gear for a living.
I’d expect the opposite. All companies controlling equipment that is part of the “Bulk Electric System” have to be NERC CIP compliant and are audited regularly with large fines for non-compliance. Doesn't guarantee perfect (or even good security) but it’s more likely to be a priority.
I’ve been out of the Power Industry/SCADA game for about 6 years now, and never had huge involvement with solar farms, so please take this with a large grain of salt, but here is my take. 15 years ago, all anyone would say about industrial networks was “air gap!”. Security within SCADA products was designed solely to prevent bad operators from doing bad things. Security on devices was essentially non-existent, and firmware could often be updated via the same connectivity that the SCADA system had to the devices (although SCADA rarely supported this; it was still possible). In addition, SCADA systems completely trusted communication coming back from the devices themselves, making it relatively simple for a rogue device to exploit a buffer overrun in the SCADA. After Stuxnet + a significant push from the US government, SCADA systems moved from “defensive boundary, trust internally” to “zero trust”. However, devices have a long, long service life. Typically they would be deployed and left alone for 10+ years, and generally had little to no security. Security researches left this space alone, because the cost of entry was too high, but anytime they did investigate a device, there were always trivial exploits.
Although SCADA (and other industrial control software), will be run on an isolated network, it will still be bridged in multiple places. This is in order to get data out of the system, but also to get data into the system (via devices, and off-site optimisation software). The other trend that happened over time was to centralize operations in order to have fewer operators controlling multiple sites. That means that compromising one network gives you access to a lot of real world hardware.
Engineers never trusted SCADA (wisely), and all of these systems would be well built with multiple fail-safes and redundancies and so on. However, if I were to be a state-actor, I’d target the SCADA. If you compromise that system, you have direct access to all devices and can potentially modify the firmware to do whatever you want. If there is security, the SCADA will be authorized.
I don’t think the security risks are overblown (they are overblown in what they think the real problems are). I think that as the systems have gotten ever more complex; we have such complicated interdependencies that it is impossible to deterministically analyse them completely. The “Northeast blackout of 2003” (where a SCADA bug lead to a cascade failure), was used as a case-study in this industry for many years, but if anything, I think the potential for intentional destruction is much higher.
Yeah, the potato cannon filled with aluminum chaff or medium caliber semi-automatic rifle can take down a substation. But this is millions of homes and businesses, which can all have an evil firmware that triggers within seconds of each other. (There will inevitably be some internal clocks that are off by days/months/years, so it's not like it will happen without warning, but noticing the warning might be difficult.)
And the growth in sales is exponential!
Technically, anything that can put a hole in an oil-filled transformer. https://en.m.wikipedia.org/wiki/Transformer_types#Liquid-coo...
You don't need to break it... just crack the radiator enough for all the circulating fluid to drain, then it overheats.
https://en.wikipedia.org/wiki/Metcalf_sniper_attack
https://www.npr.org/sections/thetwo-way/2014/02/05/272015606...
Also, the start of an S-curve can be described by an exponential function, right? So it is an exponential.
My point more generally was to not underestimate processes which increase exponentially. Even if they flatten at some point, they can drastically change the world and fast. For example, iPhones and computer chips took off slowly, but once they started moving they took over the world. (Or do you not have multiple smartphones and computer chips in your house right now?)
And yes your point that it's all an s-curve is theoretically correct. But I think it's a semantic discussion. Next time I'll say "never underestimate the first half of an S-curve."
[1]: https://ourworldindata.org/energy-production-consumption
The reason I say "always underestimate" is the obvious one. The easy problems are solved first, then the harder ones take longer because they're harder.
So that's fantastic that battery production scaled up. Only a fool would expect - would plan on - it continuing like that.
Until there's some kind of hold-harmless agreement, the various industry & government security information sharing groups can only be of limited effectiveness.
A fine is a black eye for a utility where people pride themselves on the reliability of the service they provide
Other examples are cruise ship sizes, US GDP per capita, or Microsoft stock price.
The point really is this: go back in time to some of these things a few decades years ago. You would say: “How is this even possible? This is crazy. This will probably plateau soon. This can’t continue.”
But it did. Cruise ships went from 20, to 100, to 200 and now 365 meters in length.
And the same will probably happen for batteries. People say “ah well this is crazy. It will probably plateau soon”. My point is maybe it won’t. Once these exponentials (starts of s-curves) go, they go. Standing at the bottom of an s-curve and predicting the plateau soon can lead to a massive misprediction. Like the IBM CEO who said there will never be a market for more than 10 computers. He was off by about a billion.
I am talking about real things. Cruise ship sizes improved dramatically but... actually linearly? There won't be 1,000,000 GT cruise ships in 2050. Or 60.
You have to use specific measurements like FLOPS/$ to keep moore's law alive, because the focus has been only on a certain kind of FLOP (the fp32 MAC for graphics, or perhaps the INT8/FP8 in recent years). Because in general, it's dead. It's more performance, for more money. Because lithography is really hard, harder now than ever (and water is wet).
Sure the repair is easier/quicker but the economic damage was already done...
I’m going to assume this kind of thing is likely covered in FMEA and such, so is unlikely.
