The company claims that they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the medical unit's side."

Here's how I read that: The programmers of this piece of software assumed that some I/O operation would never fail and when it does the program shits itself. So instead of hardening their software to withstand loss of telemetry gracefully, which would cost time and money for the company, they just give instructions to disable scans on their folder.

Odds are good that somewhere this scan will happen (and it did). Either IT doesn't read the release notes or goofs the configuration or an antivirus update clears the white list. Might not even be the antivirus that interferes with the telemetry briefly.

But instead of having resilient software it's "the anitvirus software's fault" or "it's IT's fault" when something goes wrong because of their bad management/engineering decision.

I can't tell you how many times we've chased down field problems that ultimately were the result of antivirus scans. It's been so bad, that one of the first questions we now ask when we get a tool-down report is "is there antivirus running and what is the configuration?"

Bringing Windows into the architecture of any type of capital equipment control system is a bane. A scourge. I mean to say, it really is a misappropriation of software. Imagine, "Yeah, Frank only knows VB, so that's what we used for the aircraft's cockpit GUI."

/xrant/

The whole structure is wrong. I used to work in medical equipment repair. Windows Embedded is running so many devices it's not funny. But it's not just Windows that's the problem.

I put a linux-system on a PACS network to diagnose equipment. It was a headless, and we asked the IT group to block it off from the Internet.

Hospital IT: "Does it have antivirus?"

Me: "..."

Back in the nineties, I wrote a nice piece of some 300kb of C code, for DOS/x86. It was a complete software package, controlling medical equipment that was testing speed of blood coagulation. These tests are crucial in the patient post-operation recovery.

This piece of C code had some hardware control code, some statistics, a bit of math, some visualisation, GUI, etc. Normally, you'd imagine a team of 2-3 people, carefully written test cases, dedicated QA person, and a year of time to write something like it. And independend lab, that would certify the thing. Well... in that case, yes, there was independent certification... but...

It was just one developer, and I was 13, when I wrote it ;) During after-school time, in around 4-6 months. And I must say, I still sometimes have chills, when I think of the code quality, and, um, unorthodox solutions of 13-year-old myself. Yes, I've had some years of experience at the time, both writing software and designing hardware, and advice from my parents, who both could write software. But, at the time, I've had zero formal training, aside from reading K&R and PC XT manuals ;). So, you might imagine the code quality ;) Even, no need to imagine, I actually still have it somewhere in the archives :)

[1] http://sunnyday.mit.edu/papers/therac.pdf

I've seen this happen time and again, where companies have some 3rd party service vendors who would install AV software on anything they can get their hand on, even a microwave or coffee machine - just to tell the client "my bill is expensive, but you can feel secure, we installed AV". I despise these folks with a passion.

The problem is not Windows. It's a lack of knowledge and understanding. Simple.

For god's sake - it's 2016 - dump the Anti Virus software. I am gonna make t-shirts this summer with this ;)

But it rarely is useful. It only causes problems. We've seen so many issues related to virus scans throughout the years it's crazy.

What's better is to lock down the servers with only minimal access. I haven't used virus scan on my main desktop for over 10 years because I don't click on weird emails and I don't go to sketchy websites ever. Sure there's the risk of malware from ads I suppose, but I'm not that worried.

>> The company claims that they included proper instructions in their documentation, advising companies to whitelist Merge Hemo's folders in order to prevent crashes from happening, so it seems that the whole incident was nothing more than an oversight on the medical unit's side.

So "RTFM"? Not very helpful.

And the hospital included full instructions to the software company on how to properly perform a heart transplant, so they were baffled why the programmer just let his teammate die of heart failure.

Come on, this kind of stuff should be a zero-configuration hardware-based black box, with its own buttons, screen, etc. --- not something that needs to be (or even can be) connected to something outside the vendor's total control.

Medical equipment require an authorization to use. Any change to the medical equipment requires another authorization or it's prohibited.

By "any change" , it includes Windows Update(it changes the system obviously).

The result: they use anti-malware software to protect(or rather, believed to protect) unpatched Windows.

At least one anti-malware software company(Trend Micro), marketing that their software can protect the medical equipment in such situation.

That the equipment is somehow configured to be susceptible to viruses.

Based upon the available information, the cause for the reported event was due to the customer not following instructions concerning the installation of anti-virus software; therefore, there is no indication that the reported event was related to product malfunction or defect

I beg to differ. I'd consider a momentary loss of file I/O due to lock contention causing a machine to require a reboot a shocking defect in - say - a word processor (which, notably, do not have this problem). That this risk is apparently known and the vendor's sole mitigation is to document a 'Don't do that then' is absolutely 100% an indication of a product defect, even in the absence of an actual occurrence.

just perfect

It took us a while to isolate Kaspersky 10, and it's not even any particular component inside of Kaspersky, but only when all features are enabled. We tried different permutations of features to try and isolate the cause of our crashes, but as soon as you have any one feature disabled in, the crashes stop, Very frustrating because ultimately our clients laid the blame at my feet (new software feature, new release, blah blah blah), and not exactly much you can do in the way of hardening against this particular crash, the app generates a burst of network data, and boom, blue screen/instant reboot.

There are kinda four flavors of machine setup I ran into while in that field: big server banks for on-site hosting (think huge enterprise VM farms, for data warehousing and record storage and virtual desktop hosting), care provider systems (think like tablets, doctor office computers, nurse workstations, room workstations), cart computers (used for things like running the sonogram or cardiogram equipment, or for other studies), and actual integrated devices (for, say, data collection).

The care provider systems are usually comically locked-down, tablets and phones having the meanest management software they can (no apps, limited connectivity, remote wiping, and so forth). Workstations tend to be centrally managed, have images pushed regularly (ha!), and often use AD and smartcards to handle authentication. One place I've seen took this a step further, and basically just booted users directly into a VM hosted on the server farms mentioned earlier. You can't use USB devices, you have highly-regulated clipboard access, and so forth--this is done to prevent HIPAA breaches. Which is kinda silly given other workarounds, but whatever makes people feel safe and the CIO happy. These workstations run some enterprise version of Windows, probably 7 Pro. Those silly-long extended service agreements you see on Microsoft? Hospitals are some of the people keeping that alive, and they will pay obnoxious amounts of money for the privilege.

The cart computers are typically like the workstations in terms of functionality, but they may have software specific to the device they're talking to. They might not be as locked down (e.g., only acting as thin clients to a remote VM), but they are still running Windows.

The device computers may run some kind of RTOS. In some cases, they'll be running a customized Windows CE installation--which is totally reasonable. There are a lot of good guarantees that that can give a development shop, least of all that they can call up Microsoft instead of StackOverflow and say "Hey, this function does x, it's documented as y, and we're paying you a lot of money, so what the fuck?". Windows Embedded (which is I think the successor, am not sure).

In all of these cases, Windows itself works pretty damned well.

It runs the software everybody needs, it has the enterprise deployment stuff figured out through decades of improvement, and really there is no reason to be scoffing at its choice.

Now, if folks have goofed up and thrown a stupid AV policy on the machine, that's a different question entirely. Health IT is full to the brim of people basically just punching a clock and being unable to get anything done in a reasonable amount of time. Sometimes, they do awesome things, but mainly they are just custodians standing between doctors and really really stupid policy decisions that seemed good at the time.

EDIT: Removed unrelated example at top.

"The antivirus was configured to scan for viruses every hour, and the scan started right in the middle of the procedure."

Who configures an antiviurs for an hourly scan on a doctor's computer?

Don't tell me that PC is connected to the internet...

That should be untimely. The opposite of timely.

http://www.merge.com/News/Article.aspx?ItemID=660

Welcome to the Health Cloud Powered by Watson.

Something is probably missing from the article. IMO, the device in question wasn't critical at all, and a failure could be expected.

It Does Not Matter if the device is connected to/able to reach the Internet.

First, it probably can reach the Internet in some way simply by being networked. I don't think I've ever seen a medical office (can't speak about hospitals) where medical diagnostic equipment was on a fully-separate network able only to talk to other network equipment and specified data destinations (PACS servers).

Second, I'm not concerned about unpatched, unprotected machines being infected from the Internet. Odds are they're running a restricted version of Windows, with a custom shell and a lot of stuff stripped out. I'm concerned that they're going to be infected by another machine on the network that's gotten infected. With all the past SQL Server security issues a decade or more ago, how many people think those SQL Server boxes could be directly reached from outside the local network?

The conjunction of those two is that even if you firewall all that stuff off, the PACS servers are still on both networks, and are probably running much more interesting and vulnerable stuff than the device controllers.

Sure you can fully wall everything off - it's really easy, just do your X-rays onto film, burn your MRIs and ultrasounds onto CDs, and print your EKGs for later scanning. Oh, and listen to people complain about how out-of-date your systems and procedures are.

There are other factors that come in as well - sure, every device manufacturer could provide fully bespoke diagnostic displays developed from the ground up in artisanal software shops providing full employment for assembly programmers working on embedded systems, along with cohorts of graphic designers creating glorious steampunk-styled interfaces. That's a beautiful dream, keep having it.

For the rest of the world, creating a UI on that custom embedded system running on something from RIM/Blackberry (yeah, they own QNX) is just going to get them crap from people because of A) how clunky it probably looks and B) How could they even consider allowing direct user interaction with the RTOS that was chosen to ensure that the dangerous bits in contact with patients/radiation/irradiated patients were safe?

There's a beautiful world out there somewhere where everything is safe and secure and seamless and updated. The rest of us live in worlds where Joe in Marketing's PC gets infected with something that allows an attacker to start scanning the network for unpatched vulnerabilities on any system, which leads to an out-of-date install of IIS on a legacy server that hasn't been updated because there's no longer a contract with the vendor (or no vendor) but it's around because there's a statutory requirement to keep the data on that system for 7-10 years.

There's a lot of ugliness out there. Antivirus is a way to try to ensure that when (not if) some of it hits you the repercussions are minimized.

This machines costs hundreds of thousands of dollars.

There should be no excuse for using Windows. None.

I would not be surprised if the "antivirus" thing was some PHB requirement

half the work done in this industry is just dealing with stupid decisions made by stupid people. i just accept this now.

We need to go much further than this, since many people will "solve" this problem by using a different platform than Windows.

In reality, bringing networking of any type into capital equipment control systems or critical infrastructure is the bane ... the scourge.

Whatever convenience or perceived function that networking (including very local networking, like USB) is dramatically outweighed by the additional attack surface.

Go back to sneakernet and check your facebook at home, Mr. Nuclear Plant Worker.

That is simply untrue, you can (and in many cases should) have unroutable subnets. But even if true, that only slightly changes the question: why is operating room equipment networked in the first place? That you've never encountered a proper setup doesn't excuse not having it.

* GUI dev tools were an order of magnitude better than under linux

* nidaq (very common daq and dio cards) didn't have or had flaky linux drivers. usb may be a better choice today but was not then available. Plus it's convenient/less expensive to push functionality to the computer side and minimize embedded development.

* in the beginning (though we fixed this mistake), we shipped software, our machine, and a ni-daq card; users provided a computer. This was a nightmare. We ended up shipping whole computers custom built by Dell to our specs with ni-daq cards installed (and later glued in to prevent them slipping out during shipping) plus our custom system images. At the time, I don't think Dell would do this for linux; you could find other vendors that would, but they weren't easy to work with or much more expensive.

* Nobody is eager to switch OSes because every software update to the machine has to be FDA certified (ie kiss at least $100k goodbye), so the company is very risk averse. Beyond even the cost to rewrite what has evolved into a relatively complicated gui app.

* As mentioned elsewhere, once you leave windows you have to train people to use linux guis that historically had window managers / gui toolkits that didn't work like windows.

All of the above is complicated by the fact most of these medical device companies pay their devs shit and have trouble hiring good devs. (Engineering shortage, obviously!)

I wish I was kidding.

Moreover, this is the kind of thing that should come up in robustness testing. Things should get bumped and wiggled. They should get unplugged and turned off. If the software is really going to run on random Windows boxes, then it should be tested on random Windows boxes. (At which point somebody will hopefully say, "Wow, this sucks, let's make it an appliance.")

No matter what happens, it shouldn't result in a "mysterious crash right in the middle of a heart procedure when the screen went black and doctors had to reboot their computer".

Please consider dropping the adversarial attitude. This place isn't like other sites. The way people converse is equally important to what they say. It's better to transcend than to dominate.

For example, we do not slip in underhanded comments like this:

> In that scenario, your surgeon would see the program suddenly freeze.

Only if the programmer or his management were incompetent

This is just short of a personal attack, which is against the rules. I know you probably didn't mean it that way, but look at how you're framing the debate. I felt as if I'd been teleported onto Fox News and forced to defend myself from an aggressive interviewer's mischaracterizations.

Now, you can take the stance that "It's not against the rules, so I can say whatever I want." That's true, you can. But we're worse off for it. We optimize for good conversation here.

The point I'm trying to get across is that if you really throw yourself into this community, wholeheartedly and without a feeling of having to prove something wherever you go, then this place has a lot to offer. You'll meet a lot of interesting people, you'll hear a lot of interesting stories, and perhaps you'll have an opportunity to contribute to something quite unexpected. But none of that will happen if you try to skewer your opponents wherever you go -- or if you see people here as opponents. We're people.

It doesn't matter what the conversation is. It doesn't matter whether it's about life-or-death, or that this one happened to be about a surgery. The goal is to put yourself in the other person's shoes and to ask yourself, "If I were them, why would I say that?"

Regarding our conversation, if you want to continue it, I'd be happy to. But unless you're trying to learn as much from me as I'm trying to learn from you, it's not going to go anywhere productive. And what would be the point? No one's looking anymore -- it's fallen off the front page, so it's just you and me here. But why should our conversation be so different just because nobody is watching?

There are things to be said, but I have no time to defend myself. You can characterize what I was saying however you want. Or, alternatively, you could ask me what I meant.

I won't pull one of those "I've been in the field for a pretty long time, so I bet you'll learn something..." routines. Those are tired refrains, usually coming from people who have long forgotten what it's like to be young and hungry. But I'm still pretty young, and money's low enough that I'm pretty hungry. Being unable to afford meat is unfortunate, but it's worth not having a job for a little while to throw myself into my research. See why there's no time to defend against aggression?

I think I wrote this because in many ways, you remind me of how I used to be. And if I could go back in time, I'd ask myself what I was doing and why. This type of discourse is an intellectual dead-end. No one is going to learn a thing from watching people try to tear each other apart. Maybe you didn't realize that's what you were doing. It's very easy to slip into that mindset without realizing it.

As for displaying random data, why would the programmer want to do this?

GPUs are bastards. They ignore what programmers want, almost by definition. And as someone who has spent way-too-many years wringing as much performance as possible from them, I assure you that this is a realistic characterization of a possible outcome.

Perhaps that piques your curiosity. If so, then that sounds like the start of a good conversation, no?

We've been close to undergoing "major" scrutiny (as it was sold to me, it was A Big Deal) from the FDA before. I, personally, just a lowly and underpaid engineer, have saved executive staff from having to sign their names on that noose. I had a manager once who seemed to want to push it that far, to stand idle-by while the walls fell down around us. I, unknowingly at the time, prevented it from happening because I was trying to help our customers. I don't regret that decision, actual patients shouldn't have to suffer because of a management teams ineptitude. I do think about it often, though. I understand this is nebulous, and I'm sorry for that. This is a reality, though.

I guess that's the thing that really gets me, the FDA. We sell FDA approved devices. Where the fuck is the FDA? We send them paperwork and they are happy. I can only form the opinion that they, the FDA, are ill prepared to handle this situation; The actual situation, the "the medical devices industry is a fucking train wreck waiting to happen" situation, and especially so they are ill prepared to handle it at scale. Audits are cursory and almost as a rule non-technical. I suppose it'll take a Toyota-level incident to bring change about.

Or developers refuse to build software without safety built in.

If they can't hire anyone to build their unsafe systems, they'll have to start building safe software.

Let the market work for you.

A fine being tax deductible does not mean zero cost to the company, it means the profit is reduced before taxes are computed, i.e. the actual cost is reduced by the marginal tax rate. A tax credit means zero cost.

This is not meant as a slight: I think you're grossly unfamiliar with software development outside of engineering-driven companies.

It's pretty much a guarantee that product managers are deciding these estimates. They might confirm with the developers, but the conversation probably went something like this:

"Does 3 weeks sound about right for this?"

"No, we'll need 6"

"Why?"

"Safety checks"

"Ok, we don't have 6 weeks. I can give you 4, but we're just gonna have to make do."

Is it scary that conversation happened about a piece of medical software? Absolutely. Would I bet $1k that it happens frequently? Absolutely.

> A reputable engineer wouldn't design and build a bridge which might collapse

Rarely does a single engineer design a bridge nowadays, so corporate liability and reputation (good luck landing more contracts if your bridge collapses) is a huge factor in much of that beyond simple ethics.

I would be shocked if anything happened to Merge as a result of this, whereas a company who designed a faulty bridge would be sued into oblivion.

Further, professional engineering in the US is a whole different game that involves licensing and regulations specifically to avoid that situation. Software "engineering" has no such equivalent currently.

Pinning the blame on the peons is a sure-fire way to make sure this situation never changes.