LogJ4 Security Inquiry – Response Required(daniel.haxx.se) |
LogJ4 Security Inquiry – Response Required(daniel.haxx.se) |
I think support is probably the best way of making money from open source, but a lot of maintainers are unlikely to have everything set up to do so (business entities, contracts, ways to receive payment, probably a dozen other things that you'd never think of, etc.).
Like Stripe Atlas for open source consulting?
> In my tweet and here in my blog post I redact the name of the company. I most probably have the right to tell you who they are, but I still prefer to not. (Especially if I manage to land a profitable business contract with them.)
If he wasn't trying to land a contract, then he would have posted the company name.
To actually make a difference when you have a platform, use it. Tweet-shame them so that the fallout actually reaches the manager in question. This is just complaining about a behahavior while at the same time more or less doing everything possible to encourage that behavior.
> In my tweet and here in my blog post I redact the name of the company. I most probably have the right to tell you who they are, but I still prefer to not. (Especially if I manage to land a profitable business contract with them.)
It's explained in the article.
This made my day. If a wealthy individual takes your tools and then calls for help while fixing-up their shed with said tools, do not move a muscle until you agree on the fee.
As a commercial SaaS vendor we received these same emails from all of the major banks / insurance companies. It's interesting to see that we got some on the Monday after the issue was discovered and some a few weeks later, with some showing a clear understanding of the risk in the context of our product and some looking like a standard copy/paste. Gives you a rare behind-the-scenes view of the information security practices of these companies.
1. Someone decides that we need inventory of all the libraries used (iirc requirement for some certifications and generally not a bad practice)
2. A system (/excel sheet) is enrolled where you have fields like $our_product, $library_used, $vendor_email
3. A dev, not quite understanding the point, dutifully fills in the data for the project they are working on
4. No-one reviews the data
5. Crisis strikes, so mass-send email to all vendors how they are handling it
Problem here is around point 4.; for the process to work, someone should have reviewed the data to check that the used libraries are from vendors with some sort of support arrangement.
I think the reply they provided is pretty promising, it makes it sound like they wanted to be a customer but are not only due an oversight.
There is a large time gap between 4 and 5 - and it seems everyone forgets who they hired for that supply chain "analysis" many moons ago.
This is clearly a scatter-gun survey because they're realised they really have no idea of their exposure. (And before you re-boggle at that, there's a whole business ecosystem in just being able to answer that question let alone do anything about security issues.)
This is the best kind of sales call: they are coming to you.
> ...because they're realised they really have no idea of their exposure.
This is partially because it is often non-engineers being asked to figure this out. The "information security analysts" at F500s are asked to do a lot of unfair work, such as analyze risks related to decades-old software they didn't build.
> ...there's a whole business ecosystem in just being able to answer that question let alone do anything about security issues.
The first part (answering "what dependencies does my software have") isn't inherently bad. I'd emphasize the underinvestment in the second part more.
I think that's putting it mildly. When it comes to responding, they'll look around and find that they only have a small number of full-time employees with the skills to partake in a response. Most of the IT organization will be dependent on vendors who struggle during the best times while their leadership has the ear of the CIO because IT is only viewed as cost.
The full-time employees will frequently be the real heroes, but when the incident passes this won't be recognized. Things will repeat themselves with the next major vulnerability discovered, but the organization may find that they have even fewer employees at that point to lead a response.
Yeah, well, I've been quite shocked how rookie some F500 devs can be and how dysfunctional large corporations can also be. Probably what happened here is someone wrote a script that compiled the dependencies of all projects they have and they sent this same email to all of them (!) regardless of any actual or potential use of log4j.
https://www.ign.com/articles/2019/03/26/man-steals-122-milli...
The tone feels off if you assume a human wrote it. But that's only because it's a form letter their legal department wrote for them to send off. They probably collected "dependencies" from the entire company (and someone wrote "curl"), and sent a mass email.
If you just reply with a simple "We're unaffected!" (or ignore them), you'll never hear from them again.
Of course it's also plausible that that's not fraud at all. But I have no way to know for sure unless I ask a lawyer, which needless to say I wouldn't do. And if it turns out that it is fraud, well, the legal department of Fortune 500 companies tends to be pretty humorless.
I am personally a bit surprised about the responses here. It is completely reasonable for this email to reach Daniel and is most likely an artifact of bad documentation by engineers in the company. At the scale this company is running the person/team sending out these emails do not have time to dig in and understand each dependency they are sending emails on.
The response is as simple as "What library/product does this email pertain to?", "Please see the licenses for the libraries or products in question.", and what Daniel responded with as well: "I would be willing the dig in further for specific questions with a support contract.".
That alone is extremely disrespectful, it means they couldn't care less about the time of open source software maintainers. To say nothing of their "request" for review.
The real "disrespect" should be whatever engineer put Daniel's name into the spreadsheet that blasted out these emails. Someone didn't do their job and is checking a box. How is the (possibly non-technical) person that is required for managing 100s of vendors and thousands of open source libraries supposed to verify all of that information?
I'm personally happy to hear that this company is trying to do SOMETHING to make sure that Log4j is patched even if it's a bit incompetent in it's implementation. There is not malice here.
I assume some developer/supplier used curl and provided a list of third party code and licenses they use.
In the aftermath of the log4j incident, companies now target everyone about this issue partly to learn about potential exposure that they are not aware yet, eg exploited infrastructure of depending services like newsletter or analytics services.
Yes, it's annoying and pointless to spam this mails to open source projects. But at least someone is now behind auditing the supply chain.
But obviously, it's not a sound approach to actual vulnerability management.
Quite well handled, not arrogant, not bending over and doing whatever they say, but being honest.
If curl is impacted or not, may not really matters for them, usually these companies go after compliance and someone who they can blame when things go wrong.
"We are happy to provide you with support regarding this issue for $5000/day"
Then if they accept, proceed to do nothing for 10 days, then reply you find none of your code is impacted and they are safe then bill them $50k.
Isn't this the sort of question you'd ask your own side, first?
The company I work for is not Fortune 500, but we have several Fortune 500 customers. The amount of inane bullshit we have to deal with as a result is mind-boggling.
Do we mock every single open source guy who displays the same amount of cluelessness about the inner workings of a business because I see plenty of that displayed here and everywhere else.
Anyone leading a shareholder action would love to see these emails. They are basic admissions that the company doesn't know how or from where it gets essential software.
>> Thank you for your reply. Are you saying that we are not a customer of your organization?
It's just so beautifully orthogonal. Oh, and they got his name wrong in the salutation.
LOL.
Based on the 2019 Fortune 500 list, that gives these possible candidates: Activision, Alaska Air, Albertsons, Altice USA, Amazon.com, Ameriprise, AutoNation, BB&T Corp., Bed Bath &, Blackstone, Booz Allen, BorgWarner, Burlington, CBRE Group, Chesapeake, CMS Energy, CVS Health, Dean Foods, DTE Energy, Enterprise, Eversource, Expeditors, Fannie Mae, First Data, Ford Motor, Home Depot, Huntington, JM Smucker, Jones Lang, Laboratory, Mastercard, McDonald's, Murphy USA, Nationwide, News Corp., NGL Energy, NRG Energy, Occidental, PBF Energy, Prudential, PulteGroup, S&P Global, State Farm, Unum Group, US Bancorp, WEC Energy, Windstream, World Fuel, WR Berkley, Yum Brands
The "or maybe they have customers who do" makes me think that this company must provide services to other companies, so probably not a Mcdonald's or Albertson's or something like that.
Wasn't Java's SecurityManager stuff supposed to prevent these kinds of exploits?
I haven't used log4j for ages, so I didn't know offhand. Somewhat curious, I gleened that none of the enterprisey stacks use SecurityManager. I guess I kinda understand; SecurityManager was fashioned and pitched for an ecosystem of applets, agents, and sandboxes.
Further, I then gleened there's a JSR to outright remove SecurityManager. With no apparent replacement, just some vague advice to roll your own capabilities based system.
So, however we got here, what's then plan? Run JVMs on top of something like OpenBSD's pledge?
That would be fraud. No, start grep on the source code and a few things like that, then provide the results: "a detailed audit found no reference to log4js, so another audit was started which found no reference to any java code in the C source; it was repeated 5 times to confirm these promising results. Another audit followed the Boltzman brain hypothesis to check if the affected log4js binary code could not be spontaneously generated during compilation, by following a Monte Carlo simulation to check for various length of binary data that would match the log4j binary code. (...)
Finally, to avoid this extremely remote risk, the code changed to switch to reproducible builts, which can guarantee this will not happen"
Or print it out on hard copy, make interns read it line by line, then charge 400% of their labor as your management fee.
What's the purpose of using regexps here? You're optimizing away your own revenue!
1. Insist that you need to talk to upper management until you get to the CEO.
2. Once there you need to sell them on a Fixed fee contract for five engineers so let’s say $1MM or more
3. Actually create a few scripts that run the log4j scanner from Google.
4. Have an extended support contract by doing this yearly at $1MM.
Fixed fee or monthly "support contract", with minimum of 1year.
Hopefully you don't do that or encourage others to. Just because F500 companies are big, stupid, slow and greedy, doesn't exactly make stealing right.
That is precisely why it's right. These capitalists have stolen our labour, and corrupted our politics for centuries. `Stealing` it BACK is the ONLY way history has shown us works.
You are all are supposed to be smart software engineers. Probably know about pre-mature optimization and efficient path.
Here's a secret about communications -- Mass emailing works and is very efficient.
I'm sure you are the same person who rants about a recruiter reaching out to you even though you are the creator of Python.
Reading through everyone's resume and tailoring a message is a waste of time and has the worst ROI for any salesperson.
"But Ha Ha Ha, you guys are clueless about not knowing operational efficiency of an mass communications. Ha Ha Ha"
Yeah, that's exactly how this sounds if the other side mocks HN/Engineers the same way you mock Sales and other "mass-outreach programs"
It's as dumb as mocking a scam email/phone call telling "You are so wrong about me". The end goal of the scammer is to make money for the total time he put. Sure, the scammer can go in great detail about your life and tailor the scamming for you, but that's not his best ROI. His best ROI is a generic message sent to everyone.
Oh and "Ha Ha Ha, that you don't know that"
And "Ha Ha Ha You, for not knowing that"
I also bet that the list of dependencies they used for this mass email was probably not generated by a lawyer.
But in software communities (and particularly in FOSS communities) tech people can do no wrong; every time a tech company does an aggressive or foolish or otherwise objectionable thing, there must be a dastardly lawyer somewhere pulling the strings.
It's as dumb as mocking scammers for their methods. They are effective in their own way. Just because it didn't apply for you, doesn't mean they aren't making money out of this -- which is their ultimate goal. Their goal is not to satisfy your ego and custom tailor a message to you
Creating a software bill of materials is a technical task. Managing software security risk is a technical task. These need to be performed by a technically literate person.
A Fortune-500 company has the resources to pay for such technical competence. They are not a mom-and-pop shop.
No Fortune-500 CEO would get their teeth done by a fly-by-night "dentist", nor would they hire "builders" who can't nail two planks together. They would pay for the expertise. If they don't know how to find the expert they would pay for the expertise of finding the expert first and then they would pay for the expertise.
But this is not what they did. They found someone who is both lacking the necessary technical common sense and is terribly arrogant. That is worthy of ridicule. And I'm not ridiculing the individual employee but the whole company.
> The premise that when the less educated and informed try to question something they don't understand only to be left with pandering and jabs is disingenuous.
That idea flies when a student is lost in the woods. When an economic juggernaut combines technical illiteracy with a lack of tack they can get the sharp ends of our tongues.
> The entire reason your salary is much larger than many other career paths is because of your ability to deal with technology.
Won't be for long if we silently support huge companies to employ muppets. Which is why asking for a support contract is the right answer here.
You can feel sorry for the poor sap that was forced to embarrass himself, but it doesn't change the fact that everyone here feels like that company can get bent.
Do you really think the security department in this specific company would not find this email dumb? In many cases, when things are reacted to hastily and in parallel its easy to take one action and generalize it to the whole company and not realize this is one of many actions the company took. No need to get bent out of shape over this and say this entire fortune 500 company is equally incompetent. If you think that you are not living in reality.
I'm willing to guess this happens a lot more than people realize. I doubt we were the only people joking about it. People joke, other people hear, some of those follow up with action. The smart ones keep quiet and stop well before getting to $122M.
Especially for the sender.
Departments (small managers) are authorized to spend small money without approval, lets say up to 200 euro/month. If they send this type of emails, someone ass is on fire. They will DO spend it just to get legal green light.
Anyway, I do not see reason to hold back, just because I am open source developer.
This is basic marketing. Airbnb, Facebook, Amazon etc are allowed to do shady stuff, but single contractor should be clean as lilium?
Agree 100%. Any engineer that got far enough to put this email address into a spreadsheet knew damn well it was inappropriate to do so. They should have put their own email address, as they made the choice to use downloaded software in their project and become responsible for that decision.
So, in your opinion, sending spam or robocalls is not in the least bit disrespectful to whomever is on the receiving end?
As I noted in another thread: A simple response pointing to the license should be the default response to requests like this which can be automated.
That's how a log4j security audit becomes an Oracle licensing debacle.
Namely: THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.