GitHub waited 3 months to notify about potential compromise

zwass · 2022-06-16T18:47:20.000Z

It seems GitHub became aware of the issue around March 2 (or before, since the fix was released on March 2), and waited until June 16 to disclose the problem.See full text below:Hi zwass,We're writing to let you know that between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC, due to a bug, GitHub Apps were able to generate new scoped installation tokens with elevated permissions. You are an owner of an organization on GitHub with GitHub Apps installed that generated at least one new token during this time period. While we do not have evidence that this bug was maliciously exploited, with our available data, we are not able to determine if a token was generated with elevated permissions.User privacy and security are essential for maintaining trust, and we want to remain as transparent as possible about events like these. GitHub itself did not experience a compromise or data breach that either created or resulted from this event. Read on for more information.* What happened? *GitHub learned via a customer support ticket that GitHub Apps were able to generate scoped installation tokens with elevated permissions. Each of these tokens are valid for up to 1 hour.GitHub quickly fixed the issue and established that this bug was recently introduced, existing for approximately 5 days between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC.GitHub Apps generate scoped installation tokens based on the scopes and permissions granted when the GitHub App is installed into a user account or organization. For example, if a GitHub App requests and is granted read permission to issues during installation, the scoped installation token the App generates to function would have `issues:read` permission.This bug would potentially allow the above installation to generate a token with `issues:write`, an elevation of permission from the granted `issues:read` permission. The bug did not allow a GitHub App to generate a token with additional scopes that were not already granted, such as `discussions:read` in the above example. The bug did not allow a GitHub App to access any repositories that the App did not already have access to.In order to exploit this bug, the GitHub App author would need to modify their app's code to request elevated permissions on generated tokens.* What information was involved? *The following GitHub Apps generated scoped installation tokens during the bug window for your organization(s). We are not able to determine if a token was generated with elevated permissions.Organization: GitHub Apps <redacted>* What GitHub is doing *GitHub immediately began working to fix the bug and started an investigation into the potential impact. However due to the scale and complexity of GitHub Apps and their short-lived tokens, we were unable to determine whether this bug was ever exploited.We are notifying all organization and user account owners that had GitHub Apps installed and had a scoped installation token generated during the bug window so that they can stay informed and perform their own audit of their installed GitHub Apps.As a followup to this investigation, GitHub is looking at ways to improve our logging to enable more in-depth analysis on scoped token generation and GitHub App permissions in the future.* What was the potential impact? *Due to the variety of GitHub Apps, their possible scopes, and the repositories they may have been given access to, we are unable to advise on any potential impacts as each customer's situation will be unique.* What you can do *While we have updated our systems to resolve this bug and no action is required on your end, we do recommend you review your installed GitHub Apps. You can use the following guidance for assessing GitHub Apps, their permissions, and their access to your private organization repositories:https://docs.github.com/en/organizations/keeping-your-organization-secure

cypherg 3 years ago |

They explained it themselves - there's no evidence of abuse/exploitation. They literally have no legal requirement to even tell you as much as they did. You should be commending them for filling you in at all.

chrisseaton 3 years ago | |

> They literally have no legal requirement to even tell you as much as they did.

Is ‘fulfilling legal requirements’ all you look for in a business relationship?

A restaurant has no legal requirement to make this food tasty but it’s what I’m looking for when choosing where to go.

PradeetPatel 3 years ago | | |

As someone who works in the reputation management sector, fulling legal requirements is crucial in establishing a presence in key markets. However, oversharing of internal information that's not required by legal requirements can lead to unnecessary reputation damage, which would lead to a decrease in value for key stakeholders.

I think many engineers often overlook the business implication of disclosing security issues, as it would impact multiple business units as well as the board's stance on security, resource allocation, and potentially the stock price too.

>A restaurant has no legal requirement to make this food tasty Food is a core deliverable for a restaurant, whereas information on a potential breach is not for a SaaS service unless it is legally required.

akagusu 3 years ago | | |

Since almost every popular tech company is a quasi monopoly, they use this "fulfilling legal requirements" strategy to abuse the market providing overpriced services with bad quality.

Unfortunately, people got used to this practice and gladly accept when such companies fulfill all their legal obligations, even when this hurt them or their business.

Doxin 3 years ago | | |

> A restaurant has no legal requirement to make this food tasty

Somewhat tangential but I'm not even sure that's entirely true. It gets all sorts of tricky due to the subjectivity, but surely fit-for-purpose laws apply here? I'd be really surprised if a five star restaurant selling $500 tasteless gruel with chunks wouldn't manage to get into trouble if they refused refunds.

dogecoinbase 3 years ago | |

> ... we were unable to determine whether this bug was ever exploited.

> ...

> Due to the variety of GitHub Apps, their possible scopes, and the repositories they may have been given access to, we are unable to advise on any potential impacts as each customer's situation will be unique.

Absence of evidence is not evidence of absence.

hanble 3 years ago | | |

That's true, but feels like these are always judgment calls. We can always armchair quarterback their judgment calls, but none of us have the full info. At least GH is sharing this info, which is a good call for trust building IMO.

zwass 3 years ago | |

I would be commending them if this notice went out March 3 after they had remediated the problem and were aware that they had no logs to determine whether there was abuse.

iepathos 3 years ago | |

This is incorrect you should re-read the post here cause I think you misunderstood the implication. They lacked the logging at the time to know what apps were impacted and the extent to which customers were compromised by this. They are legally obligated to disclose security risks like this which is why they did. You should consider setting a higher bar for your commendations.

dontbenebby 3 years ago | |

>They explained it themselves - there's no evidence of abuse/exploitation. They literally have no legal requirement to even tell you as much as they did. You should be commending them for filling you in at all.

Do they keep logs, or is that also not a legal requirement? (See how these things can combine?)

Folks literally should be mindful no oneis obligated to do business with you if they think you're untrustworthy.

philwelch 3 years ago | |

They’re obviously capable of doing more, and they’re in a competitive market, so I wholeheartedly support the freedom of dissatisfied customers to publicly shame them. Either they will learn from this and do better in future, or the community will enrich their competitors at Microsoft’s expense. And that is how we can have nice things rather than the mere minimum that our corporate overlords are “legally required” to provide us with!

_7gt4 3 years ago |

My recent experience with GitHub regarding a security issue was not very positive either.[1] It turned out, unlike two vendors I notified that were affected, they just didn't care. And they didn't bother to even tell me that they didn't care.

It's a very edge-case issue in Enterprise SSO, so I wasn't really able to generate any blowback with disclosure either. But if you find an org with just the right setup it blows a huge hole into the SSO product, to the point of making it useless.

There also seems to be an asymmetry between the core product and everything else. GitHub Enterprise has issues that aren't even considered UX issues (i.e. notifications showing "3 of 0" notifications if no SAML session exists) that'd warrant bounties if they were in the core product.

[1]: https://notes.acuteaura.net/posts/github-enterprise-security...

captn3m0 3 years ago | |

How is tailscale mitigating this? They can’t enforce GitHub SAML at their end, right?

_7gt4 3 years ago | | |

The membership API returns a 403 if no SAML session exists. Check the remedy section of the post.

This was an accidental find and GitHub has refused to document it.

throwaway78246 3 years ago | |

My guess is that GitHub ignores issues so they don't have to pay out bug bounties.

onphonenow 3 years ago | | |

I find these takes so silly. Bug bunties are a rounding error in the companies budgets, even if they paid out much more freely. There are many I think much more obvious reasons orgs are slow on issues - everything from figuring what is an issue, trying to chase down impacts and more.

lambada 3 years ago |

Given it existed for 5 days and you’re only now finding out about it, it sounds to me like it was perhaps a bug that was fixed without realising the full impact of it, or perhaps without realising it made it to production; and only an audit that happened later caught it.

Not ideal by any means. I’d be curious to know if my theory is correct or not.

zwass 3 years ago | |

Their statements indicate they were aware and investigating. My frustration is that they didn't give users the opportunity to do their own timely investigation.

> GitHub learned via a customer support ticket that GitHub Apps were able to generate scoped installation tokens with elevated permissions. Each of these tokens are valid for up to 1 hour.

> GitHub quickly fixed the issue and established that this bug was recently introduced, existing for approximately 5 days between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC.

> GitHub immediately began working to fix the bug and started an investigation into the potential impact. However due to the scale and complexity of GitHub Apps and their short-lived tokens, we were unable to determine whether this bug was ever exploited.

bearjaws 3 years ago |

I am really curious how SecOps works at GitHub.

Why not after remediation inform users of the flaw and potential impact. Then follow up with detailed impact.

Instead we get this 3 months later all they can say is "Some of your apps refreshed their tokens during a 5 day period" which is not news...

This is also the second time this year there has been significant delay in communication. Granted those involved other third parties so who knows where the delay lived.

s09dfhks 3 years ago | |

I would assume the lawyers have to get involved first to write up some document proving that github is not liable for what users do with their app tokens. CYA, then tell the public

mistrial9 3 years ago | |

they might assume that the account holder are a likely perpetrator, which might be true but also enables their intermediation of the context and control of the sequence of communication.

smarx007 3 years ago |

I just got it as well and don't understand what I can do. Can I somehow force all generated tokens to be revoked and get apps to generate new tokens to be on the safe side? Or, rather, is there a way to do this without uninstalling the apps and installing them again?

nrmitchi 3 years ago | |

>Each of these tokens are valid for up to 1 hour.

> GitHub quickly fixed the issue and established that this bug was recently introduced, existing for approximately 5 days between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC.

It doesn't sound like there is anything you can or need to do with respect to these tokens (whether they were used to take action with elevated permissions is a different thing, but it doesn't sound like that was the case either)

dundarious 3 years ago | | |

It seems like the appropriate thing to do would be to inform anyone who had tokens created during the affected time period, so they could assess if any of the permissions led to undesired changes. Instead of GitHub saying “we don’t have hard proof of anything bad happening” and waiting 3 months, just give the customer the time of relevant token creations.

onphonenow 3 years ago | |

Aren't these very time limited anyways (hours vs days)? So once they fix the bug and wait an hour the old tokens are gone anyways?

mynameisvlad 3 years ago | | |

Per the email:

> Each of these tokens are valid for up to 1 hour.

So these tokens would've been expired for 3 months now, according to when the fix was deployed.

bastardoperator 3 years ago | |

You could revoke the pem that generated these tokens, but as others have pointed out the tokens in question are only valid for 60 minutes at maximum.

njibhu 3 years ago |

Can somebody tell me if I'm wrong on my take but this bug/issue means:

- a github app which had read permission on issues could elevate its permission to write

- a github app which had read permissions to discussions could elevate its permissions to write.

So far if the org/user would have been compromise they would have seen with issues or conversations containing content from the app.

Since these are only examples, I can imagine the case with major impact would be a contents:read elevate to content write. But again with commit signing, this would also be caught by the user. What did I miss where the impact would have not been visible to the end user/org ?

zwass 3 years ago | |

contents:read to contents:write is a big deal! Just to pick out a random widely used project, nodejs [1] has a number of unsigned commits to the main branch. Their commits could have been tampered with during this timeframe.

What about release artifacts?

[1]: https://github.com/nodejs/node/commits/main

njibhu 3 years ago | | |

I guess I can see it, but branch protection rules and pull requests reviews would also prevent that to happen in my opinion

(also ability to do it with content:write is just speculation from my side, they don't make it clear if it is possible, that would need to be confirmed by github)

kenbolton 3 years ago |

In the same minute that I learned GitHub had been acquired by Microsoft, I cancelled my pro subscription and began moving my critical repositories elsewhere. I'm old enough to remember the MS that tried to choke the life out of GNU/Linux and spread FUD about all FLOSS, the one that engaged in anti-competetive behavior during the "Browser Wars". I'm not suggesting that this blunder of a delay is related to the Microsoft acquisition, but the abusive "Look at me, I'm changing" spiel never cut any mustard with me.

Vote at the ballot box and with your dollars. Do not reward executives, lawyers, or engineers who dissemble or obfuscate.

throwaway78246 3 years ago |

It was also my experience that it takes GitHub several months to fix issues and inform compromised users.

ghyty 3 years ago |

Does anyone know what type of audit logs or actions we need to look for?

darthrupert 3 years ago |

Yep. As I mentioned elsewhere on the pyright/pylance issue: Microsoft are scumbags.