Do they know that the attackers were after? Maybe they were just trying to help fix the availability problems.
This comment reminds me of a joke where the punchline is that a person is so poor that burglars break in to their house and leave money.
Similarly, I could see ransomware groups hacking in and feeling bad for GH so they improve a few things to help them get to at leave nine fives of uptime.
Many years ago there was an attack that went around that used the server’s BMC as an entry point. Thing is, BMCs are universally shit, so as part of the attack, the attackers also fixed a bunch of bugs so their connection could persist. I was working in hardware management at the time, and when we heard about that, we all gave that one a hard think…
It should be in their interest actually, since much of the malware is spread via GitHub.
Just in case you are not aware, a joke loses its fun factor if you explain it.
That's the reason I stopped installing random extensions and even themes in VS Code, they are too dangerous.
I just moved to Zed (zed.dev). Has everything I need
does it have some kind of sandboxing for its extensions?
unfortunately it's not anprroved tool in many companies. VSCode's news agents window is quite similar to zed's parallel agents UI though.
It's absolutely reprehensible that they don't immediately name the extension.
Unless it was "Waifu-SFX-AutoComplete"
That kind of thing might be a case to not publicly disclose..
Most large companies won’t allow direct access to Docker hub or PyPI, and now they’ll have to restrict access to VSCode extensions. How did the extension get poisoned?
We run an explicit whitelist, enforced through Microsoft Entra (or was it Intune).
Are we going into 99.9% Uptime era?
With this level of availability, would company remain on cloud?
npm next please
Github compromised and 3800 internal repos exposed.
Why did one developer have access, even if read-only, to more than 3,800 internal repos?
Read-only access to all non-sensitive code is how things should be. Huge engineering culture and productivity booster. It’s also very useful to keep each other honest (I’ve found so many “interesting” things hidden away in organizations with tight read access restrictions).
Devs not having read access to all code seems like a massive org smell. What’s worse, in many cases not having access doesn’t just prevent you from seeing it it also prevents you from knowing it exists. Now you don’t know what to ask for, who to ask, or what to not implement again.
There is no security risk that you could use to convince me
that ”devs should only have access to code they need to modify”.
in my org, devs don’t have access to customer data directly, and sysadmins don’t have access to modify code.
It’s a simple rule from a simpler time, to limit the risk of total compromise.
Not saying it’s good but I think it’s quite common for devs to have read only access to everything. I suspect that with all the recent news, including this, the needle might start to shift a bit.
I think it’s actually non-trivial to determine how many repos you should have read-only access to. I frequently hop through multiple repos that I don’t contribute to, just to understand how the system is architected and what it does at different stages. We even have an internal Claude skill for finding relevant repo for a given problem which relies on personal gh access (via CLI). It _can_ be done more securely but those defaults built over many years will take time to change.
I think it is pretty common that devs have read only access to all source code.
The real question is why github has 3800 internal repos.
Shoot dude, the engineering organization I mentor/teach at a high school has ~75 internal repos.
Robot source code; satellite ground station hardware; satellite ground station software; visualization; satellite hardware; satellite software; nuttx + its submodules for 2 different projects; linux kernel fork; circuitpython fork; raspberry pico tools fork; embedded programming/debugging tools; my lecture notes; my automated grading tooling; etc etc etc. That's just me + ~35 students in classes.
Pretty easy to see how when you have scale you can get to a few thousand.
each employee with personal fork of some company microservice
It's normal that a dev has *access* to all the code.
But did he clone all the repos into his machine? I doubt it. So, the hacker extracted all the 3800 repos using the employee's machine as a gateway? I doubt it as well, I'm sure they would have detected this huge amount of data much earlier than transferring all of it?
> The real question is why github has 3800 internal repos.
I guess they mean customer's private repos?
> I guess they mean customer's private repos?
I don't think so. It is even worse if a random developer has access to customers' private repos.
Good point. Then why in the world would a company have 3,500 repos? Do they create a repo for each employee?
They’ve been developing git and GitHub for over a decade. It really isn’t surprising they have made thousands of internally available repos. They probably have hundreds just for running automated tests alone.
I am sure many of their employees create repos. Is that strange?
It doesn’t mean they are all masterpieces of elaborate production code.
All the attackers need to do is steal an SSH key and they'd be able to clone everything, no?
Nah GitHub/MS doesn't allow SSH keys for their internal stuff. You have to use git-credential-manager, which enforces MFA
Depends how it's set up. Many companies add an IP address check so if you don't come via their VPN (or are not in the office) the connection will be rejected before any auth is asked.
So you'd need to authenticate for the VPN, which often has 2nd factor.
But I have no idea of how they are set up.
Security is often overlooked internally and seen as source of friction.
I worked at a popular US social media firm and it wasn't hard to get a permission that allows me to delete the entire company's dataset. Often arguments around "I'm working on org-level initiative and I need to get permission to get it done" would easily get me the permission.
It _is_ a source of friction.
I can think of _one_ product that allows you to set up low-friction access management, and AFAIK most users of that product don't set it up that way.
Software engineers _should_ be able to request access to dev resources JIT during their day-to-day work, have that access auto-approve in >99% of cases, have it auto-expire if they don't actually use the resources, and have all of that be subject to anomaly detection/approval escalations and other auditing.
Instead in most orgs it's like fill out a form, get your manager (who's always in meetings) to approve and then wait some number of days for a human to click-ops your request. At best you can open a PR and have the changes applied in an hour or two.
You _should_ be able to get access to things pretty much immediately if you need them and they're not sensitive. Then we could deny by default without cratering productivity.
Please name the product (that seems a good idea)
It’s the big advantage that small companies have over big ones.
I’ve ridden startups through the phase where they transition to “responsible adults”, and start putting in policies and locking things down and generally behaving like the giant corporations they expect to be one day (and that the locker downers came from and are used to).
You can feel the deceleration, like taking your foot off the gas on the freeway. I’ve sat through all hands meetings where the ceo asked why we don’t ship as fast anymore, and since by that time most of the fast moving folk have moved on, nobody has an explanation.
Security is often an excuse to block other teams to do legitimate work and so often it's fairly braindead. Security IMO needs to get it's act together, passkeys is a great example of security gone wrong from a UX design perspective because you can't hold them to the same standards as product or infra teams, they have the special privilege of breaking things and it increasing their metrics.
Tell them to make a better UX and they lose their minds in a huffy puff of fake crisis mode or get avoidant with stonewalling 'secret security stuff' that you can't hold them to account for. Or eat 50% of developer machine performance for "endpoint security" and the carnival of sadness goes on and on.
Signal is an example of security as a product that was actually designed for user UX in mind to give one example.
Why not? If you don't rely on security by obscurity, having access to code is not a security issue.
If you want to move fast, you need access. Unfortunately and obviously this allows threat actors to move fast, too. The tradeoff had a different risk profile a year ago, heck a couple weeks ago.
Because every developer asking for permission 3,800 times is exhausting for everyone.
[deleted][deleted]
Github is the last place someone will give a single shit about for something like that. If someone steal your debit card and withdraw money on behalf of you, without permission of you, you go to the bank and explain that. Github holds code... If something like some info stolen from your work something like that then you not work them again, you quit or go to hr, this is how it is.
Plus, github is running on your computer. People take https icon so seriously. It is nothing. There are more browsers than actual websites. You receive a browser update almost every day. All of them comes with https icons w predefined domains. Github is the one that comes with new computers. The others are the websites someone defined in your invisible /etc/hosts before you start using your own computer. Your own websites are http. I know how the internet works very very well. Github is no more than text editor with undo redo.
[deleted]
I'm not sure if this is related or not. But a few days ago, I saw commits from the "future tense" in some repositories. When you read "committed tomorrow" after a commit, it's not funny at all. I posted a screenshot in the announcement on GitHub.
That's probably unrelated. The date of a commit in git can be modified to whatever you want. I once backdated commits because my timezone was off, and I wanted the timestamps to match the ticketing system. Github displays the date stored in the commit, since there is not really a way to verify it.
Ok. Copy that. tnx
I think the commit timestamp is just passed through from timestamps in the git repo, not the time at which the commits were pushed to the server. You can probably set your system time to the future, make some commits and push them.
But you can change the commit date from cli when committing? Github just shows the commit metadata, right?
Do they know that the attackers were after? Maybe they were just trying to help fix the availability problems.
This comment reminds me of a joke where the punchline is that a person is so poor that burglars break in to their house and leave money.
Similarly, I could see ransomware groups hacking in and feeling bad for GH so they improve a few things to help them get to at leave nine fives of uptime.
Many years ago there was an attack that went around that used the server’s BMC as an entry point. Thing is, BMCs are universally shit, so as part of the attack, the attackers also fixed a bunch of bugs so their connection could persist. I was working in hardware management at the time, and when we heard about that, we all gave that one a hard think…
It should be in their interest actually, since much of the malware is spread via GitHub.
Just in case you are not aware, a joke loses its fun factor if you explain it.
That's the reason I stopped installing random extensions and even themes in VS Code, they are too dangerous.
I just moved to Zed (zed.dev). Has everything I need
does it have some kind of sandboxing for its extensions?
unfortunately it's not anprroved tool in many companies. VSCode's news agents window is quite similar to zed's parallel agents UI though.
https://xcancel.com/i/status/2056949168208552080
Why are half the comments in that thread AI generated? What value do they think they bring?
Cookie points, interaction, favorites, Super Mario Bros stars.
Money is a small thing to spend for all the fame it brings. Remeber: Value trumps everything, an everyone wants it. From investors to end users. /s
Discussion (222 points, 4 hours ago, 62 comments) https://news.ycombinator.com/item?id=48201316
Which extension was it?
If I had to guess it is the NX console extension that was compromised yesterday. But I’m not 100% sure.
https://github.com/nrwl/nx-console/security/advisories/GHSA-...
It's absolutely reprehensible that they don't immediately name the extension.
Unless it was "Waifu-SFX-AutoComplete"
That kind of thing might be a case to not publicly disclose..
Most large companies won’t allow direct access to Docker hub or PyPI, and now they’ll have to restrict access to VSCode extensions. How did the extension get poisoned?
We run an explicit whitelist, enforced through Microsoft Entra (or was it Intune).
Are we going into 99.9% Uptime era?
With this level of availability, would company remain on cloud?
npm next please
Github compromised and 3800 internal repos exposed.
Why did one developer have access, even if read-only, to more than 3,800 internal repos?
Read-only access to all non-sensitive code is how things should be. Huge engineering culture and productivity booster. It’s also very useful to keep each other honest (I’ve found so many “interesting” things hidden away in organizations with tight read access restrictions).
Devs not having read access to all code seems like a massive org smell. What’s worse, in many cases not having access doesn’t just prevent you from seeing it it also prevents you from knowing it exists. Now you don’t know what to ask for, who to ask, or what to not implement again.
There is no security risk that you could use to convince me that ”devs should only have access to code they need to modify”.
in my org, devs don’t have access to customer data directly, and sysadmins don’t have access to modify code.
It’s a simple rule from a simpler time, to limit the risk of total compromise.
Not saying it’s good but I think it’s quite common for devs to have read only access to everything. I suspect that with all the recent news, including this, the needle might start to shift a bit.
I think it’s actually non-trivial to determine how many repos you should have read-only access to. I frequently hop through multiple repos that I don’t contribute to, just to understand how the system is architected and what it does at different stages. We even have an internal Claude skill for finding relevant repo for a given problem which relies on personal gh access (via CLI). It _can_ be done more securely but those defaults built over many years will take time to change.
I think it is pretty common that devs have read only access to all source code.
The real question is why github has 3800 internal repos.
Shoot dude, the engineering organization I mentor/teach at a high school has ~75 internal repos.
Robot source code; satellite ground station hardware; satellite ground station software; visualization; satellite hardware; satellite software; nuttx + its submodules for 2 different projects; linux kernel fork; circuitpython fork; raspberry pico tools fork; embedded programming/debugging tools; my lecture notes; my automated grading tooling; etc etc etc. That's just me + ~35 students in classes.
Pretty easy to see how when you have scale you can get to a few thousand.
each employee with personal fork of some company microservice
It's normal that a dev has *access* to all the code.
But did he clone all the repos into his machine? I doubt it. So, the hacker extracted all the 3800 repos using the employee's machine as a gateway? I doubt it as well, I'm sure they would have detected this huge amount of data much earlier than transferring all of it?
> The real question is why github has 3800 internal repos.
I guess they mean customer's private repos?
> I guess they mean customer's private repos?
I don't think so. It is even worse if a random developer has access to customers' private repos.
Good point. Then why in the world would a company have 3,500 repos? Do they create a repo for each employee?
They’ve been developing git and GitHub for over a decade. It really isn’t surprising they have made thousands of internally available repos. They probably have hundreds just for running automated tests alone.
I am sure many of their employees create repos. Is that strange?
It doesn’t mean they are all masterpieces of elaborate production code.
All the attackers need to do is steal an SSH key and they'd be able to clone everything, no?
Nah GitHub/MS doesn't allow SSH keys for their internal stuff. You have to use git-credential-manager, which enforces MFA
Depends how it's set up. Many companies add an IP address check so if you don't come via their VPN (or are not in the office) the connection will be rejected before any auth is asked.
So you'd need to authenticate for the VPN, which often has 2nd factor.
But I have no idea of how they are set up.
Security is often overlooked internally and seen as source of friction. I worked at a popular US social media firm and it wasn't hard to get a permission that allows me to delete the entire company's dataset. Often arguments around "I'm working on org-level initiative and I need to get permission to get it done" would easily get me the permission.
It _is_ a source of friction.
I can think of _one_ product that allows you to set up low-friction access management, and AFAIK most users of that product don't set it up that way.
Software engineers _should_ be able to request access to dev resources JIT during their day-to-day work, have that access auto-approve in >99% of cases, have it auto-expire if they don't actually use the resources, and have all of that be subject to anomaly detection/approval escalations and other auditing.
Instead in most orgs it's like fill out a form, get your manager (who's always in meetings) to approve and then wait some number of days for a human to click-ops your request. At best you can open a PR and have the changes applied in an hour or two.
You _should_ be able to get access to things pretty much immediately if you need them and they're not sensitive. Then we could deny by default without cratering productivity.
Please name the product (that seems a good idea)
It’s the big advantage that small companies have over big ones.
I’ve ridden startups through the phase where they transition to “responsible adults”, and start putting in policies and locking things down and generally behaving like the giant corporations they expect to be one day (and that the locker downers came from and are used to).
You can feel the deceleration, like taking your foot off the gas on the freeway. I’ve sat through all hands meetings where the ceo asked why we don’t ship as fast anymore, and since by that time most of the fast moving folk have moved on, nobody has an explanation.
Security is often an excuse to block other teams to do legitimate work and so often it's fairly braindead. Security IMO needs to get it's act together, passkeys is a great example of security gone wrong from a UX design perspective because you can't hold them to the same standards as product or infra teams, they have the special privilege of breaking things and it increasing their metrics.
Tell them to make a better UX and they lose their minds in a huffy puff of fake crisis mode or get avoidant with stonewalling 'secret security stuff' that you can't hold them to account for. Or eat 50% of developer machine performance for "endpoint security" and the carnival of sadness goes on and on.
Signal is an example of security as a product that was actually designed for user UX in mind to give one example.
Why not? If you don't rely on security by obscurity, having access to code is not a security issue.
If you want to move fast, you need access. Unfortunately and obviously this allows threat actors to move fast, too. The tradeoff had a different risk profile a year ago, heck a couple weeks ago.
Because every developer asking for permission 3,800 times is exhausting for everyone.
Github is the last place someone will give a single shit about for something like that. If someone steal your debit card and withdraw money on behalf of you, without permission of you, you go to the bank and explain that. Github holds code... If something like some info stolen from your work something like that then you not work them again, you quit or go to hr, this is how it is.
Plus, github is running on your computer. People take https icon so seriously. It is nothing. There are more browsers than actual websites. You receive a browser update almost every day. All of them comes with https icons w predefined domains. Github is the one that comes with new computers. The others are the websites someone defined in your invisible /etc/hosts before you start using your own computer. Your own websites are http. I know how the internet works very very well. Github is no more than text editor with undo redo.
I'm not sure if this is related or not. But a few days ago, I saw commits from the "future tense" in some repositories. When you read "committed tomorrow" after a commit, it's not funny at all. I posted a screenshot in the announcement on GitHub.
That's probably unrelated. The date of a commit in git can be modified to whatever you want. I once backdated commits because my timezone was off, and I wanted the timestamps to match the ticketing system. Github displays the date stored in the commit, since there is not really a way to verify it.
Ok. Copy that. tnx
I think the commit timestamp is just passed through from timestamps in the git repo, not the time at which the commits were pushed to the server. You can probably set your system time to the future, make some commits and push them.
But you can change the commit date from cli when committing? Github just shows the commit metadata, right?