Lots of apologia for Github here. Aside from the fact that defending a billion-dollar company is a bit strange; especially one that is steward to the the overwhelming majority of open-source software.
Maybe that's good-will doing the work? For me it's always been a sour pill to swallow that I have to buy in to a large companies internal politics and practices in order to work on projects I love. I don't feel like I owe them anything.
Especially if they can't hold up their end of the deal.
Unfettered access to the world's software repositories, for the princely sum of a bucketload of Azure credits.
Let me ask the question in reverse: what do you have against them such that the fellow human beings struggling to maintain their operations don’t deserve even a modicum of kindness, respect, and good will? Are you unable to separate the business from the hard working people behind it?
It’s not like they don’t know that people like us are counting on them: they recognize that their service is the “dial tone” for much of the world’s software development capability. They are keenly aware of the impact.
What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?
When did OP blame the people involved personally?
If I to hire a contractor to redo my roof, and that roof leaks, whether they worked hard or not is immaterial. They did not do the task in they were paid to do. I'm not going to buy their services again just because their shingles guy was particularly charming.
MS has talented engineers, but that's a complete misdirection. Github is a service in decline: there is nothing wrong with criticizing them.
I have all the empathy for people in the world.
A corporation is not a person. If your organization cannot handle the load, then you need to adjust your practices. The organization needs to prioritize their paying users. The organization needs to shift people from new features to keeping the lights on. And maybe the organization needs to find another strategy to manage its azure transition.
OP didn’t blame the staff. His focus is on the company.
Invoking individual workers well-being to defend a billion dollar company is also very strange.
#hugops is to your coworkers, not to the nameless big-corps who can't maintain a service for paying customers. You should be raising a shitstorm when things you pay for aren't reliable or unusable.
Hot take, if it's traffic is causing issues, throttle your free-tier, pause signups, or stop giving out free things (like runner time).
[deleted]
>What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?
Would you feel the same way about a colleague who kept causing downtime in your product again and again, seemingly without making any progress in addressing whatever issue was causing their repeated mistakes?
There are web applications out there that are far more complex than GitHub but have much less downtime. It's not like they're facing an unsolvable problem.
You don’t know that it was “their mistake.” Unless you’ve personally successfully scaled a suite of nontrivial services equivalent to GitHub’s to accommodate an unexpected 14x increase in traffic, you respectfully have no basis for such an assertion.
I have.
You could argue the scales are different, but computers are also faster now.
So, argument to credentialism out of the way... What should we do as consumers if a provider that is a defacto monopoly due to network effects stops functioning?
Yeah, they should be testing for that, right? I think there's a lot of people reading comments like yours and thinking, is this person a paid shill or what?
The earn bucket loads of money, they should be planning for exactly that. And testing for it via load testing every day.
Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders.
[dead]
Executives have made a choice to not pay for top talent at Microsoft Azure and Github.
Would you consider telling this to the people working at GitHub directly? I’m sure they’d appreciate your evaluation of their skills and talent.
There are two options, either they are lousy at their jobs, or they are incapable of pushing back against unrealistic demands. Neither is a good indicator of their skill and talent as engineers.
I know I am speaking from a position of some privilege, but I have previously left workplaces that did not allow me to practice good engineering, and I do expect others to do so.
I think it depends if you pay them money. If you do, then you should indeed have strong expectations towards them and hold them accountable. If they provide a free service to you, then it's still reasonable to feel upset, but at the same time you get what you pay for.
> Aside from the fact that defending a billion-dollar company is a bit strange
More than a bit strange. This is an HNism that I'll never get. Why would you go to the comment section anywhere to passionately try to defend the honor of a trillion dollar company, unless 1. you're being paid to astroturf or 2. you own that company's stock? Satya Nadella isn't going to read a post here and say, "Gosh, how nice of that commenter! I'm going to send him some Microsoft stock as a show of appreciation for him defending us online!" I don't think I'll ever understand company-fanboys.
I'm surprised at how little the perception of GitHub changed post-acquisition. Coupled with WSL, it almost balanced things for a lot of people and put Microsoft back in the "benefit of the doubt" column. This is undoing a lot of that, on top of the operational costs. Suddenly the bad press is more noticeable and harder to ignore.
Defending a multi-trillion dollar company you mean (Microsoft).
there are two groups of people willing to die defending [billion-dollar company]: HN users and Nintendo fans
> Maybe that's good-will doing the work?
Of course. GitHub has been an enormous gift to the open source community. Arguably more than Git itself. They deserve a lot of good will.
they are not the non-profit. they make money of it and devs expect certain kind of service in return. GH failed to deliver on the service expectation.
Also, the former stewards of that open source goodness sold it to Microsoft for a cheap buck.
Any goodwill they earned has been spent.
Using "apologia" here is pretty embarrassing.
[dead]
I think its the fact that people have used the software for so long that they feel emotional to it (Hashimoto crying tears of sadness when he decided to move ghostty away from github) and there is completely nothing wrong about it as we are emotional human beings.
But, you are right in the sense that, Github has failed to accept its part of the deal which is actually to just be a usable place. People HAVE previously tolerated so much AI slop and slowness in github's UI just because of its reliability but this downtime is like the Github's achilles heel.
At some point, I recommend people to accept this and move to more healthier alternatives, there is also an momentum. For example, the only reason I joined github was that I wanted to join codeberg but so many of projects used github and involved sign in with github that I finally gave in into github and I had thought that codeberg is so good but nobody is gonna come here because of the network effects but the tide is turning and I hope more people look into codeberg and healthier alternatives.
This is a real business continuity issue for us. We’re kinda stuck with GitHub Enterprise but we may need to move from cloud to on-premises if this keeps up.
I don't think aggregating the whole platform into one number is fair. It's like adding the whole aws into one number
It’s obviously a meme website, the meme is more funny when the number isn’t high. Anyone looking for actual accurate info would go to the real status page.
I think the correct middle ground is a site that lets you select the parts of the platform you rely on and ignore the others. For example, GitHub is "down" for me when I can't push, process PRs, or release packages, but I don't care about Actions or AI features.
On the other hand when you have a reasonably complex deployment it's easy to get swamped with dashboards showing CPU, Memory, I/O, application-metrics, signups, active users/sessions, etc.
Instead it's nice to think about how you can express the state of a complete system as a single number. It might be you divide active user sessions by database-connections, and then scale by memory capacity.
But as a single digit you can then get used to normal ranges, and have it always visible somewhere obvious. A single number won't show details, but when it changes you can go look at the specific metrics. It's a cute shorthand, and it can work well as a basic "are we normal" check.
splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse
there is a subset of the site that pretty much everyone uses — git, issues, pull requests, actions — and if any part of that is broken then the site is broken and the status page should indicate how often this happens
> splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse
This is a pretty ungenerous take. You could look at it the other way: if I don't use actions then it's useful for me to know that only actions are broken, and I can continue in my normal usage. If you bundle everything up then the status page is reporting an unhelpful false positive for me.
Github has far less services and regions that AWS.
quite literally kicking github while it's down!
Supposedly commits on GitHub are up 14x YoY.
They are getting spammed by AI agents?
Yes. There’s no other explanation for 14x, that’s nuts.
Is it spam when they’ve been pushing for this shit and putting AI prompt everywhere fir a year or more?
Commits or pushes? Commits aren't really a worthwhile source of measurement in terms of load.
14x is insane, especially since the quality and quantity of IRL software has barely budged.
One could hope that we'd use these newfound agentic coding powers to actually realize value, improve quality, etc. Instead I see enshittification and stagnation. What are we even doing with all these tokens?
The same thing we’ve done with every other productivity increase in a world based on unfettered growth: garbage.
Supposedly commits on GitHub are up 14x YoY.
So?
If Microsoft can't scale, who can?
If it can't provide the service, it should stop selling until it can.
This is like the AOL dialup busy signal fiasco of the mid-90's all over again. Except this time, instead of getting mad, people are making excuses for the poor, beleaguered trillion-dollar company.
>If it can't provide the service, it should stop selling until it can.
You literally cannot buy GitHub Copilot right now [1].
With Github going up and down and Ubuntu going up and (mostly) down, there's a lot of time for intra-office sword fighting or whatever, lately. If somebody takes down Claude, everybody's going to have to just go home for the day. (https://xkcd.com/303/)
Love. Hope Github is a relic of the past inside 12 months
[delayed]
Becoming a joke is the one think that could end the GitHub monopoly.
GitHub is not a monopoly. It never has been. You've always been able to self-host or you can use gogs, gitea, gitlab, bitbucket, you get the idea
I wonder what morale is like at github. This is like gamer level hating
It's not great. Just talked to a hubber last week. They said everyone inside feels pretty dejected right now, and these posts don't help.
I feel for them -- with AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that along with the maintainers. That's a lot of work that gets done with each PR.
But they need to make some changes quickly.
But the amount of compute needed to serve is not very high. It's all text. The amount of bandwidth and compute needed to serve a Netflix or YouTube is far far harder and they managed just fine.
they also aren't using azure. IDK what youtube is on, but netflix has actually faced their problems and found solutions (freebsd, mostly)
I wouldn't feel too bad for them with their top-of-market comp and valuable RSU packages.
I don't believe they pay top of market, but even if they did, it's possible to make a lot of money and still feel bad when you have a sense of ownership and responsibility to the users of your service.
GitHub doesn't pay top of market.
You're right.
That being said, 300k TC for E4 is still pretty good. Plus the RSUs have gone up like 60% in the last several years.
My point is that they are compensated well. Don't need to split hairs and see if they are at Netflix level or not.
I just dont really buy the explanation though. It seems so solvable to hack a throttle or something in place, especially for non-paid plans. The cracks were also showing before AI hit the scene.
Im not saying this is the end-game solution but absolutely they could have put temporary safeguards in place while they "figure it out" if it _really_ is just AI driven slop setting their computers on fire.
The whole "anyone can submit a PR" thing has been a UX issue from day one. That probably needs to go away, and I doubt anyone would really miss it. Where Github could help is by providing a means to build trust that doesn't involve random unknown people slinging code at projects.
Any sort of trust requirement would break the entire model and cause some serious inequality.
How would a random kid in a 3rd world country ever get noticed enough to enter a trust circle, for example?
[flagged]
Whose opinions are left? The adults are all moving elsewhere.
Heck I stopped using it for projects in 2018, even before the acquisition.
My company was going to end a 6-figure YoY contract with a GitHub Actions competitor to move to GitHub, but scrapped those plans and renewed this morning. That move had been in planning for like 6 months.
this is some good life advice
and it leads to triple 9 availability (80.999%), or better
/ponder .oO( any irrational number has "infinite nines" )
Lots of apologia for Github here. Aside from the fact that defending a billion-dollar company is a bit strange; especially one that is steward to the the overwhelming majority of open-source software.
Maybe that's good-will doing the work? For me it's always been a sour pill to swallow that I have to buy in to a large companies internal politics and practices in order to work on projects I love. I don't feel like I owe them anything.
Especially if they can't hold up their end of the deal.
Unfettered access to the world's software repositories, for the princely sum of a bucketload of Azure credits.
Let me ask the question in reverse: what do you have against them such that the fellow human beings struggling to maintain their operations don’t deserve even a modicum of kindness, respect, and good will? Are you unable to separate the business from the hard working people behind it?
It’s not like they don’t know that people like us are counting on them: they recognize that their service is the “dial tone” for much of the world’s software development capability. They are keenly aware of the impact.
What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?
When did OP blame the people involved personally?
If I to hire a contractor to redo my roof, and that roof leaks, whether they worked hard or not is immaterial. They did not do the task in they were paid to do. I'm not going to buy their services again just because their shingles guy was particularly charming.
MS has talented engineers, but that's a complete misdirection. Github is a service in decline: there is nothing wrong with criticizing them.
I have all the empathy for people in the world.
A corporation is not a person. If your organization cannot handle the load, then you need to adjust your practices. The organization needs to prioritize their paying users. The organization needs to shift people from new features to keeping the lights on. And maybe the organization needs to find another strategy to manage its azure transition.
OP didn’t blame the staff. His focus is on the company.
Invoking individual workers well-being to defend a billion dollar company is also very strange.
#hugops is to your coworkers, not to the nameless big-corps who can't maintain a service for paying customers. You should be raising a shitstorm when things you pay for aren't reliable or unusable.
Hot take, if it's traffic is causing issues, throttle your free-tier, pause signups, or stop giving out free things (like runner time).
>What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?
Would you feel the same way about a colleague who kept causing downtime in your product again and again, seemingly without making any progress in addressing whatever issue was causing their repeated mistakes?
There are web applications out there that are far more complex than GitHub but have much less downtime. It's not like they're facing an unsolvable problem.
You don’t know that it was “their mistake.” Unless you’ve personally successfully scaled a suite of nontrivial services equivalent to GitHub’s to accommodate an unexpected 14x increase in traffic, you respectfully have no basis for such an assertion.
I have.
You could argue the scales are different, but computers are also faster now.
So, argument to credentialism out of the way... What should we do as consumers if a provider that is a defacto monopoly due to network effects stops functioning?
https://news.ycombinator.com/item?id=47947719
https://www.linkedin.com/in/jharasym/
Yeah, they should be testing for that, right? I think there's a lot of people reading comments like yours and thinking, is this person a paid shill or what?
The earn bucket loads of money, they should be planning for exactly that. And testing for it via load testing every day.
Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders.
[dead]
Executives have made a choice to not pay for top talent at Microsoft Azure and Github.
Would you consider telling this to the people working at GitHub directly? I’m sure they’d appreciate your evaluation of their skills and talent.
There are two options, either they are lousy at their jobs, or they are incapable of pushing back against unrealistic demands. Neither is a good indicator of their skill and talent as engineers.
I know I am speaking from a position of some privilege, but I have previously left workplaces that did not allow me to practice good engineering, and I do expect others to do so.
I think it depends if you pay them money. If you do, then you should indeed have strong expectations towards them and hold them accountable. If they provide a free service to you, then it's still reasonable to feel upset, but at the same time you get what you pay for.
> Aside from the fact that defending a billion-dollar company is a bit strange
More than a bit strange. This is an HNism that I'll never get. Why would you go to the comment section anywhere to passionately try to defend the honor of a trillion dollar company, unless 1. you're being paid to astroturf or 2. you own that company's stock? Satya Nadella isn't going to read a post here and say, "Gosh, how nice of that commenter! I'm going to send him some Microsoft stock as a show of appreciation for him defending us online!" I don't think I'll ever understand company-fanboys.
I'm surprised at how little the perception of GitHub changed post-acquisition. Coupled with WSL, it almost balanced things for a lot of people and put Microsoft back in the "benefit of the doubt" column. This is undoing a lot of that, on top of the operational costs. Suddenly the bad press is more noticeable and harder to ignore.
Defending a multi-trillion dollar company you mean (Microsoft).
there are two groups of people willing to die defending [billion-dollar company]: HN users and Nintendo fans
> Maybe that's good-will doing the work?
Of course. GitHub has been an enormous gift to the open source community. Arguably more than Git itself. They deserve a lot of good will.
they are not the non-profit. they make money of it and devs expect certain kind of service in return. GH failed to deliver on the service expectation.
https://xkcd.com/1150/
You're right, but that GitHub is dead.
Also, the former stewards of that open source goodness sold it to Microsoft for a cheap buck.
Any goodwill they earned has been spent.
Using "apologia" here is pretty embarrassing.
[dead]
I think its the fact that people have used the software for so long that they feel emotional to it (Hashimoto crying tears of sadness when he decided to move ghostty away from github) and there is completely nothing wrong about it as we are emotional human beings.
But, you are right in the sense that, Github has failed to accept its part of the deal which is actually to just be a usable place. People HAVE previously tolerated so much AI slop and slowness in github's UI just because of its reliability but this downtime is like the Github's achilles heel.
At some point, I recommend people to accept this and move to more healthier alternatives, there is also an momentum. For example, the only reason I joined github was that I wanted to join codeberg but so many of projects used github and involved sign in with github that I finally gave in into github and I had thought that codeberg is so good but nobody is gonna come here because of the network effects but the tide is turning and I hope more people look into codeberg and healthier alternatives.
This is a real business continuity issue for us. We’re kinda stuck with GitHub Enterprise but we may need to move from cloud to on-premises if this keeps up.
I don't think aggregating the whole platform into one number is fair. It's like adding the whole aws into one number
It’s obviously a meme website, the meme is more funny when the number isn’t high. Anyone looking for actual accurate info would go to the real status page.
I think the correct middle ground is a site that lets you select the parts of the platform you rely on and ignore the others. For example, GitHub is "down" for me when I can't push, process PRs, or release packages, but I don't care about Actions or AI features.
On the other hand when you have a reasonably complex deployment it's easy to get swamped with dashboards showing CPU, Memory, I/O, application-metrics, signups, active users/sessions, etc.
Instead it's nice to think about how you can express the state of a complete system as a single number. It might be you divide active user sessions by database-connections, and then scale by memory capacity.
But as a single digit you can then get used to normal ranges, and have it always visible somewhere obvious. A single number won't show details, but when it changes you can go look at the specific metrics. It's a cute shorthand, and it can work well as a basic "are we normal" check.
splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse
there is a subset of the site that pretty much everyone uses — git, issues, pull requests, actions — and if any part of that is broken then the site is broken and the status page should indicate how often this happens
> splitting the status page like they do, to the point where it is only a bit of humourous exaggeration to say that they track broken `git push` and `git pull` separately, is a sleight of hand / accounting / SLA-fudging that we should not excuse
This is a pretty ungenerous take. You could look at it the other way: if I don't use actions then it's useful for me to know that only actions are broken, and I can continue in my normal usage. If you bundle everything up then the status page is reporting an unhelpful false positive for me.
Github has far less services and regions that AWS.
quite literally kicking github while it's down!
Supposedly commits on GitHub are up 14x YoY.
They are getting spammed by AI agents?
Yes. There’s no other explanation for 14x, that’s nuts.
Is it spam when they’ve been pushing for this shit and putting AI prompt everywhere fir a year or more?
Commits or pushes? Commits aren't really a worthwhile source of measurement in terms of load.
14x is insane, especially since the quality and quantity of IRL software has barely budged.
One could hope that we'd use these newfound agentic coding powers to actually realize value, improve quality, etc. Instead I see enshittification and stagnation. What are we even doing with all these tokens?
The same thing we’ve done with every other productivity increase in a world based on unfettered growth: garbage.
Supposedly commits on GitHub are up 14x YoY.
So?
If Microsoft can't scale, who can?
If it can't provide the service, it should stop selling until it can.
This is like the AOL dialup busy signal fiasco of the mid-90's all over again. Except this time, instead of getting mad, people are making excuses for the poor, beleaguered trillion-dollar company.
>If it can't provide the service, it should stop selling until it can.
You literally cannot buy GitHub Copilot right now [1].
1: https://github.com/features/copilot/plans
With Github going up and down and Ubuntu going up and (mostly) down, there's a lot of time for intra-office sword fighting or whatever, lately. If somebody takes down Claude, everybody's going to have to just go home for the day. (https://xkcd.com/303/)
Love. Hope Github is a relic of the past inside 12 months
[delayed]
Becoming a joke is the one think that could end the GitHub monopoly.
GitHub is not a monopoly. It never has been. You've always been able to self-host or you can use gogs, gitea, gitlab, bitbucket, you get the idea
I wonder what morale is like at github. This is like gamer level hating
It's not great. Just talked to a hubber last week. They said everyone inside feels pretty dejected right now, and these posts don't help.
I feel for them -- with AI coders submitting 25 PRs within an hour of an issue being filed, GitHub bears the brunt of that along with the maintainers. That's a lot of work that gets done with each PR.
But they need to make some changes quickly.
But the amount of compute needed to serve is not very high. It's all text. The amount of bandwidth and compute needed to serve a Netflix or YouTube is far far harder and they managed just fine.
they also aren't using azure. IDK what youtube is on, but netflix has actually faced their problems and found solutions (freebsd, mostly)
I wouldn't feel too bad for them with their top-of-market comp and valuable RSU packages.
I don't believe they pay top of market, but even if they did, it's possible to make a lot of money and still feel bad when you have a sense of ownership and responsibility to the users of your service.
GitHub doesn't pay top of market.
You're right.
That being said, 300k TC for E4 is still pretty good. Plus the RSUs have gone up like 60% in the last several years.
My point is that they are compensated well. Don't need to split hairs and see if they are at Netflix level or not.
I just dont really buy the explanation though. It seems so solvable to hack a throttle or something in place, especially for non-paid plans. The cracks were also showing before AI hit the scene.
Im not saying this is the end-game solution but absolutely they could have put temporary safeguards in place while they "figure it out" if it _really_ is just AI driven slop setting their computers on fire.
The whole "anyone can submit a PR" thing has been a UX issue from day one. That probably needs to go away, and I doubt anyone would really miss it. Where Github could help is by providing a means to build trust that doesn't involve random unknown people slinging code at projects.
Any sort of trust requirement would break the entire model and cause some serious inequality.
How would a random kid in a 3rd world country ever get noticed enough to enter a trust circle, for example?
[flagged]
Whose opinions are left? The adults are all moving elsewhere.
Heck I stopped using it for projects in 2018, even before the acquisition.
My company was going to end a 6-figure YoY contract with a GitHub Actions competitor to move to GitHub, but scrapped those plans and renewed this morning. That move had been in planning for like 6 months.
this is some good life advice
and it leads to triple 9 availability (80.999%), or better
/ponder .oO( any irrational number has "infinite nines" )
/ponder .oO( i must be one of today's lucky 10000 https://xkcd.com/1053/ )
Microsoft is causing Github incidents when Azure data-centers are too hot and they need to make room for Palantir's workload.
This wouldn't surprise me at all, but I'd appreciate evidence to that effect.