Interesting post. I appreciate their candor and self-criticism, but, as a customer, I'm consistently surprised by how robust Tailscale ends up being, and how rarely I've experienced an issue that actually broke my tailnet. The sort of downtime that might keep me from accessing the admin tool or something else like that is rare enough, but my nodes have almost (?) never failed to talk to each other. Pretty great.
Caveat: I have a very small tailnet (<100 nodes). Anyone running with thousands of nodes may have a very different experience where inconvenience might be existential.
No issues using headscale and selfhosted derp servers.
Tailscale is great technology and protocol and facilitates decentralisation.
Hypergrowth is a synonym for unsustainable growth. The headline here is business breaks tech, again.
> Hypergrowth is a synonym for unsustainable growth.
No it's not. It's often a recognition that just one or two, maybe three companies will end up dominating a particular market simply due to economies of scale and network effects... and so the choice is between hypergrowth to try to attain/keep the #1 or #2 position, or else go out of business and lose all the time, money, and effort you already put into it.
Nothing whatsoever makes it unsustainable. You might be offering cheaper prices during hypergrowth -- those are unsustainable -- but then you raise prices back to sustainable levels afterwards. And consumers got to benefit from the subsidized prices, yay! The business is entirely sustainable, however.
Uber is the poster child of hypergrowth. They became profitable in 2023. And their stock price has ~doubled since. Totally sustainable.
> Hypergrowth is a synonym for unsustainable growth. The headline here is business breaks tech, again.
That just isn't true. Plenty of services do just fine after experiencing hypergrowth, and a few outages are not an example of tech breaking. That's a fairly common occurrence.
How do you selfhost your own derp servers? I am curious if it is an easy like headscale itself
The last time I looked (i.e. A couple of days ago), the documents sounded like Headscale now supports DERP [0].
My last company ran our own DERP servers to have more consistent endpoints we controlled
I use the built in derp server. I have run a standalone derp server hackily deployed for a month, it worked fine but didn't provide much benefit over the built in one. It was basically just a go package. If you're familiar with running Go code, it's straight forward to run, it's very, very light/unproductionised.
I have a todo task to integrate derp into my headscale deployment properly ("finish ansible role"), but when I picked it up last month, I noticed tailscale had release relay nodes, and they seem like they'd be better suited than dedicated derp nodes, but headscale hasn't implemented support for them yet.
tldr: not to hard to host DERP, just needs publicly facing endpoint (incl. letsencrypt) but the built in one is fine. But relay nodes look like they'll be a better option for most and I'd guess will be implemented in headscale sometime this year.
Tech is simply the reproductive organs for capitalism
So, things are working as designed for the few people that benefit
Why is the cover image for the post a cartoon 69 position?
It is not a cartoon, it is an interpretation of the position in Bauhaus style.
maybe BauHaus style is cartoony
A little reward for anyone who was affected by the outage?
Why isn't every cover image a cartoon 69 position?
Cannot unsee :)
TailScale is a VPN, and the article highlights a recent increase in user base. This is likely due to VPNs being required to access pornographic materials for residents of many US states.
Notably, it's a VPN for connecting your own devices together, so unless you're deploying a server elsewhere for access to porn it's probably not for that.
you can pay for the mullvad add-on to use their exit nodes
You can also just use Mullvad
It could also be that Tailscale users have many kids, who then also use Tailscale. Although if the header is meant to represent that, it's showing the wrong position.
> This is likely due to VPNs being required to access pornographic materials for residents of many US states.
420 is very controversial so what choice do you really have this days
That's the weed number
[deleted]
hypergrowth is very hard. First to be able to get there and then, once there, to keep up offering quality services.
Not my deviantart ass thinking hypergrowth meant something else
Would you mind explaining this comment? (I worked on deviantart for many years when it started so I'm curious, tho the servers did literally melt at one point)
the joke is porn
a specific type of porn
[deleted]
That sounds like quite the story.
pack considerably more spinning disks bought off the shelf at radioshack than you ever reasonably should in a box in a colo, turns out they generate a lot of heat, I don't recall the year I'd guess around 2004/5-ish - was a big problem, site was down for quite some time. Same year someone found out who one of our mods was and showed up to their house with a gun. Ask me about hypergrowth, I'm not sure if the DeviantART stories or the DigitalOcean stories are more wild. heh. :)
In addition hypergrowth isn't needed. Grow naturally and healthy or just be sustainable, that's okay too.
Hypergrowth can be natural. Random example but what if you designed a microblogging service and all of the sudden the biggest platform gets bought by a facist and users come flocking? You could start turning users away or you could work as fast as you can to accommodate them and make small mistakes along the way. Both of these are reasonable decisions and neither one is really wrong.
That's demand driven and organic, at least, and it's not the first thing that comes to mind with hypergrowth, it's just scale.
Instead, I think of hypergrowth as a supply-side attempt to capture a larger market in a highly inorganic way and to also capture the absurdly high valuation that comes with it. Usually through VC.
I think what you are referring to is the economical model of growth at all costs (for this I use the term blitzscaling)
I think of hyperscaling as more like growth faster than what the team can manage, for any reason.
[deleted]
> just be sustainable, that's okay too
Not if most of your company was built on investor money.
They want their pay day!
I think part of "grow naturally" assumed no investors. Just be self-sustaining with maybe some extra.
Kind of annoying to read. No, the P in CAP theorem isn’t when the client can’t connect to your unavailable service. That would be the A. Maybe it was down because of a P on your side, but don’t start blaming your downtime on network partitions between the client and your service.
Edit: your service going down and not being able to take requests from clients does not a network partition make
This is a common misunderstanding about the poorly named ‘Availability’ in CAP. Availability under CAP means that if your request reaches a non-failing node, that node still responds despite being unable to communicate with other nodes. This is distinct from SLA-style availability, which describes the uptime of the overall system.
I’m pretty sure the partition tolerance they’re referring to is the fact that the tailnet remains intact and continues to operate even when nodes can’t reach the coordination service.
A network partition between the client and server is a network partition between two nodes in a distributed system, which is the P.
If there was a network partition, but it sounded like their service just went down.
Interesting post. I appreciate their candor and self-criticism, but, as a customer, I'm consistently surprised by how robust Tailscale ends up being, and how rarely I've experienced an issue that actually broke my tailnet. The sort of downtime that might keep me from accessing the admin tool or something else like that is rare enough, but my nodes have almost (?) never failed to talk to each other. Pretty great.
Caveat: I have a very small tailnet (<100 nodes). Anyone running with thousands of nodes may have a very different experience where inconvenience might be existential.
No issues using headscale and selfhosted derp servers.
Tailscale is great technology and protocol and facilitates decentralisation.
Hypergrowth is a synonym for unsustainable growth. The headline here is business breaks tech, again.
> Hypergrowth is a synonym for unsustainable growth.
No it's not. It's often a recognition that just one or two, maybe three companies will end up dominating a particular market simply due to economies of scale and network effects... and so the choice is between hypergrowth to try to attain/keep the #1 or #2 position, or else go out of business and lose all the time, money, and effort you already put into it.
Nothing whatsoever makes it unsustainable. You might be offering cheaper prices during hypergrowth -- those are unsustainable -- but then you raise prices back to sustainable levels afterwards. And consumers got to benefit from the subsidized prices, yay! The business is entirely sustainable, however.
Uber is the poster child of hypergrowth. They became profitable in 2023. And their stock price has ~doubled since. Totally sustainable.
> Hypergrowth is a synonym for unsustainable growth. The headline here is business breaks tech, again.
That just isn't true. Plenty of services do just fine after experiencing hypergrowth, and a few outages are not an example of tech breaking. That's a fairly common occurrence.
How do you selfhost your own derp servers? I am curious if it is an easy like headscale itself
The last time I looked (i.e. A couple of days ago), the documents sounded like Headscale now supports DERP [0].
[0]: https://headscale.net/stable/setup/requirements/#ports-in-us...
It’s not super well fleshed out by Tailscale but they have a guide.
https://tailscale.com/kb/1118/custom-derp-servers
My last company ran our own DERP servers to have more consistent endpoints we controlled
I use the built in derp server. I have run a standalone derp server hackily deployed for a month, it worked fine but didn't provide much benefit over the built in one. It was basically just a go package. If you're familiar with running Go code, it's straight forward to run, it's very, very light/unproductionised.
I have a todo task to integrate derp into my headscale deployment properly ("finish ansible role"), but when I picked it up last month, I noticed tailscale had release relay nodes, and they seem like they'd be better suited than dedicated derp nodes, but headscale hasn't implemented support for them yet.
tldr: not to hard to host DERP, just needs publicly facing endpoint (incl. letsencrypt) but the built in one is fine. But relay nodes look like they'll be a better option for most and I'd guess will be implemented in headscale sometime this year.
Tech is simply the reproductive organs for capitalism
So, things are working as designed for the few people that benefit
Why is the cover image for the post a cartoon 69 position?
It is not a cartoon, it is an interpretation of the position in Bauhaus style.
maybe BauHaus style is cartoony
A little reward for anyone who was affected by the outage?
Why isn't every cover image a cartoon 69 position?
Cannot unsee :)
TailScale is a VPN, and the article highlights a recent increase in user base. This is likely due to VPNs being required to access pornographic materials for residents of many US states.
Notably, it's a VPN for connecting your own devices together, so unless you're deploying a server elsewhere for access to porn it's probably not for that.
you can pay for the mullvad add-on to use their exit nodes
You can also just use Mullvad
It could also be that Tailscale users have many kids, who then also use Tailscale. Although if the header is meant to represent that, it's showing the wrong position.
> This is likely due to VPNs being required to access pornographic materials for residents of many US states.
Same in the UK, recently.
https://www.bbc.com/news/articles/cvg5er4ewg6o
Took me a moment, but now I can't not see it.
I mean... shows where your head is at? Lol.
420 is very controversial so what choice do you really have this days
That's the weed number
hypergrowth is very hard. First to be able to get there and then, once there, to keep up offering quality services.
Not my deviantart ass thinking hypergrowth meant something else
Would you mind explaining this comment? (I worked on deviantart for many years when it started so I'm curious, tho the servers did literally melt at one point)
the joke is porn
a specific type of porn
That sounds like quite the story.
pack considerably more spinning disks bought off the shelf at radioshack than you ever reasonably should in a box in a colo, turns out they generate a lot of heat, I don't recall the year I'd guess around 2004/5-ish - was a big problem, site was down for quite some time. Same year someone found out who one of our mods was and showed up to their house with a gun. Ask me about hypergrowth, I'm not sure if the DeviantART stories or the DigitalOcean stories are more wild. heh. :)
In addition hypergrowth isn't needed. Grow naturally and healthy or just be sustainable, that's okay too.
Hypergrowth can be natural. Random example but what if you designed a microblogging service and all of the sudden the biggest platform gets bought by a facist and users come flocking? You could start turning users away or you could work as fast as you can to accommodate them and make small mistakes along the way. Both of these are reasonable decisions and neither one is really wrong.
That's demand driven and organic, at least, and it's not the first thing that comes to mind with hypergrowth, it's just scale.
Instead, I think of hypergrowth as a supply-side attempt to capture a larger market in a highly inorganic way and to also capture the absurdly high valuation that comes with it. Usually through VC.
I think what you are referring to is the economical model of growth at all costs (for this I use the term blitzscaling)
I think of hyperscaling as more like growth faster than what the team can manage, for any reason.
> just be sustainable, that's okay too
Not if most of your company was built on investor money.
They want their pay day!
I think part of "grow naturally" assumed no investors. Just be self-sustaining with maybe some extra.
Kind of annoying to read. No, the P in CAP theorem isn’t when the client can’t connect to your unavailable service. That would be the A. Maybe it was down because of a P on your side, but don’t start blaming your downtime on network partitions between the client and your service.
Edit: your service going down and not being able to take requests from clients does not a network partition make
This is a common misunderstanding about the poorly named ‘Availability’ in CAP. Availability under CAP means that if your request reaches a non-failing node, that node still responds despite being unable to communicate with other nodes. This is distinct from SLA-style availability, which describes the uptime of the overall system. I’m pretty sure the partition tolerance they’re referring to is the fact that the tailnet remains intact and continues to operate even when nodes can’t reach the coordination service.
A network partition between the client and server is a network partition between two nodes in a distributed system, which is the P.
If there was a network partition, but it sounded like their service just went down.
[dead]