A few years ago, at my company, we would get random TPM crashes every few months on all our machines. You'd be working and the TPM would just disappear and then any apps that rely on it for key retrieval would error out. Even worse, since the TPM chip is always running, neither a reboot nor a shutdown would fix it -- you literally had to pull the plug.
This went on for months. Then one day we had a power outage. Two months later, every single machine failed at the same time. I checked the logs and it was 49 days and few hours since that outage. It didn't take me too long to figure out what the underlying programming error inside the TPM was. At least we could then describe exactly what the problem was to our PC vendor.
So what was the programming error in the TPM?
Something breaking after 49.7 days is a classic. Someone counted milliseconds since start with a 32 bit unsigned int and some code assumed it couldn't wrap.
[deleted]
49 days is a bit under 2^32 milliseconds... So unsigned int overflow?
Wow, someone in the github comments[1] noticed that one of the bug numbers assigned internally for the issue matches to the day the number of days the driver would stay up.
* Do this with "<0" and ">=0" to only test the sign of the result. A
* good compiler would generate better code (and a really good compiler
* wouldn't care). Gcc is currently neither.
It's funny the love-hate relationship the Linux kernel has with GCC. It's the only supported compiler[1], and yet...
[1] can Clang fully compile Linux yet? I haven't followed the updates in a while.
To be fair this comment predates git history (before 2005) when GCC wasn't a very good compiler. The kernel developers at one point were sticking with a specific version of GCC because later versions would miscompile the kernel. Clang didn't exist then.
Do I understand it correctly that the logic is that if timestamp B is above timestamp A, but the difference is more than half of the unsigned range, B is considered to happen before A?
Yes. When the timestamps wrap it's fundamentally ambiguous, but this will be correct unless the timestamps are very far apart (and the failure mode is more benign: a really long time difference being considered shorter is better than all time differences being considered zero after the timestamp wraps).
Crazy, so if I understand correctly, something with B200s and nvlink is causing issues where after 66 days and 12 hours of uptime, nvidia-smi and other jobs start failing, timing out, then once you restart the cluster it starts working again.
They suspect jobs will work if you only use 1 B200, but one person power cycled so wasn’t able to test it. Hopefully they won’t have to wait another 66 days for further troubleshooting.
Some 32-bit counter somewhere used when in NVLINK overflows?
66 days + 12 hours are 5,745,600,000,000,000 ns. The log2 of this is 52.351...
Javascript and some other languages only have integer precision up to 52 bits then they switch to floating point.
Curious.
It's 32 bits of milliseconds, right? Hm, no, that would overflow much sooner (49.7 days).
It's a uint32_t of 750 Hz "jiffies", which does overflow at ~66 days.
While that seems like a convincing explanation, 750Hz is a rather odd value to use for a timer, and more importantly the overflow would be at 66d6h43m43s instead of the reported ~66d12h.
Isn't 32bit counter 49 days? Assuming that one was counting milliseconds, at least.
Only remember that because that's the limit for Windows 95…
100ns intervals. My favorite part of that story is how long after Windows 95 was released before anybody discovered the bug.
That's because people actually powered off their computer after work/leisure sessions. Someone on an unlimited night dial-up could had discovered it well before "anybody" but it's not like there was a built-in function to actually send a crash report to Redmond.
I wonder if the process to debugging this is just to search for what power of 2 times a time unit equals ~66 days
I think it's an overflow of a scaled counter.
Also, who else immediately noticed the AI-generated comment?
66 days 14 hours and 24 minutes (66.6 days) would have been a far more diabolical hang...
NVLink postRxDetLinkMask errors show up right before the hang. Has anyone captured a bug report or stack trace while nvidia-smi is stuck to see what it's blocking on?
> we were hit with this on a 256 gpu b200 cluster -- at day 66 all our jobs started randomly failing
ouch
Known pretty longstanding issue w nvidia still unresolved afaik
This is only very tangentially related, but I got flashbacks to a time where we had dozens of edge/IoT raspberry pi devices with completely unupgradeable kernels with a bug that would make the whole USB stack shut down after "roughly a week" (7-9 days) of uptime. Once it got shut down, the only way to fix it was to do a full restart, and, at the time, we couldn't really be restarting those devices (not even at night).
This means that every single device would seemingly randomly completely break: touchscreen, keyboard, modems, you name it. Everything broke. And since the modem was part of it, we would lose access to the device — very hard to solve because maintenance teams were sometimes hours (& flights!) away.
It seemed to happen at random, and it was very hard to trace it down because we were also gearing up for an absolutely massive (hundreds of devices, and then a couple of months later, thousands) launch, and had pretty much every conceivable issue thrown at us, from faulty USB hubs, broken modems (which would also kill the USB hub if they pulled too much power), and I'm sure I've forgotten a bunch of other issues.
Plus, since the problem took a week to manifest, we couldn't really iterate on fixes quickly - after deploying a "potential fix", we'd have to wait a whole week to actually see if it worked. I can vividly remember the joy I had when I managed to get the issue to consistently happen only in the span of 2 hours instead of a week. I had no idea _why_, but at least I could now get serviceable feedback loops.
Eventually, after trying to mess with every variable we could, and isolating this specific issue from the other ones, we somehow figured out that the issue was indeed a bug in the kernel, or at least in one of its drivers: https://github.com/raspberrypi/linux/issues/5088 . We had many serial ports and a pattern of opening and closing them which triggered the issue. Upgrading the kernel was impossible due to a specific vendor lock-in, and we had to fix live devices and ship hundreds of them in less than a month.
In the end, we managed to build several layers on top of this unpatchable ever-growing USB-incapacitating bug: (i) we changed our serial port access patterns to significantly reduce the frequency of crashes; (ii) we adjusted boot parameters to make it much harder to trigger (aka "throw more memory at the memory leak"); (iii) we built a system that proactively detected the issue and triggered a USB reset in a very controlled fashion (this would sometimes kill the network of the device for a while, but we had no choice!); (iv) if, for some reason, all else failed, a watchdog would still reboot the system (but we really _really_ _reaaaally_ didn't want this to happen).
In a way, even though these issues suck, it's when we are faced with them that we really grow. We need to grab our whole troubleshooting arsenal, do things that would otherwise feel "wrong" or "inelegant", and push through the issues. Just thinking back to that period, I'm engulfed by a mix of gratitude for how much I learned, and an uneasy sense of dread (what if next time I won't be able to figure it out)?
Even National Instruments had this type of bug in their nivisa driver, that powers a good portion of lab and test equipment of the world. Every 31 days our test equipment would stop working, which happens to be the overflow of one of the windows timers. was also one of the fasted bug fix updates I ever saw, after reporting it!
I've always been sceptical of the modern tendency of throwing powerful hardware at every embedded projects. In most cases good old atmel AVR or even 8051 would suffice.
A week? I've had some Pis lose usb in 1-2 days. Fortunately we could afford to make them self restart every couple hours.
a pet peeve of mine, (along with people brigading on issues/threads e.g. posting them to unrelated news sites... op....) is woefully incorrect language.
> at day 66 all our jobs started randomly failing
if there's a definable pattern, you can call it unpredictabily, but you can't call it randomly.
They've meant something like "arbitrary", in its "without any good/justifiable reason" sense. The word "random" is also used in this sense, especially when talking about human-made decisions.
Unexpectedly is probably what they meant
IMHO, what they said means that on day 65 all jobs work, on day 66, jobs work or don't, seemingly at random.
But what they seem to be indicating is that all jobs fail on day 66. There's no randomness in evidence.
It's from the perspective of not knowing anything about the issue. It would look like jobs failing randomly one day when everything was fine the day before. Not hard to understand.
Seems quite predictable given the others in the bug report encountering the same.
A few years ago, at my company, we would get random TPM crashes every few months on all our machines. You'd be working and the TPM would just disappear and then any apps that rely on it for key retrieval would error out. Even worse, since the TPM chip is always running, neither a reboot nor a shutdown would fix it -- you literally had to pull the plug.
This went on for months. Then one day we had a power outage. Two months later, every single machine failed at the same time. I checked the logs and it was 49 days and few hours since that outage. It didn't take me too long to figure out what the underlying programming error inside the TPM was. At least we could then describe exactly what the problem was to our PC vendor.
So what was the programming error in the TPM?
Something breaking after 49.7 days is a classic. Someone counted milliseconds since start with a 32 bit unsigned int and some code assumed it couldn't wrap.
49 days is a bit under 2^32 milliseconds... So unsigned int overflow?
Wow, someone in the github comments[1] noticed that one of the bug numbers assigned internally for the issue matches to the day the number of days the driver would stay up.
1: https://github.com/NVIDIA/open-gpu-kernel-modules/issues/971...
You mean number of seconds, but yes, I think everyone looking at this would be converting units to see if there was a particular boundary being met.
Timestamps should NOT be compared like this. Exactly this is why time_before() or time_after() exist.
https://elixir.bootlin.com/linux/v6.15.7/source/include/linu...
Offtopic...
It's funny the love-hate relationship the Linux kernel has with GCC. It's the only supported compiler[1], and yet...[1] can Clang fully compile Linux yet? I haven't followed the updates in a while.
To be fair this comment predates git history (before 2005) when GCC wasn't a very good compiler. The kernel developers at one point were sticking with a specific version of GCC because later versions would miscompile the kernel. Clang didn't exist then.
GCC is a different beast and far better nowadays.
Yes it can [1].
https://docs.kernel.org/kbuild/llvm.html
Do I understand it correctly that the logic is that if timestamp B is above timestamp A, but the difference is more than half of the unsigned range, B is considered to happen before A?
Yes. When the timestamps wrap it's fundamentally ambiguous, but this will be correct unless the timestamps are very far apart (and the failure mode is more benign: a really long time difference being considered shorter is better than all time differences being considered zero after the timestamp wraps).
Crazy, so if I understand correctly, something with B200s and nvlink is causing issues where after 66 days and 12 hours of uptime, nvidia-smi and other jobs start failing, timing out, then once you restart the cluster it starts working again.
They suspect jobs will work if you only use 1 B200, but one person power cycled so wasn’t able to test it. Hopefully they won’t have to wait another 66 days for further troubleshooting.
Some 32-bit counter somewhere used when in NVLINK overflows?
66 days + 12 hours are 5,745,600,000,000,000 ns. The log2 of this is 52.351...
Javascript and some other languages only have integer precision up to 52 bits then they switch to floating point.
Curious.
It's 32 bits of milliseconds, right? Hm, no, that would overflow much sooner (49.7 days).
It's a uint32_t of 750 Hz "jiffies", which does overflow at ~66 days.
While that seems like a convincing explanation, 750Hz is a rather odd value to use for a timer, and more importantly the overflow would be at 66d6h43m43s instead of the reported ~66d12h.
Isn't 32bit counter 49 days? Assuming that one was counting milliseconds, at least.
Only remember that because that's the limit for Windows 95…
100ns intervals. My favorite part of that story is how long after Windows 95 was released before anybody discovered the bug.
That's because people actually powered off their computer after work/leisure sessions. Someone on an unlimited night dial-up could had discovered it well before "anybody" but it's not like there was a built-in function to actually send a crash report to Redmond.
https://i.sstatic.net/p9hUgGfg.png
I wonder if the process to debugging this is just to search for what power of 2 times a time unit equals ~66 days
I think it's an overflow of a scaled counter.
Also, who else immediately noticed the AI-generated comment?
66 days 14 hours and 24 minutes (66.6 days) would have been a far more diabolical hang...
NVLink postRxDetLinkMask errors show up right before the hang. Has anyone captured a bug report or stack trace while nvidia-smi is stuck to see what it's blocking on?
> we were hit with this on a 256 gpu b200 cluster -- at day 66 all our jobs started randomly failing
ouch
Known pretty longstanding issue w nvidia still unresolved afaik
This is only very tangentially related, but I got flashbacks to a time where we had dozens of edge/IoT raspberry pi devices with completely unupgradeable kernels with a bug that would make the whole USB stack shut down after "roughly a week" (7-9 days) of uptime. Once it got shut down, the only way to fix it was to do a full restart, and, at the time, we couldn't really be restarting those devices (not even at night).
This means that every single device would seemingly randomly completely break: touchscreen, keyboard, modems, you name it. Everything broke. And since the modem was part of it, we would lose access to the device — very hard to solve because maintenance teams were sometimes hours (& flights!) away.
It seemed to happen at random, and it was very hard to trace it down because we were also gearing up for an absolutely massive (hundreds of devices, and then a couple of months later, thousands) launch, and had pretty much every conceivable issue thrown at us, from faulty USB hubs, broken modems (which would also kill the USB hub if they pulled too much power), and I'm sure I've forgotten a bunch of other issues.
Plus, since the problem took a week to manifest, we couldn't really iterate on fixes quickly - after deploying a "potential fix", we'd have to wait a whole week to actually see if it worked. I can vividly remember the joy I had when I managed to get the issue to consistently happen only in the span of 2 hours instead of a week. I had no idea _why_, but at least I could now get serviceable feedback loops.
Eventually, after trying to mess with every variable we could, and isolating this specific issue from the other ones, we somehow figured out that the issue was indeed a bug in the kernel, or at least in one of its drivers: https://github.com/raspberrypi/linux/issues/5088 . We had many serial ports and a pattern of opening and closing them which triggered the issue. Upgrading the kernel was impossible due to a specific vendor lock-in, and we had to fix live devices and ship hundreds of them in less than a month.
In the end, we managed to build several layers on top of this unpatchable ever-growing USB-incapacitating bug: (i) we changed our serial port access patterns to significantly reduce the frequency of crashes; (ii) we adjusted boot parameters to make it much harder to trigger (aka "throw more memory at the memory leak"); (iii) we built a system that proactively detected the issue and triggered a USB reset in a very controlled fashion (this would sometimes kill the network of the device for a while, but we had no choice!); (iv) if, for some reason, all else failed, a watchdog would still reboot the system (but we really _really_ _reaaaally_ didn't want this to happen).
In a way, even though these issues suck, it's when we are faced with them that we really grow. We need to grab our whole troubleshooting arsenal, do things that would otherwise feel "wrong" or "inelegant", and push through the issues. Just thinking back to that period, I'm engulfed by a mix of gratitude for how much I learned, and an uneasy sense of dread (what if next time I won't be able to figure it out)?
Even National Instruments had this type of bug in their nivisa driver, that powers a good portion of lab and test equipment of the world. Every 31 days our test equipment would stop working, which happens to be the overflow of one of the windows timers. was also one of the fasted bug fix updates I ever saw, after reporting it!
I've always been sceptical of the modern tendency of throwing powerful hardware at every embedded projects. In most cases good old atmel AVR or even 8051 would suffice.
A week? I've had some Pis lose usb in 1-2 days. Fortunately we could afford to make them self restart every couple hours.
a pet peeve of mine, (along with people brigading on issues/threads e.g. posting them to unrelated news sites... op....) is woefully incorrect language.
> at day 66 all our jobs started randomly failing
if there's a definable pattern, you can call it unpredictabily, but you can't call it randomly.
They've meant something like "arbitrary", in its "without any good/justifiable reason" sense. The word "random" is also used in this sense, especially when talking about human-made decisions.
Unexpectedly is probably what they meant
IMHO, what they said means that on day 65 all jobs work, on day 66, jobs work or don't, seemingly at random.
But what they seem to be indicating is that all jobs fail on day 66. There's no randomness in evidence.
It's from the perspective of not knowing anything about the issue. It would look like jobs failing randomly one day when everything was fine the day before. Not hard to understand.
Seems quite predictable given the others in the bug report encountering the same.
*China specific code leaked into mainline.