You might have even better precision if you stay away from CPU0 and also set idle=poll in your kernel command line. Lots of things (including other interrupts) often land on CPU0. It would not be my first choice for something where I wanted high timing precision.
I came here to post this. We make a lot of the same sorts of optimizations for our OS distro (debian based) -- disabling frequency scaling, core pinning, etc. Critically, CPU0 has a bunch of stuff you cannot push, and you're better off with using one of the other cores as an isolated island.
This is what the scheduler latency looks like on our isolated core:
# Total: 000300000
# Min Latencies: 00001
# Avg Latencies: 00005
# Max Latencies: 00059
# Histogram Overflows: 00000
(those are uS!)
Very cool. What are you running on it? What's your use case?
Why not put a resistor (for heating) and a bit of foam insulation on the crystal?
This is way more direct than spacebar heating.
You could also add a transistor attached to the resistor and a GPIO and use the clock drift as a proxy for temperature. PID is probably enough but since you have a 24 hour cycle you could calculate a baseline heating schedule.
It's an SBC-scale OXCO. I half wonder if adding a larger heatsink, or even putting thermal mass around the existing oscillator could also help, or if the heating is more localized in the PCB itself.
Always fun new things to learn when doing something "simple" like setting up an NTP server!
Flirc makes a metal Pi case where the CPU is pressed against the metal body of the case, resulting in a huge thermal mass for passive cooling. I have a bunch of them and it works very well. No fan necessary.
I was thinking it might be nice to add some insulation around some the pi's enclosure, to reduce its cooling significantly. A little bit to tamp down any potential rapid fluctuations in the room's temperature (if someone opens a window, steps out of the bath, whatever). But more so that it could save a watt or two of power, by having the time-burner cores working much less.
You're right that this is a over-controller oscillator. The goal generally with ovens is to keep heat! (To an extent of course.)
> I half wonder if adding a larger heatsink, or even putting thermal mass around the existing oscillator could also help, or if the heating is more localized in the PCB itself.
That would likely make it worse. The trick here is that the other cores are running at essentially their maximum temperature and and will dynamically reduce their clockspeed if required to keep from going above that limit. In essence, the environment becomes actively temperature controlled. If the ambient heat goes higher, the cores clock lower, if it gets colder the cores clock higher (up to a point).
If you add too much heat dissipation, the total power used by those cores might not be enough to keep well above ambient.
Extreme power dissipation would keep temperature stable so that this whole setup might not be needed, though.
Author should experiment with liquid nitrogen ;) [1]
The timing crystals don't work better when colder but worse. That's why they are heated in high end time appliances, not cooled.
Isn't the issue here temperature stability? (Also, humour)
Right, and they are heated because a hot wire is much simpler than a fridge.
Couldn't you model the effect of temperature on clock drift and try to factor that in dynamically (e.g. using a temperature sensor) instead of burning CPU unnecessarily?
I wonder about using an RPI Pico for this instead, using the Pico's synchronous PIO gizmo to intercept the PPS pulses.
It's the good old OCXO - Oven Controlled Crystal Oscillator. But the heating element is the CPU. Fucking hilarious.
There was a paper on this in 2022. Not sure if it's used in production or not.
That's a neat software solution. My first inclination would be to grab a soldering iron and replace the crystal with either a TCXO or a socket to provide an external clock disciplined to the 1PPS.
Wouldn't a temperature compensating algorithm be just as effective?
[deleted]
The related question is:
Is the Pi going the Pentium 4 route?
What is this even supposed to mean? What's "the Pentium 4 route"?
I'm an old fart :)
Intel tried to scale frequency up with the Pentium 4 in the name of performance, and it ended up extremely hot and power hungry. Just like some high end CPUs now, but then it applied to every model from Intel.
I suppose you don't remember when a Raspberry Pi could run fine even without a heatsink, let alone active cooling. That's more recent than the Pentium 4.
I love this. Chasing perfection for perfection alone.
Amazing project, great write-up. Would love to see a temperature graph as well! I'm wondering how good the PID controller here is working.
For future improvements, a cheap but effective win might be to put a temperature sensor on the oscillator (or two or three in various places). And use that to drive the PID loop.
Even if just experimental & not long term, it would be nice to have data on how strong the correlation is between the cpu & oscillator temperatures. To see their difference and how much that changes over time. Another graph! CPU vs txco (vs ambient?) temperatures over time.
> put a temperature sensor on the oscillator
At that point, couldn't we just use the temperature value to compensate for the drift?
What's the point in reading posts like this when the solution "they" came up with is basically, "tell Claude to make a script which does whatever"? I read blog posts to read thoughts from people, not computers
Are you similarly frustrated that he didn't sit there 24/7, heating the oscillator with a small lighter when needed, but automated it instead? Why would this be more interesting for you if he'd written the script himself?
> Are you similarly frustrated that he didn't sit there 24/7, heating the oscillator with a small lighter when needed, but automated it instead?
No
> Why would this be more interesting for you if he'd written the script himself?
Was I unclear? I read blog posts to read thoughts from humans, not from computers.
You might have even better precision if you stay away from CPU0 and also set idle=poll in your kernel command line. Lots of things (including other interrupts) often land on CPU0. It would not be my first choice for something where I wanted high timing precision.
I came here to post this. We make a lot of the same sorts of optimizations for our OS distro (debian based) -- disabling frequency scaling, core pinning, etc. Critically, CPU0 has a bunch of stuff you cannot push, and you're better off with using one of the other cores as an isolated island.
This is what the scheduler latency looks like on our isolated core:
# Total: 000300000 # Min Latencies: 00001 # Avg Latencies: 00005 # Max Latencies: 00059 # Histogram Overflows: 00000
(those are uS!)
Very cool. What are you running on it? What's your use case?
Why not put a resistor (for heating) and a bit of foam insulation on the crystal?
This is way more direct than spacebar heating.
You could also add a transistor attached to the resistor and a GPIO and use the clock drift as a proxy for temperature. PID is probably enough but since you have a 24 hour cycle you could calculate a baseline heating schedule.
It's an SBC-scale OXCO. I half wonder if adding a larger heatsink, or even putting thermal mass around the existing oscillator could also help, or if the heating is more localized in the PCB itself.
Always fun new things to learn when doing something "simple" like setting up an NTP server!
Flirc makes a metal Pi case where the CPU is pressed against the metal body of the case, resulting in a huge thermal mass for passive cooling. I have a bunch of them and it works very well. No fan necessary.
I was thinking it might be nice to add some insulation around some the pi's enclosure, to reduce its cooling significantly. A little bit to tamp down any potential rapid fluctuations in the room's temperature (if someone opens a window, steps out of the bath, whatever). But more so that it could save a watt or two of power, by having the time-burner cores working much less.
You're right that this is a over-controller oscillator. The goal generally with ovens is to keep heat! (To an extent of course.)
> I half wonder if adding a larger heatsink, or even putting thermal mass around the existing oscillator could also help, or if the heating is more localized in the PCB itself.
That would likely make it worse. The trick here is that the other cores are running at essentially their maximum temperature and and will dynamically reduce their clockspeed if required to keep from going above that limit. In essence, the environment becomes actively temperature controlled. If the ambient heat goes higher, the cores clock lower, if it gets colder the cores clock higher (up to a point).
If you add too much heat dissipation, the total power used by those cores might not be enough to keep well above ambient.
Extreme power dissipation would keep temperature stable so that this whole setup might not be needed, though.
Author should experiment with liquid nitrogen ;) [1]
[11] https://www.xda-developers.com/liquid-nitrogen-cooling-raspb...
The timing crystals don't work better when colder but worse. That's why they are heated in high end time appliances, not cooled.
Isn't the issue here temperature stability? (Also, humour)
Right, and they are heated because a hot wire is much simpler than a fridge.
Couldn't you model the effect of temperature on clock drift and try to factor that in dynamically (e.g. using a temperature sensor) instead of burning CPU unnecessarily?
I wonder about using an RPI Pico for this instead, using the Pico's synchronous PIO gizmo to intercept the PPS pulses.
It's the good old OCXO - Oven Controlled Crystal Oscillator. But the heating element is the CPU. Fucking hilarious.
There was a paper on this in 2022. Not sure if it's used in production or not.
https://www.usenix.org/conference/nsdi22/presentation/najafi
That's a neat software solution. My first inclination would be to grab a soldering iron and replace the crystal with either a TCXO or a socket to provide an external clock disciplined to the 1PPS.
Wouldn't a temperature compensating algorithm be just as effective?
The related question is:
Is the Pi going the Pentium 4 route?
What is this even supposed to mean? What's "the Pentium 4 route"?
I'm an old fart :)
Intel tried to scale frequency up with the Pentium 4 in the name of performance, and it ended up extremely hot and power hungry. Just like some high end CPUs now, but then it applied to every model from Intel.
I suppose you don't remember when a Raspberry Pi could run fine even without a heatsink, let alone active cooling. That's more recent than the Pentium 4.
I love this. Chasing perfection for perfection alone.
Amazing project, great write-up. Would love to see a temperature graph as well! I'm wondering how good the PID controller here is working.
For future improvements, a cheap but effective win might be to put a temperature sensor on the oscillator (or two or three in various places). And use that to drive the PID loop.
Even if just experimental & not long term, it would be nice to have data on how strong the correlation is between the cpu & oscillator temperatures. To see their difference and how much that changes over time. Another graph! CPU vs txco (vs ambient?) temperatures over time.
> put a temperature sensor on the oscillator
At that point, couldn't we just use the temperature value to compensate for the drift?
What's the point in reading posts like this when the solution "they" came up with is basically, "tell Claude to make a script which does whatever"? I read blog posts to read thoughts from people, not computers
Are you similarly frustrated that he didn't sit there 24/7, heating the oscillator with a small lighter when needed, but automated it instead? Why would this be more interesting for you if he'd written the script himself?
> Are you similarly frustrated that he didn't sit there 24/7, heating the oscillator with a small lighter when needed, but automated it instead?
No
> Why would this be more interesting for you if he'd written the script himself?
Was I unclear? I read blog posts to read thoughts from humans, not from computers.
Well, I guess your era of reading is over, sadly.