> I bought a Kingston SNV3S/1000G NVMe drive. It supposedly can read and write at 6 GB/s and 5 GB/s respectively
That's a cheapo QLC drive making those numbers very conditional on several factors. Under sustained load it's not a surprise to see such drives drop to SATA speeds.
> So the limiting factor is the enclosure, that should still give me a generous 1000 MB/s.
That's capped by the USB controller, which means 512MB/s max.
I believe the USB controller still shares a bus with the NIC, which means maxing out the network connection will affect disk rw performance. You could confirm this by observing a concurrent iperf affecting fio results.
> The Raspberry Pi is notorious for not being excellent with encryption algorithms.
Well, that's specifically AES precompiles. Just like for kopia, you can benchmark and change LUKS algo for something more suitable for the platform like xchacha20: https://rr-developer.github.io/LUKS-on-Raspberry-Pi/
Author here, thanks for pointing out the mistake! The USB controller speed is indeed 4 Gbps and not 4 GB/s, I have updated the post accordingly.
> I believe the USB controller still shares a bus with the NIC, which means maxing out the network connection will affect disk rw performance. You could confirm this by observing a concurrent iperf affecting fio results.
Thanks for the pointers, I didn't know about iperf! That's something I'll likely try at some point for the sake of completeness, although I have other projects competing for my attention.
> Well, that's specifically AES precompiles. Just like for kopia, you can benchmark and change LUKS algo for something more suitable for the platform like xchacha20: https://rr-developer.github.io/LUKS-on-Raspberry-Pi/
> So I can read from my disk at 117 MB/s. We’re far from the theoretical 1000 MB/s.
I think you are confusing megabytes a second and megabit. Gigabit speed is approximately 125 Megabytes per second. This is close to the speed you got.
Yes but not there, later they get 350MB/s with the same setup.
> I put that drive an ICY BOX IB-1817M-C31 enclosure, with a maximum theoretical speed of 1000 MB/s.
Checks out, it has a 10Gb/s USB port.
The mistake is
> the USB controller of the Pi has a bandwidth of 4GB/s shared across all 4 ports
It's actually 4Gb/s = 512MB/s
USB 3 is 5 Gbps, but real world never exceeds 400 MB/sec. 350 MB/sec isn't bad for a single USB 3 port.
This is absolutely right, thanks for pointing it out! The post has been updated to reflect that.
When you add up the component costs for RPi with NVMe, suitable case, cooling and power supply, it is not much cheaper than a mini PC which generally outperforms the RPi.
I use several RPis of various models. But there are times when even an old Intel/AMD laptop is a more suitable solution.
For me a huge win for the pi is the power consumption vs a cheap mini PC refurbished from Dell.
I’m not sure what electricity costs where you live, but my calculations tell me I’d have to run an intel n4000 for 5+ years before I break even compared to buying a CanaKit rpi 5.
An old laptop has a built in UPS too!
a UPS has a battery that is safe under failure, a laptop usually doesn't.
Eh? I'm using an old MacBook as a home server; I kind of naively assumed that there was a whole load of protection circuitry and firmware controls in there that would turn it off before it expands or the magic smoke comes out.
Do you have experience otherwise?
"A beefy VPS (16 vCPU, 48 GB of RAM, 1 TB SSD) in a German datacenter
A Raspberry Pi 4 (8 GB of RAM) in my living room"
While I loved the sleuthing, I did not get why the comparison had to be between thise extreme ends of the spectrum.
Why not test a small extra cheap VPS, or, as the author does in the end, a somewhat normal PC at home?
Author here, that's a great question. I only host a handful of services on the VPS, including Nextcloud. I was mostly interested in storage, not so much in compute power.
Netcup offered VPS with 1 TB of storage for €30 / month back when I subscribed. The server has always been oversized for my needs, but that was the most financially interesting back then.
I recently switched to another ISP at home that gives me a fixed IP. I decided to move my services at home to shave off the VPS costs. Since I already had a Raspberry Pi 4, I only had to buy a SSD and an enclosure.
Just wanted to add: Pi 5 does 1-1.5GB/s AES per core. And about 1GB/s per core for ChaPoly. This can be verified with `openssl speed`.
This article has nothing to do with the speed of a Raspberry Pi as a server, unless the purpose of this server is to run "kopia".
I ran an 8 gig Raspberry Pi 4 with mirrored 8 TB spinning rust disks for quite a while. Performance is very good, once you stop playing with all the meta tasks and run, you know, server stuff. I ultimately switched to another system not because of CPU, but because of memory. SearXNG [1] likes a lot of memory.
> I bought a Kingston SNV3S/1000G NVMe drive. It supposedly can read and write at 6 GB/s and 5 GB/s respectively
That's a cheapo QLC drive making those numbers very conditional on several factors. Under sustained load it's not a surprise to see such drives drop to SATA speeds.
> So the limiting factor is the enclosure, that should still give me a generous 1000 MB/s.
That's capped by the USB controller, which means 512MB/s max.
I believe the USB controller still shares a bus with the NIC, which means maxing out the network connection will affect disk rw performance. You could confirm this by observing a concurrent iperf affecting fio results.
> The Raspberry Pi is notorious for not being excellent with encryption algorithms.
Well, that's specifically AES precompiles. Just like for kopia, you can benchmark and change LUKS algo for something more suitable for the platform like xchacha20: https://rr-developer.github.io/LUKS-on-Raspberry-Pi/
Author here, thanks for pointing out the mistake! The USB controller speed is indeed 4 Gbps and not 4 GB/s, I have updated the post accordingly.
> I believe the USB controller still shares a bus with the NIC, which means maxing out the network connection will affect disk rw performance. You could confirm this by observing a concurrent iperf affecting fio results.
Thanks for the pointers, I didn't know about iperf! That's something I'll likely try at some point for the sake of completeness, although I have other projects competing for my attention.
> Well, that's specifically AES precompiles. Just like for kopia, you can benchmark and change LUKS algo for something more suitable for the platform like xchacha20: https://rr-developer.github.io/LUKS-on-Raspberry-Pi/
Absolutely, that disk was encrypted using xchacha20. I used the excellent sdm project (https://github.com/gitbls/sdm) to make my Raspi boot from an encrypted disk. They have great documentation pointing out that xchacha20 should be used on the Pi 4 and earlier (https://github.com/gitbls/sdm/blob/master/Docs/Disk-Encrypti...)
> So I can read from my disk at 117 MB/s. We’re far from the theoretical 1000 MB/s.
I think you are confusing megabytes a second and megabit. Gigabit speed is approximately 125 Megabytes per second. This is close to the speed you got.
Yes but not there, later they get 350MB/s with the same setup.
> I put that drive an ICY BOX IB-1817M-C31 enclosure, with a maximum theoretical speed of 1000 MB/s.
Checks out, it has a 10Gb/s USB port.
The mistake is
> the USB controller of the Pi has a bandwidth of 4GB/s shared across all 4 ports
It's actually 4Gb/s = 512MB/s
USB 3 is 5 Gbps, but real world never exceeds 400 MB/sec. 350 MB/sec isn't bad for a single USB 3 port.
This is absolutely right, thanks for pointing it out! The post has been updated to reflect that.
When you add up the component costs for RPi with NVMe, suitable case, cooling and power supply, it is not much cheaper than a mini PC which generally outperforms the RPi.
I use several RPis of various models. But there are times when even an old Intel/AMD laptop is a more suitable solution.
For me a huge win for the pi is the power consumption vs a cheap mini PC refurbished from Dell.
I’m not sure what electricity costs where you live, but my calculations tell me I’d have to run an intel n4000 for 5+ years before I break even compared to buying a CanaKit rpi 5.
An old laptop has a built in UPS too!
a UPS has a battery that is safe under failure, a laptop usually doesn't.
Eh? I'm using an old MacBook as a home server; I kind of naively assumed that there was a whole load of protection circuitry and firmware controls in there that would turn it off before it expands or the magic smoke comes out.
Do you have experience otherwise?
"A beefy VPS (16 vCPU, 48 GB of RAM, 1 TB SSD) in a German datacenter A Raspberry Pi 4 (8 GB of RAM) in my living room"
While I loved the sleuthing, I did not get why the comparison had to be between thise extreme ends of the spectrum.
Why not test a small extra cheap VPS, or, as the author does in the end, a somewhat normal PC at home?
Author here, that's a great question. I only host a handful of services on the VPS, including Nextcloud. I was mostly interested in storage, not so much in compute power.
Netcup offered VPS with 1 TB of storage for €30 / month back when I subscribed. The server has always been oversized for my needs, but that was the most financially interesting back then.
I recently switched to another ISP at home that gives me a fixed IP. I decided to move my services at home to shave off the VPS costs. Since I already had a Raspberry Pi 4, I only had to buy a SSD and an enclosure.
Just wanted to add: Pi 5 does 1-1.5GB/s AES per core. And about 1GB/s per core for ChaPoly. This can be verified with `openssl speed`.
This article has nothing to do with the speed of a Raspberry Pi as a server, unless the purpose of this server is to run "kopia".
I ran an 8 gig Raspberry Pi 4 with mirrored 8 TB spinning rust disks for quite a while. Performance is very good, once you stop playing with all the meta tasks and run, you know, server stuff. I ultimately switched to another system not because of CPU, but because of memory. SearXNG [1] likes a lot of memory.
[1] https://docs.searxng.org
Get a Wyse 5070 or anything like that instead of a raspberry pi as a stable small server. Used they are basically the same price point.
I have 3 running. One got a RAM and SSD upgrade and serves as my desktop/media thingy/TV and mobile audio lap in my camper.