Just like that author, many years ago, I went through the process of understanding the DEFLATE compression standard and producing a short and concise decompressor for gzip+DEFLATE. Here are the resources I published as a result of that exploration:
His also omits CRC, which is part of the 25k lines, no --fast/--best/etc, missing some output formats, and so on. I'm sure the 25k includes a lot of bloat, but the comparison is odd. Comparing to your list would make much more sense.
Crc32 can be written in handful lines of code. Although it'd be better to use the vector instruction set - e.g. AVX when available.
I would expect a CRC to add a negligible number of lines of code. The reason that production-grade decompressors are tens of thousands of LOC is likely attributable to extreme manual optimization. For example, I wouldn't be surprised if a measurable fraction of those lines are actually inline assembly.
True. A most basic CRC implementation is about 7 lines of code: (presented in Java to avoid some C/C++ footguns)
int crc32(byte[] data) {
int crc = ~0;
for (byte b : data) {
crc ^= b & 0xFF;
for (int i = 0; i < 8; i++)
crc = (crc >>> 1) ^ ((crc & 1) * 0xEDB88320);
}
return ~crc;
}
Or smooshed down slightly (with caveats):
int crc32(byte[] data) {
int crc = ~0;
for (int i = 0; i < data.length * 8; i++) {
crc ^= (data[i / 8] >> (i % 8)) & 1;
crc = (crc >>> 1) ^ ((crc & 1) * 0xEDB88320);
}
return ~crc;
}
That's java code, though... bit weird, esp. i % 8 (which is just i & 7). The compiler should be able to optimize it since 'i' is guaranteed to be non-negative, still awkward.
Java CRC32 nowadays uses intrinsics and avx128 for crc32.
Doesn't need to be inline assembly, just pre-encoded lookup tables and intrinsics-based vectorized CRC alone will add quite a lot of code. Most multi-platform CRC algorithms tend to have at least a few paths for byte/word/dword at a time, hardware CRC, and hardware GF(2) multiply. It's not really extreme optimization, just better algorithms to match better hardware capabilities.
The Huffman decoding implementation is also bigger in production implementations for both speed and error checking. Two Huffman trees need to be exactly complete except in the special case of a single code, and in most cases they are flattened to two-level tables for speed (though the latest desktop CPUs have enough L1 cache to use single-level).
Finally, the LZ copy typically has special cases added for using wider than byte copies for non-overlapping, non-wrapping runs. This is a significant decoding speed optimization.
Yes, there's subdirs with language bindings for many non-C langs, an examples folder with example C code, win32 specific C code, test code, etc.
More reasons it's an odd comparison.
gzip also contains a significant amount of compatibility code for different platforms.
But probably without any error checking.
Feels like Rust culture inherited "throw and forget" as an error handling "strategy" from Java
Sigh.
Hehe, why "probably"? It says "250 lines" right there in the subject. Surely one can skim the single file of code (https://github.com/ieviev/mini-gzip/blob/main/src/main.rs) and offer criticism that isn't based on hypotheticals?
Anyway, I skimmed the file for you this time, and basically you're either correct or wrong, depending on your definition of "error checking." The code handles error conditions by aborting the process. Seeing as it's a standalone CLI program and not a library meant for reuse, safely shutting down with a meaningful message sounds like fair game to me.
This is an educational project. Not something for production. The article even says so!
You can leave the snide comments about “Rust culture” (whatever that is) out next time.
Why people ascribe error handling practices to languages is baffling. What language doesn't allow punting error handling until later? Even Haskell has "panic" functionality that fudges the type constraints to allow this.
Another dev who doesn’t show respect to what has been done and expect a particular language will do wonders for him. Also I don’t see this is much better in term of readability.
Where do you see the lack of respect? The author wanted to learn how gzip works and chose to implement it in a language they like to do so. As a learning tool, not because the world needs another gzip decompressor.
> so i wrote a gzip decompressor from scratch
After skimming through the author's Rust code, it appears to be a fairly straightforward port of puff.c (included in the zlib source): https://github.com/madler/zlib/blob/develop/contrib/puff/puf...
Just like that author, many years ago, I went through the process of understanding the DEFLATE compression standard and producing a short and concise decompressor for gzip+DEFLATE. Here are the resources I published as a result of that exploration:
* https://www.nayuki.io/page/deflate-specification-v1-3-html
* https://www.nayuki.io/page/simple-deflate-decompressor
* https://github.com/nayuki/Simple-DEFLATE-decompressor
> twenty five thousand lines of pure C not counting CMake files. ...
Keep in mind this is also 31 years of cruft and lord knows what.
Plan 9 gzip is 738 lines total:
Even the zipfs file server that mounts zip files as file systems is 391 lines.edit - post a link to said code: https://github.com/9front/9front/tree/front/sys/src/cmd/gzip
> ... (and whenever working with C always keep in mind that C stands for CVE).
Sigh.
You forgot to include https://github.com/9front/9front/tree/front/sys/src/libflate which gzip is built around, which brings it closer to 10k lines.
His also omits CRC, which is part of the 25k lines, no --fast/--best/etc, missing some output formats, and so on. I'm sure the 25k includes a lot of bloat, but the comparison is odd. Comparing to your list would make much more sense.
Crc32 can be written in handful lines of code. Although it'd be better to use the vector instruction set - e.g. AVX when available.
I would expect a CRC to add a negligible number of lines of code. The reason that production-grade decompressors are tens of thousands of LOC is likely attributable to extreme manual optimization. For example, I wouldn't be surprised if a measurable fraction of those lines are actually inline assembly.
True. A most basic CRC implementation is about 7 lines of code: (presented in Java to avoid some C/C++ footguns)
Or smooshed down slightly (with caveats): But one reason that many CRC implementations are large is because they include a pre-computed table of 256× 32-bit constants so that one byte can processed at a time. For example: https://github.com/madler/zlib/blob/7cdaaa09095e9266dee21314...That's java code, though... bit weird, esp. i % 8 (which is just i & 7). The compiler should be able to optimize it since 'i' is guaranteed to be non-negative, still awkward.
Java CRC32 nowadays uses intrinsics and avx128 for crc32.
Doesn't need to be inline assembly, just pre-encoded lookup tables and intrinsics-based vectorized CRC alone will add quite a lot of code. Most multi-platform CRC algorithms tend to have at least a few paths for byte/word/dword at a time, hardware CRC, and hardware GF(2) multiply. It's not really extreme optimization, just better algorithms to match better hardware capabilities.
The Huffman decoding implementation is also bigger in production implementations for both speed and error checking. Two Huffman trees need to be exactly complete except in the special case of a single code, and in most cases they are flattened to two-level tables for speed (though the latest desktop CPUs have enough L1 cache to use single-level).
Finally, the LZ copy typically has special cases added for using wider than byte copies for non-overlapping, non-wrapping runs. This is a significant decoding speed optimization.
Yes, there's subdirs with language bindings for many non-C langs, an examples folder with example C code, win32 specific C code, test code, etc.
More reasons it's an odd comparison.
gzip also contains a significant amount of compatibility code for different platforms.
But probably without any error checking.
Feels like Rust culture inherited "throw and forget" as an error handling "strategy" from Java
Sigh.
Hehe, why "probably"? It says "250 lines" right there in the subject. Surely one can skim the single file of code (https://github.com/ieviev/mini-gzip/blob/main/src/main.rs) and offer criticism that isn't based on hypotheticals?
Anyway, I skimmed the file for you this time, and basically you're either correct or wrong, depending on your definition of "error checking." The code handles error conditions by aborting the process. Seeing as it's a standalone CLI program and not a library meant for reuse, safely shutting down with a meaningful message sounds like fair game to me.
This is an educational project. Not something for production. The article even says so!
You can leave the snide comments about “Rust culture” (whatever that is) out next time.
Why people ascribe error handling practices to languages is baffling. What language doesn't allow punting error handling until later? Even Haskell has "panic" functionality that fudges the type constraints to allow this.
Another dev who doesn’t show respect to what has been done and expect a particular language will do wonders for him. Also I don’t see this is much better in term of readability.
Where do you see the lack of respect? The author wanted to learn how gzip works and chose to implement it in a language they like to do so. As a learning tool, not because the world needs another gzip decompressor.
he does mention https://github.com/trifectatechfoundation/zlib-rs not just https://github.com/madler/zlib, but it would be interesting to hear from those developers also