174

Go Cryptography State of the Union

I don't know why the standard library crypto packages insist on passing around `[]byte` for things like a seed value, or why we can't just pass in a seed value to a single unambiguous constructor when generating asymmetric keys. Or how the constructor for a key pair could possibly return an error, when the algorithm is supposed to be deterministic.

It all just seems a bit sloppy. Asking for a seed value like `[32]byte` could at least communicate to me that the level of security is at most 256 bits. And removing all dependencies on rand would make it obvious where the entropy must be coming from (the seed parameter). Cloudflare's CIRCL[0] library does a bit better, but shares some of the same problems.

[0] https://github.com/cloudflare/circl

4 days agoalphazard

> I don't know why the standard library crypto packages insist on passing around `[]byte` for things like a seed value

These are actually very deliberate choices, based on maybe unintuitive experience.

We use []byte instead of e.g. [32]byte because generally you start with a []byte that's coming from somewhere: the network, a file format, a KDF.

Then you have two options to get a [32]byte: cast or copy. They both have bad failure modes. If you do a ([32]byte)(foo) cast, you risk a panic if the file/packet/whatever is not the size you expected (e.g. because it's actually attacker controlled). If you do a copy(seed, foo) it's WAY WORSE, because you risk copying only 5 bytes and leaving the rest to zero and not noticing.

Instead, we decided to move the length check into the library everywhere we take bytes, so at worst you get an error, which presumably you know how to handle.

> why we can't just pass in a seed value to a single unambiguous constructor when generating asymmetric keys

I am not sure what you are referring to here. For e.g. ML-KEM, you pass the seed to NewDecapsulationKey768 and you get an opaque *DecapsulationKey768 to pass around. We've been moving everything we can to that.

> Or how the constructor for a key pair could possibly return an error, when the algorithm is supposed to be deterministic.

Depends. If it takes a []byte, we want to return an error to force handling of incorrect lengths. If the key is not a seed (which is only an option for private keys), it can also be invalid, deterministic or not. (This is why I like seeds. https://words.filippo.io/ml-kem-seeds/)

> removing all dependencies on rand would make it obvious where the entropy must be coming from (the seed parameter)

Another place where experience taught us otherwise. Algorithms that take a well-specified seed should indeed just take that (like NewDecapsulationKey768 does!), but where the spec annoyingly takes "randomness from the sky" (https://words.filippo.io/avoid-the-randomness-from-the-sky/) in an unspecified way, taking a io.Reader gave folks the wrong impression that they could use that for deterministic key generation, which then breaks as soon as we change the internals.

There is only one place to get entropy from in a Go program, anyway: crypto/rand. Anything else is a testing need, and it can be handled with test affordances like the upcoming crypto/mlkem/mlkemtest or testing/cryptotest.SetGlobalRandom.

4 days agoFiloSottile

It's been many years since I wrote any Go for a living, but does Go seriously lack a way to say "foo is probably 32 bytes, give me the 32 byte array, and if I'm wrong about how big foo is, let me handle that" ?

If the caller was expected to provide a duration and your language has a duration type, you presumably wouldn't take a string, parse that and if it isn't a duration return some not-a-duration error, you'd just make the parameter a duration. It seems like this ought to be a similar situation.

4 days agotialaramex

> It's been many years since I wrote any Go for a living, but does Go seriously lack a way to say "foo is probably 32 bytes, give me the 32 byte array, and if I'm wrong about how big foo is, let me handle that" ?

Not in the static type signature, but you can do that as a runtime check either by casting and handling the potential panic (as described above) or by checking the size and returning an error if it's not as expected, which is what the library does.

4 days agothrowaway894345

How did you write Go for a living and simultaneously not know anything about the language?

4 days ago0x696C6961

Because ten years ago Go was a rather different language? Because I don't work only in languages where I would consider myself an expert and so be confident I know every feature ?

This reminds me of a C programmer confidently insisting to me that bool had "always" been a keyword. Nope, only since C23 - but it had been used like a keyword anyway. So it makes sense many practitioners wouldn't know that.

3 days agotialaramex

> Because ten years ago Go was a rather different language?

It was not.

a day ago0x696C6961

You can create a new empty array of variable size backed by a 32 Byte array as it's starting size.

The difference is really that [32]Byte is a single pointer (in compiler hands that you never touch) to a slab of 32 bytes of memory; the []Byte (of internal size 32 use 0-32) has the current allocated size, current used size, and a pointer (none of which you directly touch but two of which are trivial to affect with language semantics) values that point to a 32 Byte backing array.

The only time this matter is timing or performance critical code. With respect to cryptography, timing might be critical for performance, but it's absolutely critical for never taking _variable_ time to perform an operation based on data as well as not on key. In that respect this doesn't matter.

4 days agomjevans
[deleted]
4 days ago

Who are u expecting an answer from?

4 days agodes429

> If you do a ([32]byte)(foo) cast, you risk a panic if the file/packet/whatever is not the size you expected (e.g. because it's actually attacker controlled)

Can you give an example of a situation where that is actually a concern? It doesn't really seem like a realistic threat model to me. Knowledge of the key is pretty much how these algorithms define attackers vs. defenders. If the attacker has the key that's gg.

There are lots of things in Go that can panic. Even in syntax, the conversion is very similar to an interface conversion, and those haven't been a problem for me in practice, partly because of good lint rules to force checking the "okay" boolean.

4 days agoalphazard

A cloud service that lets users upload their certificates and private keys, to be served by the service's CDN. Here the attacker is attacking the system's availability, not the key.

(But also, it's easy to see how this is a problem for public keys and ciphertexts, and it would be weird to have an inconsistent API for private keys.)

4 days agoFiloSottile

Those aren’t arguments for having []byte instead of [32]byte like you think they are. They’re arguments for having an unambiguous IV type that can be constructed from a []byte or [32]byte, or responsibly generated on your behalf. The error-handling logic can be expressed once in the conversion process, and the rest of your crypto APIs can assume the happy path.

Of course, this isn’t really reasonable given golang’s brain-dead approach to zero values (making it functionally impossible to structurally prevent using zero IVs). But it just serves as yet another reminder that golang’s long history of questionable design choices actively impede the ability to design safe, hard-to-misuse APIs.

4 days agostouset

I'm not sure what you are referring to, but we were talking about keys, not IVs.

Also, "an unambiguous key type that can be constructed from a []byte or responsibly generated on your behalf" is exactly what crypto/mlkem exposes.

4 days agoFiloSottile

What’s wrong with zero values? They free the developer from guessing hidden allocations. IMO this benefit outweighs cast riddles by orders of magnitude.

4 days agokalterdev

I'm not nearly as angsty as the parent on this subject, but they don't really free the developer from guessing about hidden allocations--Go's allocations are still very much hidden even if developers can reasonably guess about the behavior of the escape analyzer. I think it would have been better if Go _required_ explicit assignment of all variables. That said, despite not being a big fan of this particular decision, Go is still by far the most productive language I've used--it strikes an excellent balance between type safety and productivity even if I think some things could be improved.

4 days agothrowaway894345

Zero values prioritize implementation convenience (we always have a zero value so we don't need to handle any cases where we don't have a value, just says those are zero) over application convenience (maybe my type should not have a default and the situation where it has no value is an error)

Take either of Rust's library types named Ordering - core::cmp::Ordering (is five less than eight or greater?) or core::sync::atomic::Ordering (even if it's on another core this decrement definitely happens before that check) neither of these implements Default because even though internally they both use the zero bit pattern to mean something, that's a specific value, not a default.

4 days agotialaramex

They're like PHP: silent failures that ever push forwards through the system.

4 days agothe_gipsy
[deleted]
4 days ago

boggle

I’m not a golang programmer, but I find this quite bizarre. Sure, C++ arrays are awful and even std::array may not be able to legally alias a vector. But Rust (no surprise) gets this right — there is a properly fallible conversion from slice reference to array reference.

But I guess Go doesn’t. This seems silly to me.

4 days agoamluto

Right now, type casts (called conversions in the spec) always produce a single value. The idiomatic way to have checked casts would IMO be a two-value form, as this would be consistent with type assertions, channel receives, and map indexing, off the top of my head. End result would be something like:

  var x []byte
  y, ok := [32]byte(x)
  // ...
However, it feels like a relatively significant change to the language just for a niche use. Even the ability to cast from []T to [N]T or *[N]T is actually fairly new (Go 1.20 and 1.17, respectively). I don't think it's that hard to check the length before casting, though a checked cast would be convenient since you wouldn't have to repeat the length between the check and the cast.
4 days agokbolino

But it would be a subvariant of a much more common pattern in code - destructuring which is quite noisy now but could just be:

  a, b, rest := strings.Split(somestr, "/")
Which would be conventional for the whole thing, and the check would be for an empty type after rest.

I usually wind up using something like the samber/lo library to reduce the noise here. You wind up doing this all the time.

4 days agoXorNot

I must disagree. Destructuring is nifty, but it is almost completely unrelated. The second value in any of the builtin two-value assignment forms is invariant on the type or size of RHS; it's always a boolean, and its meaning relates to the "kind" of RHS, not the exact type:

  msg, ok := <-ch    // ok means "channel ch is not closed"
  val, ok := m[key]  // ok means "map m contains key"
  dyn, ok := i.(T)   // ok means "interface i's dynamic type is T or implements T"
This new operation would be similar:

  arr, ok := [N]T(s) // ok means "slice s has len(s) == N"
For all of these cases, ok being true indicates that the other value is safe to use.
4 days agokbolino
[deleted]
4 days ago

Try creating new AES cypher, you will see that you have to provide 16, 24 or 36 bytes in order to get AES-128, AES-192, or AES-256. There is no single [32]byte, for example, as the length cannot be inherently fixed due to this.

4 days agogethly

I agree with the author’s sentiment about FIPS 140. I find NIST to be incredibly slow. I understand there must be some stability, but they are too slow. For example, I think it's horrible that they are still recommending PBKDF2 in 2025.

4 days agopregnenolone

A big part of the problem I have with it is that it's a "ceiling" on security. Things like electrical code or building code are a "floor" on quality, you have to be at least as good as the code requirements, but can freely be better. FIPS-140 bounds you both ways. If you could more easily do better it'd be much less of a problem that NIST are slow.

4 days agoSAI_Peregrinus

I don't love FIPS either, but cryptosystems don't work the same way as buildings and electrical codes. It's very easy to have "secure cryptosystem A" and "secure cryptosystem B", and then have massive security holes in "cryptosystem A + B". This happens all the time, and is one of the main reasons for the classic "don't roll your own crypto" admonition. The FIPS "whole system" mandate is meant to forestall this failure mode.

4 days agoAnalemma_

Even in building and electrical, just because B is better than A does not mean it’s allowed.

IIRC the first wago parts (221) were UL-listed in 2017, the 221 were released in 2014, and the original push-lever splices (the 222) were released in 2004.

4 days agomasklinn

In fairness, it's one thing for an implementation like a building to be as over-enginereed as possible in its own right, but it's another when a standard has to ensure that multiple implementations can interoperate. I'm not saying FIPS-140 has only that kind of limitation (far from it), just that this isn't the best analogy.

4 days agofl0ki

Is any of FIPS about ensuring interoperability?

4 days agoakerl_

Yeah, there's a ton of correctness testing involved. That's mostly at the algorithm, rather than the module level, so it'll fall under CAVP/ACVP rather than CMVP.

4 days agodlgeek

That's not for interop, that's for "are you actually doing the crypto you said you'd do". It's designed to prevent broken crypto, not to ensure coordination between parties.

4 days agoakerl_

I'm more of a C person than a Go person, but I am unbelievably happy that someone in that community is using the word "cryptography" to mean cryptography and not Bitcoin.

4 days agoOhMeadhbh

Wasn't it just the shorthand "crypto" that got co-opted by the Shitcoin Industrial Complex? I think "cryptography" still means what it always meant regardless of who you ask.

4 days agojsheard

That's mostly the case, but I've seen job postings for "cryptography experts" that are, as best I can tell, looking for block chain hucksters. But I'm unlikely to work for Microsoft, so I just ignore them.

4 days agoOhMeadhbh

Well yeah, someone hired to work with the low-level nuts and bolts of blockchains would ideally need to know their way around actual bona-fide cryptography.

4 days agojsheard

Respectfully disagree. My experience is that they've read a couple chapters from Applied Cryptography and think "IACR" is a router manufacturer.

But we all see different parts of the industry. Happy to hear you're encountering more capable people in that industry.

4 days agoOhMeadhbh

There are definitely better cryptographers than me working at Zcash, for example.

4 days agoFiloSottile

I was at Microsoft for meetings once and one of the guys asked if I was a cryptographer. I replied that it depends on who else is in the room. If it's just me and my boss, I'd probably qualify as being a cryptographer. But if I'm in the room with Niels Ferguson or Adi Shamir, I should be considered a software engineer who is somewhat decent at reading math papers. At about the same time, Niels Ferguson walked in the room and I quickly commented "you should not consider me a cryptographer for the rest of this meeting."

4 days agoOhMeadhbh

Oh no I'm not speaking from experience, just saying that in principle you would want actual cryptography experts in that position, so there's no contradiction in terms. How that tends to play out in practice is another question.

4 days agojsheard

Downvoted for mentioning that people confuse cryptography with Bitcoin? Good thing I didn't mention I think we're in an AI bubble. Or that I prefer emacs to vi.

4 days agoOhMeadhbh

Let's revert the self-scrutiny trend and actually enjoy a fellow mentioning his downvote. At least I am enjoying, your perception is valid.

4 days agokunley

I'm curious about how GC languages handle crypto. Is it a risk that decrypted stuff or keys and things may be left in memory (heap?) before the next GC cycle?

4 days agoedoceo

What we did with Java (J/SAFE) was to add explicit methods to zero out sensitive info. It was a bit of a PITA because Java's never had consistent semantics about when final(ize,ly) methods were called. Later we added code to track which objects were allocated, but no longer needed, which also wasn't much fun.

Back in the Oak days Sun asked us (I was at RSADSI at the time) to review the language spec for security implications. Our big request was to add the "secure" storage specifier for data. The idea being a variable, const, whatever that was marked "secure" would be guaranteed not to be swapped out to disk (or one of a number of other system specific behaviors). But it was hard to find a concrete behavior that would work for all platforms they were targeting (mostly smaller systems at the time.)

My coworker Bob Baldwin had an existing relationship with Bill Joy and James Gosling (I'm assuming as part of the MIT mafia) so he led the meetings. Joy's response (or maybe Goslings, can't remember anymore) was "Language extension requests should be made on a kidney. Preferably a human kidney. Preferably yours. That way you'll think long and hard about it and you sure as hell won't submit 2."

4 days agoOhMeadhbh

If you have access to the local machine no language will save you.

4 days agoThaxll

Sure. But there are several graduations of threat between "zero access" and "complete access." On the intarwebs, every request is from a potential attacker. Attackers are known for violating RFC3514, so it is frequently useful to build a trust model and use existing access control mechanism to deny "sensitive" data (or control functions) to protocol participants who cannot verify their identity and/or access permission.

These models can get complex quickly, but are nevertheless important to evaluate a system's specified behaviour.

No system is perfect and your mileage may vary.

4 days agoOhMeadhbh

To oversimplifiy, it's like the same-ish risk level as JS or PHP or Ruby? (assuming the underlying algorithm is good)

4 days agoedoceo

I'm pretty sure that C's free() and C++'s delete() will happily leave private data on the heap also.

Not sure why you'd think GC'd languages carry more risk.

4 days agostackghost

Cause I was ignorant, until the helpful responses.

3 days agoedoceo

It can be, another risk it that a secret value is left on the stack, and is never overwritten because the stack doesn't get to that memory address again, so it's never overwritten or zerod.

Go really just needs a few `crypto.Secret` values of various sizes, or maybe a generic type that could wrap arrays. Then the runtime can handle all the best practices, like a single place in memory, and aggressive zeroing of any copies, etc.

4 days agoalphazard

It's not that simple! What about intermediate values during arithmetic computations? What about registers spilling during signal handling?

I honestly thought it could not be done safely, but the runtime/secret proposal discussion proved me wrong.

4 days agoFiloSottile

clear([]byte) if you want to go to the extreme and clean your own memory.

4 days agogethly

This only wipes one copy. The GC is free to move the allocation around, and is not required to clear the old copies.

4 days agokbolino

Wrong, the []byte is a pointer. There is no copy anywhere. That's why using []byte in encryption makes sense - you can wipe it out whenever you need, whereas string, the only immutable type in Go, would be a problem. Same can be done with array([32]byte).

4 days agogethly

Go has a precise garbage collector that knows where all the pointers are. So while yes, []byte is a (fat) pointer, the GC is free to move its target to a new location, as long as it updates all the pointers to it. This means that you may observe the address returned by &s[0] for some slice s changing even though you did nothing to s yourself.

The process of actually moving allocated memory around by the GC is known as compaction. It's not done aggressively today but Go reserves the right to do so in the future.

3 days agokbolino

What’s up with all these bots posting 3-4 sentence summaries in the comment section?

4 days agoleoqa

Farming points for later shilling.

4 days agoronsor

[dead]

11 hours agoqiufeng111

[flagged]

4 days agoShelby-Thomas
[deleted]
4 days ago

[flagged]

4 days agoAman_Kalwar

[flagged]

4 days agoShelby-Thomas

[flagged]

4 days agoSteve-Tony

[flagged]