The author mentions HMAC at the end. I think HMAC is really an underrated technique. I remember reading Colin Percival's classic Cryptographic Right Answers[0] and saw a section about "symmetric signatures." I pondered to myself what scheme I could use for that before I looked at the answer: of course it's just HMAC. I feel like this is another perspective that ought to be more widely known: if you want something to be like a signature, but the two parties (or just a single party at different times) can share a key, HMAC really is the right answer. Things like, a server needs to cryptographically sign a cookie to prevent tempering: that's HMAC. Or a server needs to know an API request is coming from an expected client: that's also HMAC.
I'd also throw in that HMAC is overrated. It's a workaround for bad hash algorithms that are vulnerable to length-extension attacks.
If you're using a "good" hash algorithm, then MAC-ing is simple: hash over your key and message.
It's pretty weird that SHA-256 has been king for so long, when SHA-512/256 (which, as I've noticed people don't understand, means SHA-512 truncated to 256 bits) was there from the beginning and is immune from this attack.
Anyway, in general it's a pet peeve of mine that many people so often say "HMAC" when really they just mean MAC.
Yes and no. HMAC is very inefficient for short messages, but that inefficiency quickly vanishes into noise for anything over a kB or two. (HKDF and HMAC-DRBG are probably the worst offenders as they are always running HMAC on small inputs).
But, on the other hand, HMAC has repeatedly proven itself to be resilient to all kinds of attacks. I definitely didn’t mean any MAC when I recommended HMAC: eg I don’t think Poly1305 is a good general purpose MAC. PRF maybe, but sometimes you need the MAC to be committing too. Yes, some hash functions can be used with a simple prefix MAC, but then you need to list which specific hash functions to use (and most of those are not yet widely available).
Ah yes of course in 2018 it's still HMAC.
They published a followup to that article two months ago, and the correct answer in 2024 is still HMAC.
One question I always wondered about with cookie signing is: Why not store the user and the cookie in a database and check against that when they try to present it to you? Performance reasons?
It's mostly about performance. If you can store all the required info about the user inside the cookie then you can avoid a DB query roundtrip before sending a response.
Now that your cookie looks like this (probably also base64 encoded):
You don't have to hit the DB to display "Hi John" to the user and hide the jucy "Admin" panel. Without HMAC, an attacker could flip the "is_admin" boolean in the cookie.
You could also create a cookie that is just random bytes
F2x8V0hExbWNMhYMCUqtMrdpSNQb9dwiSiUBId6T3jg
and then store it in a DB table with similar info but now you would have to query that table for each request. For small sites it doesn't matter much and if it becomes a problem you can quite easily move that info into a faster key-value store like Redis. And when Redis also becomes too slow you are forced to move to JSON Web Tokens (JWT) witch is just a more standardized base64 encoded json wrapped with HMAC to avoid querying a database for each request.
But even if you are using random bytes as your session identifier, you should still wrap it in a HMAC so that you can drop invalid sessions early. Just for making it harder for someone to DDOS your DB.
A bit of a tangent. This isn't a dig on HMAC itself, but using HTTP request body or query string as the HMAC "message" is the worst. My employer provides some APIs with that sort of scheme and it's a very common source of technical customer support tickets.
The problem is that many people are using web frameworks that automatically turn body and query into some kind of hash map data structure. So when you tell them "use the request body as the HMAC message", they go "OK, message = JSON.stringify(request.body)", and then it's up to fate whether or not their runtime produces the same exact same JSON as yours. Adding a "YOU MUST USE THE RAW REQUEST BODY" to the docs doesn't seem to work. We've even had customers outright refuse to do so after we ask them to do so in the "why are my verifications failing" ticket. And good luck if it's a large/enterprise customer. Get ready to have 2 different serialization routines: one for the general populous, and one for the very large customer that wrote their integration years ago and you only now found out that their runtime preserves "&" inside JSON strings but yours escapes it.
Rant over...
> "&" inside JSON strings but yours escapes it
What escaping of "&" inside JSON are you talking about? Some unholy mix of JSON and urlencode?
Ruby on Rails turns "&" into "\u0026".
See rails/rails, activesupport/lib/active_support/json/encoding.rb.
Are things like Diffie Hellman generally available such that you can always get a symmetric key? Or is that a special case?
I'm no cryptographer, but I would say that it is indeed the case that you can assume that two parties can derive a shared key over an untrusted channel. The post Cryptography Right Answers PQ [1], linked in another comment, addresses this in the section "Key Exchange". Rather than thinking about Diffie-Hellman directly, you would turn to a Key Exchange Mechanism (KEM).
Before post-quantum cryptography concerns, KEM were indeed mostly built on top of Diffie-Hellman key agreement, but you could also build one on top of RSA, or on top of some lattice constructs. But you wouldn't build one yourself, there are good constructions to choose from! The OP actually has a 3-part series on KEMs, although I don't think it addresses post-quantum issues [2].
Yeah, if you have a shared secret, HMAC is the way to go.
It's also super simple: It's almost literally just concatenating the secret and the message you want to authenticate together, and take an ordinary hash (like SHA256) of that, the rest of it is just to deal with padding.
It's super intuitive how HMAC works: If you just mash secret and message together on your side, and get the same answer as what the other side told you, then you know that the other side had the secret key (and exactly this message), because there's obviously no way to go from SHA256 to the input.
HMAC is also useful if you want to derive new secret keys from other secret keys. Take an HMAC with the secret key and an arbitrary string, you get a new secret key. The other side can do the same thing. Here's the kicker, the arbitrary string does not have to be secret to anyone, it can be completely public!
Why would you do that? Well, maybe you want the derived key to have a different lifetime and scope. A "less trusted" component could be given this derived key to do its job without having to know the super-secret key it was derived from (which could be used to derive other keys for other components, or directly HMAC or decrypt other stuff).
> It's also super simple: It's almost literally just concatenating the secret and the message you want to authenticate together, and take an ordinary hash (like SHA256) of that, the rest of it is just to deal with padding.
It's not quite as simple as that. The output of the first hash is hashed a second time (to prevent length extension attacks).
This article is very relevant in the context of the EU Digital Identity Wallet, and digital credentials in general, such as ISO/IEC 18013-5 mobile driver licenses and other mdocs.
We may accidentially end up with non-repudiation of attribute presentation, thinking that this increases assurance for the parties involved in a transaction. The legal framework is not designed for this and insufficiently protects the credential subject for example.
Instead, the high assurance use cases should complement digital credentials (with plausible deniability of past presentations) with qualified e-signatures and e-seals. For these, the EU for example does provide a legal framework that protects both the relying party and the signer.
Isn't non-repudiation something we want for cases like this? If e.g. a car rental place checks your driving license before renting you a car, and then you get into a crash, no-one wants you to be able to claim that you never showed them your driving license and they never checked.
To prove that the car rental company has seen the driver licence, they just need to show the judge a copy of the licence which is e-sealed by its issuing authority. No need to include a non-repudiable proof-of-possession signature of the holder. Having that in addition would just introduce legal ambiguity and information asymmetry to the disadvantage of the holder.
The opponent may still claim that the car rental place is showing a copy that was obtained illegally, and not in holder presentation. To avoid such a claim, the car rental company should ask for a qualified e-signature before providing the car key. The signed data can include any relevant claims that both parties confirm as part of the transaction. To provide similar assurance to the customer, the company should counter-sign that document, or provide it pre-sealed if it is an automated process.
Note that with the EU Digital Identity, creating qualified e-signatures is just as easy as presenting digital credentials.
I mean it's not a super big deal if the EU identity private key leaks in some arcane attack or if someone steals it the normal way, you can just cancel it and order a new one like a credit card. It expires every two years I think anyway.
This reminds me of a specific number that Americans have to give in plain text as proof of digital identity that they only get one of and can't change it ever. Lol.
That doesn’t matter. The claim being made by the grandparent post is that the legal system isn’t well-equipped to deal with scenarios like, “yes the digital signature is valid but it was improperly authorized.”
> This reminds me of a specific number that Americans have to give in plain text as proof of digital identity that they only get one of and can't change it ever. Lol.
You can get up to ten replacements of your card in your lifetime. They do all have the same number though.
> As well as authenticating a message, they also provide third-party verifiability and (part of) non-repudiation.
I think digital signatures and third party verification are an incredibly useful feature. The ability to prove you received some data from some third party lets you prove things about yourself, and enables better data privacy long-term, especially when you have selective disclosure when combined with zero knowledge proofs. See: https://www.andrewclu.com/sign-everything -- the ability to make all your data self-sovereign and selectively prove data to the outside world (i.e. prove I'm over 18 without showing my whole passport) can be extremely beneficial, especially as we move towards a world of AI generated content where provenant proofs can prove content origin to third parties. You're right that post quantum signature research is still in progress, but I suspect that until post-quantum supremacy, it's still useful (and by then I hope we'll have fast and small post quantum signature schemes).
EU's digital signatures let you do this for your IDs and https://www.openpassport.app/ lets you do this for any country passport, but imagine you could do this for all your social media data, personal info, and login details. we could have full selective privacy online, but only if everyone uses digital signatures instead of HMACs.
The author mentions HMAC at the end. I think HMAC is really an underrated technique. I remember reading Colin Percival's classic Cryptographic Right Answers[0] and saw a section about "symmetric signatures." I pondered to myself what scheme I could use for that before I looked at the answer: of course it's just HMAC. I feel like this is another perspective that ought to be more widely known: if you want something to be like a signature, but the two parties (or just a single party at different times) can share a key, HMAC really is the right answer. Things like, a server needs to cryptographically sign a cookie to prevent tempering: that's HMAC. Or a server needs to know an API request is coming from an expected client: that's also HMAC.
[0]: https://www.daemonology.net/blog/2009-06-11-cryptographic-ri...
More generally, a MAC. You don't necessarily need one based on a hash.
(Unrelated) see also the more recent https://www.latacora.com/blog/2018/04/03/cryptographic-right...
I'd also throw in that HMAC is overrated. It's a workaround for bad hash algorithms that are vulnerable to length-extension attacks.
If you're using a "good" hash algorithm, then MAC-ing is simple: hash over your key and message.
It's pretty weird that SHA-256 has been king for so long, when SHA-512/256 (which, as I've noticed people don't understand, means SHA-512 truncated to 256 bits) was there from the beginning and is immune from this attack.
Anyway, in general it's a pet peeve of mine that many people so often say "HMAC" when really they just mean MAC.
Yes and no. HMAC is very inefficient for short messages, but that inefficiency quickly vanishes into noise for anything over a kB or two. (HKDF and HMAC-DRBG are probably the worst offenders as they are always running HMAC on small inputs).
But, on the other hand, HMAC has repeatedly proven itself to be resilient to all kinds of attacks. I definitely didn’t mean any MAC when I recommended HMAC: eg I don’t think Poly1305 is a good general purpose MAC. PRF maybe, but sometimes you need the MAC to be committing too. Yes, some hash functions can be used with a simple prefix MAC, but then you need to list which specific hash functions to use (and most of those are not yet widely available).
Ah yes of course in 2018 it's still HMAC.
They published a followup to that article two months ago, and the correct answer in 2024 is still HMAC.
https://www.latacora.com/blog/2024/07/29/crypto-right-answer...
One question I always wondered about with cookie signing is: Why not store the user and the cookie in a database and check against that when they try to present it to you? Performance reasons?
It's mostly about performance. If you can store all the required info about the user inside the cookie then you can avoid a DB query roundtrip before sending a response.
Now that your cookie looks like this (probably also base64 encoded):
You don't have to hit the DB to display "Hi John" to the user and hide the jucy "Admin" panel. Without HMAC, an attacker could flip the "is_admin" boolean in the cookie.You could also create a cookie that is just random bytes
and then store it in a DB table with similar info but now you would have to query that table for each request. For small sites it doesn't matter much and if it becomes a problem you can quite easily move that info into a faster key-value store like Redis. And when Redis also becomes too slow you are forced to move to JSON Web Tokens (JWT) witch is just a more standardized base64 encoded json wrapped with HMAC to avoid querying a database for each request.But even if you are using random bytes as your session identifier, you should still wrap it in a HMAC so that you can drop invalid sessions early. Just for making it harder for someone to DDOS your DB.
A bit of a tangent. This isn't a dig on HMAC itself, but using HTTP request body or query string as the HMAC "message" is the worst. My employer provides some APIs with that sort of scheme and it's a very common source of technical customer support tickets.
The problem is that many people are using web frameworks that automatically turn body and query into some kind of hash map data structure. So when you tell them "use the request body as the HMAC message", they go "OK, message = JSON.stringify(request.body)", and then it's up to fate whether or not their runtime produces the same exact same JSON as yours. Adding a "YOU MUST USE THE RAW REQUEST BODY" to the docs doesn't seem to work. We've even had customers outright refuse to do so after we ask them to do so in the "why are my verifications failing" ticket. And good luck if it's a large/enterprise customer. Get ready to have 2 different serialization routines: one for the general populous, and one for the very large customer that wrote their integration years ago and you only now found out that their runtime preserves "&" inside JSON strings but yours escapes it.
Rant over...
> "&" inside JSON strings but yours escapes it
What escaping of "&" inside JSON are you talking about? Some unholy mix of JSON and urlencode?
Ruby on Rails turns "&" into "\u0026".
See rails/rails, activesupport/lib/active_support/json/encoding.rb.
Are things like Diffie Hellman generally available such that you can always get a symmetric key? Or is that a special case?
I'm no cryptographer, but I would say that it is indeed the case that you can assume that two parties can derive a shared key over an untrusted channel. The post Cryptography Right Answers PQ [1], linked in another comment, addresses this in the section "Key Exchange". Rather than thinking about Diffie-Hellman directly, you would turn to a Key Exchange Mechanism (KEM).
Before post-quantum cryptography concerns, KEM were indeed mostly built on top of Diffie-Hellman key agreement, but you could also build one on top of RSA, or on top of some lattice constructs. But you wouldn't build one yourself, there are good constructions to choose from! The OP actually has a 3-part series on KEMs, although I don't think it addresses post-quantum issues [2].
[1]: https://www.latacora.com/blog/2024/07/29/crypto-right-answer... [2]: https://neilmadden.blog/2021/01/22/hybrid-encryption-and-the...
Yeah, if you have a shared secret, HMAC is the way to go.
It's also super simple: It's almost literally just concatenating the secret and the message you want to authenticate together, and take an ordinary hash (like SHA256) of that, the rest of it is just to deal with padding.
It's super intuitive how HMAC works: If you just mash secret and message together on your side, and get the same answer as what the other side told you, then you know that the other side had the secret key (and exactly this message), because there's obviously no way to go from SHA256 to the input.
HMAC is also useful if you want to derive new secret keys from other secret keys. Take an HMAC with the secret key and an arbitrary string, you get a new secret key. The other side can do the same thing. Here's the kicker, the arbitrary string does not have to be secret to anyone, it can be completely public!
Why would you do that? Well, maybe you want the derived key to have a different lifetime and scope. A "less trusted" component could be given this derived key to do its job without having to know the super-secret key it was derived from (which could be used to derive other keys for other components, or directly HMAC or decrypt other stuff).
> It's also super simple: It's almost literally just concatenating the secret and the message you want to authenticate together, and take an ordinary hash (like SHA256) of that, the rest of it is just to deal with padding.
It's not quite as simple as that. The output of the first hash is hashed a second time (to prevent length extension attacks).
This article is very relevant in the context of the EU Digital Identity Wallet, and digital credentials in general, such as ISO/IEC 18013-5 mobile driver licenses and other mdocs.
We may accidentially end up with non-repudiation of attribute presentation, thinking that this increases assurance for the parties involved in a transaction. The legal framework is not designed for this and insufficiently protects the credential subject for example.
Instead, the high assurance use cases should complement digital credentials (with plausible deniability of past presentations) with qualified e-signatures and e-seals. For these, the EU for example does provide a legal framework that protects both the relying party and the signer.
Isn't non-repudiation something we want for cases like this? If e.g. a car rental place checks your driving license before renting you a car, and then you get into a crash, no-one wants you to be able to claim that you never showed them your driving license and they never checked.
To prove that the car rental company has seen the driver licence, they just need to show the judge a copy of the licence which is e-sealed by its issuing authority. No need to include a non-repudiable proof-of-possession signature of the holder. Having that in addition would just introduce legal ambiguity and information asymmetry to the disadvantage of the holder.
The opponent may still claim that the car rental place is showing a copy that was obtained illegally, and not in holder presentation. To avoid such a claim, the car rental company should ask for a qualified e-signature before providing the car key. The signed data can include any relevant claims that both parties confirm as part of the transaction. To provide similar assurance to the customer, the company should counter-sign that document, or provide it pre-sealed if it is an automated process.
Note that with the EU Digital Identity, creating qualified e-signatures is just as easy as presenting digital credentials.
I mean it's not a super big deal if the EU identity private key leaks in some arcane attack or if someone steals it the normal way, you can just cancel it and order a new one like a credit card. It expires every two years I think anyway.
This reminds me of a specific number that Americans have to give in plain text as proof of digital identity that they only get one of and can't change it ever. Lol.
That doesn’t matter. The claim being made by the grandparent post is that the legal system isn’t well-equipped to deal with scenarios like, “yes the digital signature is valid but it was improperly authorized.”
> This reminds me of a specific number that Americans have to give in plain text as proof of digital identity that they only get one of and can't change it ever. Lol.
You can get up to ten replacements of your card in your lifetime. They do all have the same number though.
[1] https://secure.ssa.gov/poms.nsf/lnx/0110205400
Well, at least you can laminate it
[dead]
> As well as authenticating a message, they also provide third-party verifiability and (part of) non-repudiation.
I think digital signatures and third party verification are an incredibly useful feature. The ability to prove you received some data from some third party lets you prove things about yourself, and enables better data privacy long-term, especially when you have selective disclosure when combined with zero knowledge proofs. See: https://www.andrewclu.com/sign-everything -- the ability to make all your data self-sovereign and selectively prove data to the outside world (i.e. prove I'm over 18 without showing my whole passport) can be extremely beneficial, especially as we move towards a world of AI generated content where provenant proofs can prove content origin to third parties. You're right that post quantum signature research is still in progress, but I suspect that until post-quantum supremacy, it's still useful (and by then I hope we'll have fast and small post quantum signature schemes).
EU's digital signatures let you do this for your IDs and https://www.openpassport.app/ lets you do this for any country passport, but imagine you could do this for all your social media data, personal info, and login details. we could have full selective privacy online, but only if everyone uses digital signatures instead of HMACs.