Adding support for compact offers

[robbiehanson]

In the Bolt Card community, there is a new proposed protocol that uses Bolt 12 offers.

One of the problems with the current bolt card protocol is that is uses lnurl-withdraw (which creates various problems, especially for self-custodial wallets). So there's a push to switch to offers & onion messages (i.e. direct communication over the lightning network, and no need for an HTTP server).

Background

A short summary of how the new protocol works is like this:

• Bob is a Bolt Card user. When he links his Bolt Card to his (possibly self-custodial) wallet, he writes his bolt 12 offer to the card. (Along with randomly generated keys, known only by his wallet and the card.)
• Bob goes to Alice's bakery to buy some bread. He taps his bolt card to the reader, and Alice's wallet reads:
  • Bob's offer (including node ID or blinded path)
  • The dynamic values generated by the card (encrypted with keys known only to card & Bob's wallet). This includes a counter that is incremented everytime the card is read.
• Alice sends a CardPaymentRequest message to Bob that includes:
  • an invoice for the bread
  • the dynamic values read from the card (i.e. encrypted data)
  • a reply path for Alice's wallet
• When Bob's wallet receives the message, he can decrypt the data, and verify the request:
  • verifies the message authentication code generated by the card
  • verifies the card UID
  • verifies the counter has been incremented since last payment attempt
  • verifies the card isn't frozen
  • verifies the amount doesn't exceed daily/monthly spending limits, etc
• If everything checks out, Bob's wallet will send the payment
• If not, Bob's wallet sends a CardPaymentError message back to Alice

This new protocol will be a dramatic improvement over the old protocol for a number of reasons:

• it's faster
• it's more secure
• it doesn't require an HTTP server
• it better supports self-custodial wallets
• error messages are built into the protocol now (instead of just waiting for a timeout)
• it supports non-interactive refunds (Alice's wallet associates received payment with Bob's offer. So Alice can issue a refund to Bob without needing to ask Bob for an invoice.)

The Problem

There's one hiccup: the physical card only contains 256 bytes of memory. Plus there are several reserved bytes for the NDEF message header and the dynamically generated values. So the actual max size of an offer is 200 bytes. And an offer can be larger than this, especially when a blinded path is included.

Our default offer (on mainnet) is currently 206 bytes.

In our default offer, we have the following encryptedData sizes:

• encryptedData for trampoline node = 51 bytes

  • outgoing_node_id TLV = 35 bytes
  • mac = 16 bytes

• encryptedData for final recipient = 16 bytes

  • mac = 16 bytes

I was very curious as to why we bother to include any encryptedData for the final recipient, when it's actually an empty TLV. And I found the following rationale in Bolt 4:

The final recipient must verify that the blinded route is used in the right context (e.g. for a specific payment) and was created by them. Otherwise a malicious sender could create different blinded routes to all the nodes that they suspect could be the real recipient and try them until one accepts the message.

Challenging this requirement: For an incoming payment for our default offer, and for a CardPaymentRequest, we have no incentive to "verify that the blinded route is used in the right context".

In the case of our defaultOffer, there's nothing to verify about the payment because there's no amount or description or anything. And I don't think we care, since the assumption about our defaultOffer is that it's "shared widely".

In the case of a CardPaymentRequest, the verification is actually separate from the blinded path. That is, the CardPaymentRequest contains the encryptedData directly from the card, and it's really this data we need to verify.

If the encryptedData for the final recipient was dropped, the size drops to 190 bytes, and the offer would fit on the card.

In discussing the issue internally, it was decided that:

The requirements in the BOLTs only make sense for offers created by public nodes.

and:

we can drop this on the lightning-kmp side, because since we're only using private channels, we aren't exposed to channel or node probing issues.

However, we really don't want to change the current default offer. Doing so could affect contacts. Luckily we don't need to.

Proposed Solution

This PR proposes a new function in NodeParams:

fun compactOffer(trampolineNodeId: PublicKey): OfferTypes.OfferAndKey

It will return an offer with null encryptedData for the finalRecipient. I.e. an offer of size 190 bytes.

The existing functions we're currently using (NodeParams.defaultOffer & NodeParams.randomOffer) are unchanged.

Technical Discussion

A user would be allowed to link multiple cards to their wallet. (e.g. create cards for your children and set spending limits on them) Ideally each card has a different offer. Otherwise merchants can associate them. For this reason, the compactOffer uses a random nonce. And each generated compact offer is unique. (i.e given 2 offers, merchant wouldn't be able to tell if they were for 2 different wallets or the same wallet)

There's a function OfferManager.isOurOffer() which is used when receiving an InvoiceRequest. It continues to function as expected, even when presented with a compact offer.

It now looks like this:

private fun isOurOffer(
  offer: OfferTypes.Offer,
  pathId: ByteVector?,
  blindedPrivateKey: PrivateKey
): Boolean = when {
    pathId == null -> { // Compact offer
        nodeParams.compactOfferKeys.value.contains(blindedPrivateKey.publicKey())
    }
    pathId.size() != 32 -> false
    else -> { // Deterministic offer
        val expected = deterministicOffer(nodeParams.chainHash, nodeParams.nodePrivateKey, walletParams.trampolineNode.id, offer.amount, offer.description, pathId?.let { ByteVector32(it) })
        expected == OfferTypes.OfferAndKey(offer, blindedPrivateKey)
    }
}

Which uses a new value in NodeParams:

val compactOfferKeys: MutableStateFlow<Set<PublicKey>>

Since compactOffer is only designed for a narrow set of use cases, it's the caller's responsibility to store & restore the set of keys. In my use case, this will be simple because the user will only create a few cards.

[t-bast]

I've fixed the OfferManager check in eafc91f (you cannot assume that an empty path_id means that a compact offer is used, the default offer also uses an empty path_id).

This looks ok to me, we could bikeshed what the wallet exactly stores (storing the nonce instead of the blinded public key would be slightly more consistent with deterministic offers), but we probably don't care.

@thomash-acinq what do you think about this change?

[thomash-acinq]

This doesn't seem compatible with the requirement to store the key in compactOfferKeys which will not be restored from the seed.

[robbiehanson]

Good catch. That was not accurate. Fixed in 1226ecc

[thomash-acinq]

Why does the caller need to do it, it seems that the function already does it.

[robbiehanson]

Good point. Lightning-kmp does automatically update the value. But those values aren't persisted to disk. So the next time the app restarts, it's the callers responsibility to restore the set. I've updated the text to be more specific in 1226ecc

[t-bast]

A question to you @robbiehanson: how many such compact offers do you expect to create? If it's a small amount, we could derive them all deterministically based on a counter between 0 and N. You could store in Phoenix the last one you used to ensure that the next offer is different (you'll use the next counter value for the next offer), and when restoring from seed, you would simply end up reusing previous offers (by starting the counter again from 0).

[robbiehanson]

how many such compact offers do you expect to create?

Good question. Basically, a new offer would get created for each linked card. In theory this means only a handful of cards. In practice... users lose cards. And if the cards only cost a few dollars, and you can choose from hundreds of thousands of designs (holiday designs, your favorite sports teams, ...) people might buy several because it's fun (and their non-fun bank never let them do it).

There was also a discussion about merchant tracking. In the lnurl version, every card using the same HTTP sever had the same URL. This had the nice property that it provided an anonymity set, and prevented any kind of useful merchant tracking. With offers, since each offer is unique, it opens the door to merchant tracking. And it was pointed out that this problem could be solved with blinded paths. That is, you can simply update the card with a newly generated offer (and new random nonce). You don't even have to change the keys on the card. So from the app's perspective it's the same card (same keys, same card UID). But from the merchant's perspective, it now looks like a completely different card. This is an advanced feature... but would be nice to have the support in lightning-kmp should we decide to include it. Which means, possibly, multiple offers per card.

we could derive them all deterministically based on a counter between 0 and N

This is an interesting idea.

Say a user loses a card, and they mark it as lost within the app. I suppose we would want to stop accepting InvoiceRequest's for this offer? Maybe not right away, but maybe after 90 days? (Since InvoiceRequest from card offer == merchant refund attempt) Or maybe we don't worry about this?

and when restoring from seed, you would simply end up reusing previous offers

Right now, each card gets backed up. So if you lose your phone, for example, and restore your phoenix wallet, then your cards are restored as well. Using the current code, we'd have to also backup the compactOfferPubKey in this dataset. With this new idea, I would just need to backup & restore the N value. Which would be easy.

[t-bast]

Right now, each card gets backed up. So if you lose your phone, for example, and restore your phoenix wallet, then your cards are restored as well.

But here, the backup is a cloud backup, which users can disable and which could be lost (if access to the cloud account is lost), this is a weaker backup than just restoring from seed (which is sufficient for other types of offers).

That is, you can simply update the card with a newly generated offer (and new random nonce). You don't even have to change the keys on the card.

It looks like it's desirable to be able to generate new compact offers whenever wanted: I think my proposal of brute-forcing a small number of compact offers based on a counter doesn't work really well then. And being able to revoke an offer (in case the card is lost) is IMO an important feature, which is hard to handle with a counter-based set of offers.

I'm thinking of a slightly different way to backup/revoke these compact offers then. What do you think of the following:

• lightning-kmp exposes the compactOffer method, which generates a new random compact offer based on a random nonce (which is what this PR already implements)
• Phoenix provides lightning-kmp with a flow containing all the compact offers (the Offer object), which lightning-kmp uses in the isOurOffer function to accept payments or CardPaymentRequests
• this way, it's Phoenix's responsibility to handle revocation by simply removing an offer from this flow
• Phoenix backs up these offers in its cloud storage, if enabled
• And importantly, these offers are actually also backed up in the physical cards themselves: a user restoring from its seed, without a cloud backup, can simply restore card offers by reading them from the cards they own (does that work even if the card keys have been lost from the wallet, since it was only restored from seed?)

I think the ability to use the cards themselves as a backup would be really nice, let me know if that's feasible (for example by making sure the "randomly generated card keys" you mention are actually deterministically derived based on the seed and something that's hard-wired into the card or the offer).

[thomash-acinq]

I've made some changes in #841 so that compact routes are only accepted when we need it.

[thomash-acinq]

"compact offers" are not real offers in the sense that we won't accept invoice requests for them, so they don't need to be handled here.

[t-bast]

Actually they are real offers, we do want to accept payments for them (for example when merchant does a refund or cashback). It's also handy to use that card simply to provide our offer that anyone can pay, so I think it's worth handling them here?

[thomash-acinq]

OK, then we should at least check that it has only a blinded route and no other field (no description that we would commit to).

[thomash-acinq]

By tapping the card, we give permission to the other party to collect a (small) payment. It seems weird and unsafe to reuse this interaction to also mean "pay me".

[thomash-acinq]

The compact format should still be disallowed for payments and for standard offers. We should enable it explicitly only when we need it.

[robbiehanson]

I think the ability to use the cards themselves as a backup would be really nice, let me know if that's feasible

That's totally feasible. And a great idea.

When creating/linking the card, the steps would be:

• First we generate a new compact offer (with random nonce)
• Then we combine Offer.offerId with a key derived from our seed to create the master key for the card
• The master key is also called "key 0". And we derive keys 1 & 2 from key 0.
• Then we write the keys & offer to the card

Restoring the card via a tap would simply require:

• Read the offer from the card and calculate Offer.offerId
• Derive master key as we did above
• Attempt to login to the card using the master key
• If we can login, then the card & offer are associated with our wallet
• Otherwise this is a card linked to a different wallet, and we ignore

[t-bast]

Nice, that sounds like a good plan to ensure that those offers can be restored from the cards. In that case we only need Phoenix to store the set of Offer.offerId of the compact offers and provide that to lightning-kmp through a flow that we'll use in isOurOffer.

@robbiehanson can you take a look at #841 which introduces the CardPaymentRequest to see if it matches what you need?

[robbiehanson]

can you take a look at #841 which introduces the CardPaymentRequest to see if it matches what you need?

Yeah, we had a conversation about it, and we're going to bump that to a future PR. The specs for the CardPaymentRequest are still a work-in-progress.