Mirror leaf node stream on the main server

[maurice-freitag]

I didn't have any luck getting answers on Slack, so here goes:

We have a main cluster on our setup with a NATS server, a service that manages NATS entities like accounts and JetStream streams and a bunch of services doing pub/sub. These service share an account and get their credentials from the mentioned management service. We have multiple tenants where every tenant has their own JS stream for persistence. Now we also need to attach leaf nodes that run on our customers servers. Every leaf node gets their own account and exports all events with external.> subject. The account on the main server imports these and rewrites the subject so they can be correlated to the correct tenant. Our services consume some of these events which I already got working. The missing piece in our setup is persistence. Locally we have a JS stream for the external subject. My initial idea was to mirror the stream in our main system and then add this stream as a source to the marching tenant stream. However I can't get this to work, local events are never available on the mirror stream. The JS domain should be correctly configured on the leaf node as well as the mirror. The main NATS server logs the correct domain when a leaf node connects. What am I missing here?

The account JWT for my leaf node account (for testing I allowed nearly everything):

{
  "jti": "2LKLHEEWT7NIGV4Q2W52IBYLYMIOV2PLZEONEZKN4NC6PLDRRIIQ",
  "iat": 1764237088,
  "iss": "OASLFGV7NFJLCIEO72T7ED2ONGEA2TTYMK2G4QVCETIH5AOOXZAVREOE",
  "name": "test1",
  "sub": "ACI3TDAIGA4S6C2IUEPNY3YSMYVQPMZBMBNXWQMNGQ74QKMVXDZS3OBC",
  "nats": {
    "exports": [
      {
        "name": "ExternalStream",
        "subject": "$JS.>",
        "type": "service"
      },
      {
        "name": "ExternalSys",
        "subject": "$SYS.>",
        "type": "service"
      },
      {
        "name": "ExternalDeliver",
        "subject": "deliver.>",
        "type": "service"
      },
      {
        "name": "External",
        "subject": "external.>",
        "type": "stream"
      }
    ],
    "limits": {
      "subs": -1,
      "data": -1,
      "payload": -1,
      "imports": -1,
      "exports": -1,
      "wildcards": true,
      "conn": -1,
      "leaf": -1,
      "mem_storage": -1,
      "disk_storage": -1,
      "streams": 10,
      "consumer": 10,
      "max_ack_pending": -1,
      "mem_max_stream_bytes": -1,
      "disk_max_stream_bytes": -1
    },
    "default_permissions": {
      "pub": {
        "allow": [
          "external.>"
        ]
      },
      "sub": {}
    },
    "authorization": {},
    "type": "account",
    "version": 2
  }
}

The user JWT the leaf node uses:

{
  "jti": "RKPYTBYNCYFUJBBNIC6JWBRSII5PFDBDFRNGMFCVQSQAUQSSPLOA",
  "iat": 1764237101,
  "iss": "ACI3TDAIGA4S6C2IUEPNY3YSMYVQPMZBMBNXWQMNGQ74QKMVXDZS3OBC",
  "name": "test1-leafnode",
  "sub": "UBN4HLDIVMW6EYWS4JIJJAW3YJSVV2MXCXVFJ5PMOC6VVK6UF2O7WXF3",
  "nats": {
    "pub": {
      "allow": [
        "external.>",
        "$SYS.>",
        "$JS.>",
        ">",
        "deliver.>"
      ]
    },
    "sub": {
      "allow": [
        "_INBOX.>",
        "$SYS.>",
        "$JS.>",
        ">",
        "external.>",
        "deliver.>"
      ]
    },
    "subs": -1,
    "data": -1,
    "payload": -1,
    "type": "user",
    "version": 2
  }
}

[maurice-freitag]

Some more digging I've done:

What I'm trying to accomplish now is to see the leaf node's JS status with nats server report jetstream --creds sys.creds using credentials for the system account on the hub. As of now this does not work. I followed the (working) example in the docs but here the system account is shared between the hub and the leaf node. Is this a requirement? I would like to avoid spreading sys account credentials to customer servers if possible. I tried that as well and didn't even get it to work. I also found this in source (leafnode.go):

What exactly has to be done to make the JS server status from the leaf node available in the hub?

[derekcollison]

If you are trying to extend a NATS system, and hence a JS domain, yes you should connect the system accounts across the leafnode. If you want separate JS domains, do not.

I have not looked in detail at what you are trying to accomplish, but mirroring streams across a leafnode is very common.