IPIP-462: Ipfs-Path-Affinity on Gateways #462
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| --- | ||
| title: "IPIP-0462: Ipfs-Path-Affinity on Gateways" | ||
| date: 2024-02-16 | ||
| ipip: proposal | ||
| editors: | ||
| - name: Marcin Rataj | ||
| github: lidel | ||
| url: https://lidel.org/ | ||
| affiliation: | ||
| name: IP Shipyard | ||
| url: https://ipshipyard.com | ||
| relatedIssues: | ||
| - https://github.com/ipfs/kubo/issues/10251 | ||
| - https://github.com/ipfs/kubo/issues/8676 | ||
| order: 462 | ||
| tags: ['ipips'] | ||
| --- | ||
|
|
||
| ## Summary | ||
|
|
||
| This IPIP adds gateway support for optional `Ipfs-Path-Affinity` HTTP request header. | ||
|
|
||
| ## Motivation | ||
|
|
||
| Endpoints that implement :cite[trustless-gateway] may receive requests for a | ||
| single block, or a CAR request sub-DAG of a biger tree. | ||
|
|
||
| While every piece of data that is supposed to be able to be accessed | ||
| independently should be advertised on routing system, not every CID is today. | ||
| Some providers limit announcements to top-level root CIDs due to time, cost, or | ||
| misconfiguration. | ||
|
|
||
| What does this mean for the ecosystem? It should adapt and ensure | ||
| implementations leverage all infromation provided by the end user. | ||
|
|
||
| Over time, both clients and servers should leverage the concept of "affinity". | ||
|
|
||
| The introduction of an optional `Ipfs-Path-Affinity` header aims to increase | ||
| the success rate of the gateway retrieving internal standalone blocks or byte | ||
| ranges, especially if the requested blocks are not announced on routing | ||
| systems, but belong to a bigger DAG, and only the root CID of that parent DAG | ||
| is announced. | ||
|
|
||
| ## Detailed design | ||
|
There was a problem hiding this comment. I really feel like this is missing some guidance on what the values should or can be. With existing wording, it's too vague and could result in significant client pain of having to implement different values for different providers. Also: we should really speak to the how of implementing this server on the client and server sides: at least some best practices. E.g. how are we going to implement this in There was a problem hiding this comment. Clarified format in https://en-wikipedia--on--ipfs-org.ipns.inbrowser.dev/wiki/Books fetching necessary blocks from https://trustless-gateway.link it would have in every |
||
|
|
||
| Introduce `Ipfs-Path-Affinity` HTTP request header to allow HTTP client to | ||
| inform gateway about the context of block/CAR request. | ||
|
Comment on lines
+46
to
+47
There was a problem hiding this comment. What is the format of the data that goes here? Is it There was a problem hiding this comment. IMO, it should be There was a problem hiding this comment. I would much prefer an ipfs:// protocol instead of pathing, but i guess whatever the users is likely to have is better ux There was a problem hiding this comment. Went with content path, as we already talk about them all over the specs. NOTE: because UnixFS can have whitespace, Clarified format in |
||
|
|
||
| Client asking gateway for a block SHOULD provide a hint about the DAG the block | ||
| belongs to, if such information is available. | ||
|
Comment on lines
+49
to
+50
There was a problem hiding this comment. Is it only a strict IPLD DAG where we'd recommend this? It seems like you could plausibly do this for a set of related data that aren't explicitly linked via IPLD (e.g. a website that has HTML that loads jpegs from within the same or a different root DAG). There was a problem hiding this comment. This reminds me of some chats we've had about it being likely that a provider of "bafyFoo1" would likely also have "bafyFoo2" There was a problem hiding this comment. The way field format is specified, these can be arbitrary content paths. It is up to the client to provide a meaningful hint. Most of the time it will be the content path the client tries to load, but it could also be |
||
|
|
||
| A gateway unable to find providers for internal block should be | ||
| able to leverage affinity information sent by client and use CIDs of parent | ||
| path segments as additional content routing lookup hints. | ||
|
|
||
| ## Design rationale | ||
|
|
||
| ### User benefit | ||
|
|
||
| When supported by both client and server: | ||
|
|
||
| - Light clients are able to use trustless HTTP gateway endpoints more | ||
| efficiently, resume downloads faster. | ||
| - Gateway operators are able to leverage the hint and save resources related to | ||
| provider lookup. | ||
| - Content providers are able to implement smarter announcement mechanisms, | ||
|
There was a problem hiding this comment. What exactly do you mean by "smarter announcement mechanisms"? There was a problem hiding this comment. "roots" isn't really a word here. Root of what? At every layer in the DAG going up you could slice off the top of the tree and declare a new root. Every piece of content that needs to be independently addressable should be advertised. See https://github.com/ipfs/specs/pull/462/files#r1492996318 So at the very least if you make a block-request for the middle of a tar.gz file (where no part of the file really needs to be addressed on its own) you should be able to find it even if the provider has only advertised the root of the file. As mentioned in the linked comment I do think we need to be careful not to mislead people though. There was a problem hiding this comment.
Yes, that makes sense in theory, but if we go to the
Here, the only independently addressable content is the tar.gz file and for range block-requests in the middle, you'd need to pass the affinity header to be able to fetch that. If we're on the same page thus far, what would be smarter announcement mechanisms/strategies? Is the idea for those to be codec-aware in the sense that you could tell the node to advertise all UnixFS Files and Directories? There was a problem hiding this comment. @2color we have some ideas for smarter announcement mechanisms/strategies in ipfs/kubo#8676 and more actionable ipfs/kubo#10365 (which has wip implementation). ps. we have a concept of There was a problem hiding this comment. It sounds like "smarter" in this case means "frugal but smart" in the sense that it involves potentially less announcements that are effective enough for routing to work. There was a problem hiding this comment. Exactly, there are "frugal" things we could do on both client and server to announce less, but be as efficient at retrieval in real world usage patterns (website browsing, video streaming, download resume or parallel downloads etc). |
||
| without worrying that some internal blocks are not announced (intentionally or unintentionally). | ||
|
|
||
| ### Compatibility | ||
|
|
||
| This is an optional HTTP header which makes it backward-compatible with | ||
| existing ecosystem of HTTP clients and IPFS Gateways. | ||
|
|
||
| ### Security | ||
|
|
||
| The client is in control when the affinity information is sent in the header, | ||
| and an implementation SHOULD allow an end user to disable it in context where parent | ||
| content path information is considered sensitive information. | ||
|
|
||
| Gateway implementation that supports `Ipfs-Path-Affinity` header being present | ||
| more than once MUST also set limit (e.g. max 3) to avoid abuse. | ||
|
|
||
| ### Alternatives | ||
|
|
||
| - Why not just an arbitrary identifier the user could use to establish a | ||
| relationship between requests? | ||
| - Requires server to keep state, which breaks or complicates gateway | ||
| deployments with horizontal scaling. | ||
| - Does not help when client is sending requests for different blocks/sub-DAGs | ||
| to different trustless gateways, and none of them has the whole picture, | ||
| and majority of them do not know what is the parent content path. | ||
|
|
||
| ## Test fixtures | ||
|
|
||
| N/A | ||
|
|
||
| ### Copyright | ||
|
|
||
| Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). | ||
Uh oh!
There was an error while loading. Please reload this page.