You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on May 3, 2024. It is now read-only.
Following warning was fixed :
Warn when the spacing between a list item’s bullet and its content violates.
Signed-off-by: Pratik Patil <[email protected]>
Copy file name to clipboardExpand all lines: doc/Seagate-FDMI-Design-Notes.md
+15-15Lines changed: 15 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,8 +15,8 @@ The processing is to be done on an incoming FOL record against the Filter set. T
15
15
Filter Store is to be filled in by means of Subscription API. Filter Index is updated internally on adding Filter.
16
16
17
17
Currently 2 options for Plug-in architecture are anticipated:
18
-
1. Option 1: FDMI-Plug-in. Each plug-in is linked with FDMI making use of internal FDMI API only (some callback, for instance). See Fig.1
19
-
1. Option 2: FDMI Plug-in transforms to Mero Core Plug-in. Mero core in this case most likely provides limited features for RPC only. Mero RPC is used to collect notifications from all Mero instances.
18
+
1.Option 1: FDMI-Plug-in. Each plug-in is linked with FDMI making use of internal FDMI API only (some callback, for instance). See Fig.1
19
+
1.Option 2: FDMI Plug-in transforms to Mero Core Plug-in. Mero core in this case most likely provides limited features for RPC only. Mero RPC is used to collect notifications from all Mero instances.
20
20
21
21
## Plug-in API
22
22
@@ -25,8 +25,8 @@ The API is to allow the plug-ins making use of FDMI to register with the latter
25
25
Besides, plug-in may additionally inform FDMI about its specifics, like FOL type it is to process, optimization/tweak requirements (e.g. no batching), etc.
26
26
27
27
Public Entries:
28
-
* Initialize plug-in
29
-
* <..>
28
+
*Initialize plug-in
29
+
*<..>
30
30
31
31
**N.B.** Plug-in API design details depend on selected architecture option 1 or 2.
32
32
@@ -49,18 +49,18 @@ Failure results in dropping current FOL copy
49
49
For both options FOLs are fed to Filer Processor the same way: locally.
50
50
51
51
Public Entries:
52
-
* <..>
53
-
* <..>
52
+
*<..>
53
+
*<..>
54
54
55
55
56
56
## Subscription API
57
57
58
58
The API is to provide a way for adding Filter rules to FDMI instances, identifying FOL processing traits as well as associating Filter with Consumer
59
59
60
60
Public Entries:
61
-
* Register Filter
62
-
* Unregister Filter
63
-
* <..>
61
+
*Register Filter
62
+
*Unregister Filter
63
+
*<..>
64
64
65
65
## Notification API
66
66
@@ -74,17 +74,17 @@ Option 2, Filter Processor always sends a Notification to Consumer using Mero RP
74
74
75
75
Public Entries:
76
76
77
-
* <..>
77
+
*<..>
78
78
79
-
* <..>
79
+
*<..>
80
80
81
81
## Assumptions
82
82
83
83
### Filter
84
84
85
85
Filter identifies:
86
-
* Consumer to be notified
87
-
* Conditions FOL to meet
86
+
*Consumer to be notified
87
+
*Conditions FOL to meet
88
88
The index format: TBD
89
89
90
90
Filter once registered will be eventually spread across the whole number of Cluster nodes running FDMI services.
@@ -94,8 +94,8 @@ The number of Filters registered in Mero Cluster is expected to be of no explici
94
94
### Filter semantics
95
95
96
96
Filter syntax need to remain flexible to adopt any future improvements/enhancements, but fully covering initial requirements of multi-type support and human readability. Possible options are:
97
-
* Native-language-like syntax (SQL-like)
98
-
* Symbolic object notation, easily parsed (some standard adherent notation preferred), easily extensible, including object nesting, e.g. JSON (current DSR’s choice)
97
+
*Native-language-like syntax (SQL-like)
98
+
*Symbolic object notation, easily parsed (some standard adherent notation preferred), easily extensible, including object nesting, e.g. JSON (current DSR’s choice)
99
99
100
100
NB: Filter being parsed on ingestion may be transformed to a combination of elementary rules in case the transformation does not change Filter semantics but potentially improves Processor performance (decomposition is being performed)
Copy file name to clipboardExpand all lines: doc/outdated/Containers.md
+26-17Lines changed: 26 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,20 +4,29 @@ This document summarizes container discussions in the Motr architecture team. Th
4
4
Container is a low level abstraction that insulates higher Motr layers of the knowledge of storage device addresses (block numbers). The bulk of data and meta-data is stored in containers.
5
5
6
6
# Items
7
-
+ A container is a storage for data or meta-data. There are several types of container: data container, meta-data container, possibly others.
8
-
9
-
+ A container provides a very simple interface: it has an internal namespace consisting of keys and provides methods to fetch and update records stored at a given key. Nomenclature of keys and constraints on record structure depend on container type. For data container, keys are logical page offsets and records are simply pages full of data without any internal structure. For meta-data container keys are opaque identifiers of meta-data records.
10
-
+ A container stores its contents in a backing store---a storage device. A container allocates space from its backing store and returns no longer used space back as necessary.
11
-
+ Local RAID is implemented by using a collection of containers to stripe data across. Note that local RAID is not implemented in the containers layer to avoid cyclic dependency between layers: the data structures (specifically layouts) that local RAID implementation is required to share with SNS are interpreted by Motr back-end that itself uses containers to store its data.
12
-
+ Snapshots. Containers are used for data and meta-data snap shotting. When a local snapshot is made as a part of object or system level snap shotting, a container involved into the snapshot is COW-ed, so that all updates are re-directed to a new container.
13
-
+ After a container is COW-ed no update should ever touch the blocks of the read-only primary container. This is a necessary prerequisite of a scalable fsck implementation, that will achieve reasonable confidence in system consistency by background scan and check of periodically taken global snapshots.
14
-
15
-
+ Migration. A container (either read-only or read-write) can migrate to another node. Typical scenarios of such migration are bulk re-integration of updates from a proxy-server to a set of primary servers and moving snapshots.
16
-
17
-
+ To make migration efficient, a container must be able to do a fast IO of its contents to another node. Specifically, it should be possible to send container contents over network without iterating through individual container records. This condition also implies that a container allocates space from its backing store in a relatively large contiguous chunks.
18
-
Self-identification. A container has an identity that is stored in the container and migrated together with the container. Higher layers address container records by (container-id, key) pairs and such addresses remain valid across container migrations, including "migration" where a hard drive is pulled out of one server and inserted into another. In the latter case the system must be able to determine what containers were moved between nodes and update configuration respectively, also it should able determine whether any storage or layout invariants were violated, e.g., whether multiple units of the same parity group are now located on the same server.
19
-
+ Layouts. Higher layers address and locate data and meta-data through container identifiers and container keys. On the other hand, a layout produces a location of a given piece of data or meta-data. Together this means that lowest level layouts produce locations in the form of (container-id, key) pairs.
20
-
+ A container can be merged into another container. Inclusion of one container into another can be done administrative reasons, to efficiently migrate a large number of smaller containers or for some other purpose. On inclusion, a container retains its identity (does it? Losing identity would require updating (and, hence, tracking) all references).
21
-
+ Fids. A fid (file identifiers) is an immutable globally and temporally unique file identifier. As file meta-data record ("inode") is stored in some container, it's logical to use (container-id, key) address of this record as fid. Note that this is formally quite similar to the structure of Lustre fid, composed of a sequence identifier and offset within the sequence.
22
-
+ CLDB. A method is required to resolve container addresses to node identifiers. This method should work while containers migrate between nodes and merge with each other. Container Location Data-Base (CLDB) is a distributed data-base tracking containers in the cluster. This database is updated transactionally on container migrations and merges (and splits? There must be splits for symmetry.). Note that CLDB supersedes Fid Location Data-Base (FLDB), see above on fids.
23
-
+ Data integrity. A container is a possible place to deal with data integrity issues. Alternatively this can be relegated to lower levels (self-checking device pairs) or higher levels (DMU-like file system with check-sums in meta-data).
7
+
+ A container is a storage for data or meta-data. There are several types of container: data container, meta-data container, possibly others.
8
+
9
+
+ A container provides a very simple interface: it has an internal namespace consisting of keys and provides methods to fetch and update records stored at a given key. Nomenclature of keys and constraints on record structure depend on container type. For data container, keys are logical page offsets and records are simply pages full of data without any internal structure. For meta-data container keys are opaque identifiers of meta-data records.
10
+
11
+
+ A container stores its contents in a backing store---a storage device. A container allocates space from its backing store and returns no longer used space back as necessary.
12
+
13
+
+ Local RAID is implemented by using a collection of containers to stripe data across. Note that local RAID is not implemented in the containers layer to avoid cyclic dependency between layers: the data structures (specifically layouts) that local RAID implementation is required to share with SNS are interpreted by Motr back-end that itself uses containers to store its data.
14
+
15
+
+ Snapshots. Containers are used for data and meta-data snap shotting. When a local snapshot is made as a part of object or system level snap shotting, a container involved into the snapshot is COW-ed, so that all updates are re-directed to a new container.
16
+
17
+
+ After a container is COW-ed no update should ever touch the blocks of the read-only primary container. This is a necessary prerequisite of a scalable fsck implementation, that will achieve reasonable confidence in system consistency by background scan and check of periodically taken global snapshots.
18
+
19
+
+ Migration. A container (either read-only or read-write) can migrate to another node. Typical scenarios of such migration are bulk re-integration of updates from a proxy-server to a set of primary servers and moving snapshots.
20
+
21
+
+ To make migration efficient, a container must be able to do a fast IO of its contents to another node. Specifically, it should be possible to send container contents over network without iterating through individual container records. This condition also implies that a container allocates space from its backing store in a relatively large contiguous chunks.
22
+
Self-identification. A container has an identity that is stored in the container and migrated together with the container. Higher layers address container records by (container-id, key) pairs and such addresses remain valid across container migrations, including "migration" where a hard drive is pulled out of one server and inserted into another. In the latter case the system must be able to determine what containers were moved between nodes and update configuration respectively, also it should able determine whether any storage or layout invariants were violated, e.g., whether multiple units of the same parity group are now located on the same server.
23
+
24
+
+ Layouts. Higher layers address and locate data and meta-data through container identifiers and container keys. On the other hand, a layout produces a location of a given piece of data or meta-data. Together this means that lowest level layouts produce locations in the form of (container-id, key) pairs.
25
+
26
+
+ A container can be merged into another container. Inclusion of one container into another can be done administrative reasons, to efficiently migrate a large number of smaller containers or for some other purpose. On inclusion, a container retains its identity (does it? Losing identity would require updating (and, hence, tracking) all references).
27
+
28
+
+ Fids. A fid (file identifiers) is an immutable globally and temporally unique file identifier. As file meta-data record ("inode") is stored in some container, it's logical to use (container-id, key) address of this record as fid. Note that this is formally quite similar to the structure of Lustre fid, composed of a sequence identifier and offset within the sequence.
29
+
30
+
+ CLDB. A method is required to resolve container addresses to node identifiers. This method should work while containers migrate between nodes and merge with each other. Container Location Data-Base (CLDB) is a distributed data-base tracking containers in the cluster. This database is updated transactionally on container migrations and merges (and splits? There must be splits for symmetry.). Note that CLDB supersedes Fid Location Data-Base (FLDB), see above on fids.
31
+
32
+
+ Data integrity. A container is a possible place to deal with data integrity issues. Alternatively this can be relegated to lower levels (self-checking device pairs) or higher levels (DMU-like file system with check-sums in meta-data).
0 commit comments