You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on May 3, 2024. It is now read-only.
The document is intended to specify the design of Mero FDMI interface. FDMI is a part of Mero product. FDMI provides interface for Mero plugins and allows horizontally extending the features and capabilities of the system.
15
16
16
-
## 1.2 Intended Audience
17
+
## 1.2 Intended Audience ##
17
18
* Product Architect
18
19
* Developers
19
20
* QA Engineers
20
21
21
-
## 1.3 Definitions, Acronyms, and Abbreviations
22
+
## 1.3 Definitions, Acronyms, and Abbreviations ##
22
23
FDMI: File data manipulation interface
23
24
24
-
## 1.4 References
25
+
## 1.4 References ##
25
26
1. “Mero Object Store Architecture: Technical” MeroTechnicalWhitepaper.pdf
26
-
1. “mero a scalable storage platform” Mero technical (toi).pdf
27
-
1. fdmihighleveldecomposition.pdf
27
+
2. “mero a scalable storage platform” Mero technical (toi).pdf
28
+
3. fdmihighleveldecomposition.pdf
28
29
29
-
# Overview
30
+
# Overview #
30
31
Mero is a storage core capable of deployment for a wide range of large scale storage regimes, from cloud and enterprise systems to exascale HPC installations. FDMI is a part of Mero core, providing interface for plugins implementation. FDMI is build around the core and allows for horizontally extending the features and capabilities of the system in a scalable and reliable manner.
31
32
32
-
## 1.5 Product Purpose
33
+
## 1.5 Product Purpose ##
33
34
TBD
34
35
35
-
## 1.6 Assumptions and Limitations
36
+
## 1.6 Assumptions and Limitations ##
36
37
TBD
37
38
38
39
# Architecture
@@ -51,7 +52,7 @@ In this section only architectural information like the following is displayed b
51
52
52
53
53
54
54
-
## 1.7 FDMI position in overall Mero Core design
55
+
## 1.7 FDMI position in overall Mero Core design ##
55
56
56
57
FDMI is an interface allowing Mero Core scale horizontally. The scaling includes two aspects:
57
58
@@ -74,7 +75,7 @@ Considering the amount of data Mero Core operates with it obvious that plug-in t
74
75
75
76
Source in its turn refreshes its own subset of filters against the database. The subset is selected from overall filter set based on the knowledge about data types the source is able to feed FDMI with as well as operation with the data the source supports.
76
77
77
-
## 1.8 FDMI Roles
78
+
## 1.8 FDMI Roles ##
78
79
FDMI consists of APIs implementing particular roles in accordance with FDMI use cases. The roles are:
79
80
80
81
* Plug-in dock, responsible for:
@@ -94,8 +95,8 @@ FDMI consists of APIs implementing particular roles in accordance with FDMI use
94
95
95
96

96
97
97
-
## 1.9 FDMI Plugin dock
98
-
### 1.9.1 initialization
98
+
## 1.9 FDMI Plugin dock##
99
+
### 1.9.1 initialization###
99
100
100
101

101
102
@@ -111,7 +112,7 @@ Further initialization consists of registering a number of filters in **filterd
111
112
**NB:**
112
113
TBD if we really need to determine the moment when all sources appear to be running filter sets consistent across the whole system. Currently we need to consider if Plug-in should be notified about this point.
113
114
114
-
### 1.9.2 Data Processing
115
+
### 1.9.2 Data Processing###
115
116

116
117
117
118
Remote FDMI instance running Source Dock role provides data payload via RPC channel. RPC sink calls back local FDMI
@@ -128,8 +129,8 @@ confirmation it deregisters the associated RPC sink and plug-in’s callback fun
128
129
129
130
All registered sources are notified about changes in filter set, if any occurred as the result of plug-in coming off
130
131
131
-
## 1.10 FDMI Source Dock
132
-
### 1.10.1 Initialization
132
+
## 1.10 FDMI Source Dock##
133
+
### 1.10.1 Initialization###
133
134

134
135
135
136
* TBD where to validate, on Source side or inside FDMI
@@ -138,7 +139,7 @@ Source registration starts with registering source callbacks in local FDMI insta
138
139
As an optimization, execution plan could be built for every added filter to be kept along with the one. As an option, execution
139
140
plan can be built on every data filtering action to trade off memory consumption for CPU ticks.
140
141
141
-
### 1.10.2 Input Data Filtering
142
+
### 1.10.2 Input Data Filtering###
142
143

143
144
144
145
*In case of RPC channel failure, input data reference counter has to be decremented. TBD if this to be done explicitly or in context of
@@ -162,7 +163,7 @@ When done with traversing, FDMI engine calculates final Boolean result for the f
162
163
163
164
The method “get next filter” is the place where the behavior is to be implemented.
Input data may require to remain locked in the Source until the moment when plug-in does not need it anymore. The data processing inside plug-in is an asynchronous process in general, and plug-in is expected to notify corresponding source allowing it to release the data. The message comes from plug-in to FDMI instance hosting the corresponding source.
@@ -171,7 +172,7 @@ Input data may require to remain locked in the Source until the moment when plug
171
172
TBD: We need to establish a way to resolve resource id to FDMI instance hosting particular source. Most probably
172
173
resource id itself may contain the information, easily deduced or calculated.
0 commit comments