-
Notifications
You must be signed in to change notification settings - Fork 301
Convert file system implementations to use C file system API in tensorflow #1111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Before we can migrate to modular file system, additional work on tensorflow side is needed. The following is the progress of related PRs: |
Additional fixes that need to happen on tensorflow/tensorflow: |
One more fix (remove duplication of |
cc @zouxu09 Would anyone in your team be interested in helping out with OSS storage support? |
@terrytangyuan One remaining issue in AZFS is that we commented out logging temporarily https://github.com/tensorflow/io/pull/1143/files#r517351641 This issue can be addressed by using C logging API in TensorFlow. A PR in tensorflow repo has been created to expose the header files in pip package: |
With AZFS and HTTP file system converted to modular file system C API we can thinking about maintain forward-compatibility now. One thing we can do, is to lazy load the
I have created a PR #1208 for that purpose. The only outstanding issue is that, if we lazy-load /cc @terrytangyuan do you know if there is any timeline with respect to oss moving to modular file system C API? |
Uh oh!
There was an error while loading. Please reload this page.
As C file system API in tensorflow is available (see tensorflow/community#101), we should start migrate the file system implementations in tensorflow-io to use the C file system API.
This issue track the progresses for the following:
The text was updated successfully, but these errors were encountered: