|
| 1 | +# Building and Contributing to TensorFlow Java |
| 2 | + |
| 3 | +## Building |
| 4 | + |
| 5 | +To build all the artifacts, simply invoke the command `mvn install` at the root of this repository (or the Maven command of your choice). It is also |
| 6 | +possible to build artifacts with support for MKL enabled with |
| 7 | +`mvn install -Djavacpp.platform.extension=-mkl` or CUDA with `mvn install -Djavacpp.platform.extension=-gpu` |
| 8 | +or both with `mvn install -Djavacpp.platform.extension=-mkl-gpu`. |
| 9 | + |
| 10 | +When building this project for the first time in a given workspace, the script will attempt to download |
| 11 | +the [TensorFlow runtime library sources](https://github.com/tensorflow/tensorflow) and build of all the native code for your platform. This requires a |
| 12 | +valid environment for building TensorFlow, including the [bazel](https://bazel.build/) |
| 13 | +build tool and a few Python dependencies (please read [TensorFlow documentation](https://www.tensorflow.org/install/source) |
| 14 | +for more details). |
| 15 | + |
| 16 | +This step can take multiple hours on a regular laptop. It is possible though to skip completely the native build if you are working on a version that |
| 17 | +already has pre-compiled native artifacts for your platform [available on Sonatype OSS Nexus repository](#Snapshots). You just need to activate |
| 18 | +the `dev` profile in your Maven command to use those artifacts instead of building them from scratch |
| 19 | +(e.g. `mvn install -Pdev`). |
| 20 | + |
| 21 | +Modifying the native op generation code (not the annotation processor) or the JavaCPP configuration (not the abstract Pointers) will require a |
| 22 | +complete build could be required to reflect the changes, otherwise `-Pdev` should be fine. |
| 23 | + |
| 24 | +### Native Builds |
| 25 | + |
| 26 | +In some cases, like when adding GPU support or re-generating op classes, you will need to re-build the native library. 99% of this is building |
| 27 | +TensorFlow, which by default is configured for the [CI](.github/workflows/ci.yml). The build configuration can be customized using the same methods as |
| 28 | +TensorFlow, so if you're building locally, you may need to clone the [tensorflow](https://github.com/tensorflow/tensorflow) project, run its |
| 29 | +configuration script (`./configure`), and copy the resulting |
| 30 | +`.tf_configure.bazelrc` to `tensorflow-core-api`. This overrides the default options, and you can add to it manually (i.e. adding `build --copt="-g"` |
| 31 | +to build with debugging info). |
| 32 | + |
| 33 | +### GPU Support |
| 34 | + |
| 35 | +Currently, due to build time constraints, the GPU binaries only support compute capacities 3.5 and 7.0. |
| 36 | +To use with un-supported GPUs, you have to build it yourself, after changing the value [here](tensorflow-core/tensorflow-core-api/build.sh#L27), |
| 37 | +setting the environment variable `TF_CUDA_COMPUTE_CAPABILITIES`, or configuring it in a bazel rc file ( |
| 38 | +i.e. `build --action_env TF_CUDA_COMPUTE_CAPABILITIES="6.1"`). While this is far from ideal, we are working on getting more build resources, and for |
| 39 | +now this is the best option. |
| 40 | + |
| 41 | +To build for GPU, pass `-Djavacpp.platform.extension=-gpu` to maven. By default, the CI options are used for the bazel build, see the above section |
| 42 | +for more info. If you add `bazelrc` files, make sure the `TF_CUDA_COMPUTE_CAPABILITIES` value in them matches the value set elsewhere, as it will take |
| 43 | +precedence if present. |
| 44 | + |
| 45 | +## Running Tests |
| 46 | + |
| 47 | +`ndarray` can be tested using the maven `test` target. `tensorflow-core` and `tensorflow-framework`, however, should be tested using |
| 48 | +the `integration-test` target, due to the need to include native binaries. It will **not** be ran when using the `test` target of parent projects, but |
| 49 | +will be ran by `install` or `integration-test`. If you see a `no jnitensorflow in java.library.path` error from tests it is likely because you're |
| 50 | +running the wrong test target. |
| 51 | + |
| 52 | +### Native Crashes |
| 53 | + |
| 54 | +Occasionally tests will fail with a message like: |
| 55 | + |
| 56 | +``` |
| 57 | +Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.0:test(default-test)on project tensorflow-core-api:There are test failures. |
| 58 | +
|
| 59 | + Please refer to C:\mpicbg\workspace\tensorflow\java\tensorflow-core\tensorflow-core-api\target\surefire-reports for the individual test results. |
| 60 | + Please refer to dump files(if any exist)[date]-jvmRun[N].dump,[date].dumpstream and[date]-jvmRun[N].dumpstream. |
| 61 | + The forked VM terminated without properly saying goodbye.VM crash or System.exit called? |
| 62 | + Command was cmd.exe/X/C"C:\Users\me\.jdks\adopt-openj9-1.8.0_275\jre\bin\java -jar C:\Users\me\AppData\Local\Temp\surefire236563113746082396\surefirebooter5751859365434514212.jar C:\Users\me\AppData\Local\Temp\surefire236563113746082396 2020-12-18T13-57-26_766-jvmRun1 surefire2445852067572510918tmp surefire_05950149004635894208tmp" |
| 63 | + Error occurred in starting fork,check output in log |
| 64 | + Process Exit Code:-1 |
| 65 | + Crashed tests: |
| 66 | + org.tensorflow.TensorFlowTest |
| 67 | + org.apache.maven.surefire.booter.SurefireBooterForkException:The forked VM terminated without properly saying goodbye.VM crash or System.exit called? |
| 68 | + Command was cmd.exe/X/C"C:\Users\me\.jdks\adopt-openj9-1.8.0_275\jre\bin\java -jar C:\Users\me\AppData\Local\Temp\surefire236563113746082396\surefirebooter5751859365434514212.jar C:\Users\me\AppData\Local\Temp\surefire236563113746082396 2020-12-18T13-57-26_766-jvmRun1 surefire2445852067572510918tmp surefire_05950149004635894208tmp" |
| 69 | + Error occurred in starting fork,check output in log |
| 70 | + Process Exit Code:-1 |
| 71 | + Crashed tests: |
| 72 | + org.tensorflow.TensorFlowTest |
| 73 | + at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:671) |
| 74 | + at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533) |
| 75 | + at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:278) |
| 76 | + at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:244) |
| 77 | +``` |
| 78 | + |
| 79 | +This is because the native code crashed (i.e. because of a segfault), and it should have created a dump file somewhere in the project that you can use |
| 80 | +to tell what caused the issue. |
| 81 | + |
| 82 | +## Contributing |
| 83 | + |
| 84 | +### Formatting |
| 85 | + |
| 86 | +Java sources should be formatted according to the [Google style guide](https://google.github.io/styleguide/javaguide.html). It can be included |
| 87 | +in [IntelliJ](https://github.com/google/styleguide/blob/gh-pages/intellij-java-google-style.xml) and |
| 88 | +[Eclipse](https://github.com/google/styleguide/blob/gh-pages/eclipse-java-google-style.xml). |
| 89 | +[Google's C++ style guide](https://google.github.io/styleguide/cppguide.html) should also be used for C++ code. |
| 90 | + |
| 91 | +### Code generation |
| 92 | + |
| 93 | +Code generation for `Ops` and related classes is done during `tensorflow-core-api`'s `compile` phase, using the annotation processor in |
| 94 | +`tensorflow-core-generator`. If you change or add any operator classes (annotated with `org.tensorflow.op.annotation.Operator`), endpoint methods ( |
| 95 | +annotated with `org.tensorflow.op.annotation.Endpoint`), or change the annotation processor, be sure to re-run a |
| 96 | +`mvn install` in `tensorflow-core-api` (`-Pdev` is fine for this, it just needs to run the annotation processor). |
| 97 | + |
| 98 | +### Working with Bazel generation |
| 99 | + |
| 100 | +`tensorflow-core-api` uses Bazel-built C++ code generation to generate most of the `@Operator` classes. See [Native Builds](#native-builds) for |
| 101 | +instructions on configuring the bazel build. To run the code generation, use the `//:java_op_generator` target. The resulting binary has good help |
| 102 | +text (viewable in |
| 103 | +[op_gen_main.cc](tensorflow-core/tensorflow-core-api/src/bazel/op_generator/op_gen_main.cc#L31-L48)). Generally, it should be called with arguments |
| 104 | +that are something like: |
| 105 | + |
| 106 | +``` |
| 107 | +bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/libtensorflow_cc.so --output_dir=src/gen/java --api_dirs=bazel-tensorflow-core-api/external/org_tensorflow/tensorflow/core/api_def/base_api,src/bazel/api_def |
| 108 | +``` |
| 109 | + |
| 110 | +(called in `tensorflow-core-api`). |
0 commit comments