com.sun.jna.NativeLibraryTest fails on macOS 12 (Monterey)

[ezarko]

1. Version of JNA and related jars
   0774f82 master
2. Version and vendor of the java virtual machine
   java 17.0.2 2022-01-18 LTS
   Java(TM) SE Runtime Environment (build 17.0.2+8-LTS-86)
   Java HotSpot(TM) 64-Bit Server VM (build 17.0.2+8-LTS-86, mixed mode, sharing)
3. Operating system
   macOS Monterey 12.2.1 (21D62)
4. System architecture (CPU type, bitness of the JVM)
   Apple M1 Pro
5. Complete description of the problem
   [junit] Testsuite: com.sun.jna.NativeLibraryTest
   [junit] Tests run: 22, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.21 sec
   [junit]
   [junit] Testcase: testAvoidDuplicateLoads(com.sun.jna.NativeLibraryTest): FAILED
   [junit] Library should be newly loaded after explicit dispose of all native libraries expected:<1> but was:<10>
   [junit] junit.framework.AssertionFailedError: Library should be newly loaded after explicit dispose of all native libraries expected:<1> but was:<10>
   [junit] at com.sun.jna.NativeLibraryTest.testAvoidDuplicateLoads(NativeLibraryTest.java:91)
   [junit] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
   [junit] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit]
   [junit]
   [junit] Test com.sun.jna.NativeLibraryTest FAILED
6. Steps to reproduce
   ant

[matthiasblaesing]

From my perspective you are seeing a flaky test. From the failing test:

    // Give the system a moment to unload the library; on OSX we
    // occasionally get the same library handle back on subsequent dlopen
    Thread.sleep(2);

Please rerun the test.

[ezarko]

I tried it twice more. I got the same result each time.

[matthiasblaesing]

Sorry, not reproducible by me.

• 644b708
• 0774f82

On:

• aarch64 with the default JDK on that system Temurin-17.0.1+12 (the system provided by @dkocher)
• aarch64 with JDK Temurin-17.0.2+8 (latest Temurin build)
• amb64 with 15.0.2+7-27 (if I remember correctly this was an AdoptOpenJDK build)

all run clean.

[matthiasblaesing]

@dkocher indicated, that he can reproduce the problem. I still fail and we need to better pin point it. Could you (@ezarko and @dkocher ) please indicate, which ant version you are using?

[ezarko]

Apache Ant(TM) version 1.10.12 compiled on October 13 2021

[tresf]

Consolidating some mailing list discussions here:

@matthiasblaesing wrote:

Hi all,

I was preparing the release (the final PR is in). When doint this, I
noticed this test failure:

-  [junit] Testcase: testAvoidDuplicateLoads(com.sun.jna.NativeLibraryTest):   FAILED
- [...]

@dbwiddis wrote:

Haven't actually run any code yet but have a suspicion.

The failures began when the GitHub runners were changed from macOS 11 to macOS 12. So we could "fix" the test failures by forcing older macos version, but that just hides the problem.

Looking around the internet, it seems that dlopen behavior was changed in macOS 12 (contrary to posix standard) to search a fallback location. Multiple issues similar to this one cl-plus-ssl/cl-plus-ssl#138 Not sure if that could be relevant here.

I think playing around with the DYLD_LIBRARY_PATH environment variable could help isolate this.

cl-plus-ssl has some great information on this as well, quoting the author of that bug report:

Full disclosure, I don't have a Mac running Monterey yet so I can't exhaustively test this theory. But it seems to jibe with the reports seen recently in #114, what a coworker is showing me, and this community bug report.

It appears that in Monterey, calling dlopen(3) on a non-existent path will cause it to "helpfully" search for the missing library for you in some standard paths. The end result for cl-plus-ssl is that if MacPorts isn't installed, dlopen("/opt/local/lib/libcrypto.dylib") will instead be equivalent to dlopen("libcrypto.dylib"), insta-killing the process :(.

Given that it's Apple, we probably won't know if this is a bug or intentional for quite a while.

Sooooo, not sure what the best way forward is, but it seems like cl-plus-ssl can no longer count on throwing all the possible libcrypto and libssl paths for Darwin into the define-foreign-library and letting CFFI sort it all out.

Further down there's some really good information on this: cl-plus-ssl/cl-plus-ssl#138 (comment), which goes into specific detail on loading logic.

[matthiasblaesing]

Thank you for the inspiration.

I slightly modified the test and ran it in isolation with the DYLD_PRINT_APIS environment variable set to 1. This shows, that the test library is only dlopened once and dlclose is invoked on that handle. dyld does not report whether or not it thinks dlclose succeeded, but I added some debugging around it and (a) JNA already checks the return and raises an exception if dlclose would have failed and return value indicates, that it succeeded.

Reading this: rust-lang/rust#28794 (comment) being able to unload a library on mac OS or derivates is not something I'd rely on.

[tresf]

I tried changing all calls to RTLD_LOCAL to see if this could be coerced, but no luck. Also, no luck manipulating DYLD_<FOO> flags. No matter what I do, callCount() always remains at 10.

[tresf]

Pinging @dyorgio, he's helped me with some obscure macOS issues in the past. 🍻

[matthiasblaesing]

Here is the test setup:

https://github.com/java-native-access/jna/tree/macos_debugging

And this is the result:

https://github.com/java-native-access/jna/actions/runs/3913589087/jobs/6689649929

[dbwiddis]

And this is the result:

https://github.com/java-native-access/jna/actions/runs/3913589087/jobs/6689649929

I see two dlopen() calls at Line 1152 and again at Line 1264.

So the reference count is still 1 at dlclose().

Er, nevermind, there it is at Line 1247.

[dbwiddis]

I think I'm convinced this test should be removed (or altered to test the java-side reference, not the native handle), at least on macOS and possibly all *nix.

The POSIX spec does not require the OS to unload the library:

An application writer may use dlclose() to make a statement of intent on the part of the process, but this statement does not create any requirement upon the implementation.

and

Although a dlclose() operation is not required to remove any functions or data objects from the address space, neither is an implementation prohibited from doing so.

Some other poking about indicates that there may be a gcc compiler flag which could be relevant. This man page (on Linux but referencing gcc) might be of interest:

-fno-gnu-unique
On systems with recent GNU assembler and C library, the C++
compiler uses the "STB_GNU_UNIQUE" binding to make sure that
definitions of template static data members and static local
variables in inline functions are unique even in the presence
of "RTLD_LOCAL"; this is necessary to avoid problems with a
library used by two different "RTLD_LOCAL" plugins depending
on a definition in one of them and therefore disagreeing with
the other one about the binding of the symbol. But this
causes "dlclose" to be ignored for affected DSOs; if your
program relies on reinitialization of a DSO via "dlclose" and
"dlopen", you can use -fno-gnu-unique.