> For the complete documentation index, see [llms.txt](https://docs.xplatform.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.xplatform.com/talon/operating-applications/deployment/native-libraries.md). # Native Libraries How Talon's bundled native libraries get onto disk, how to share them between instances, and how to diagnose a failure to load them. ## Overview Parts of the platform are implemented natively. Those libraries ship inside the platform JAR, and the JVM cannot load a library from inside an archive, so Talon extracts them to a directory on disk at startup and loads them from there. For most deployments this is invisible and needs no configuration. It becomes worth understanding in three situations: * Several instances start at once against the same extraction directory, typically because they share a home directory or a mounted volume. * The libraries need to be on disk before the process starts, for instance baked into a container image. * A library fails to load and you need to know whether extraction is the cause. The directory is set by `nv.native.libdir` and defaults to `${user.home}/.nvx/native`. See [Configuration Reference](/talon/reference/configuration.md#native-library-properties) for that and the other three properties named on this page. ## How Extraction Works At startup, before it loads anything, the platform: 1. Locates the bundled libraries on the classpath. 2. Takes an exclusive lock on `.nvx-extract.lock` in the extraction directory, waiting if another process holds it. 3. Compares the libraries on the classpath against the record in `.nvx-extract.complete`. If they match, extraction is skipped. 4. Otherwise clears the directory, extracts, and writes `.nvx-extract.complete` as the final step. 5. Releases the lock and loads the libraries. Both bookkeeping files live in the extraction directory alongside the libraries themselves. Neither is removed by the clearing step: the lock file is what another process may be waiting on, and on Windows it cannot be deleted while that process holds it open. ### Why the Record Is a Fingerprint `.nvx-extract.complete` holds a version string and a fingerprint of the library content, and it is the fingerprint that decides whether to re-extract. Recording only the version would be simpler and wrong. A SNAPSHOT build, or a locally rebuilt platform JAR, changes the libraries without changing the version string. A version-keyed record would treat the directory as already up to date and the process would load the previous build's libraries, which fails later and somewhere else. Fingerprinting the content means a changed library is always re-extracted, whatever the version says. ### Why the Record Is Written Last A process that dies part way through extraction leaves a partial set of libraries and no record. The next start finds no record, so it clears and extracts again rather than trusting what it finds. There is no state in which a partial extraction looks complete. ## Sharing an Extraction Directory Several instances may share one `nv.native.libdir` safely, including when they start simultaneously. The lock serializes them: the first to acquire it extracts, the rest find a matching record and skip. No instance observes a directory another is still writing to. {% hint style="info" %} This was not always the case. Earlier releases cleared and re-extracted on every start with no locking, so concurrent starts could interleave and one process could load a partially written library. The recommended workaround was a one-time standalone extraction followed by `nv.native.suppressextraction=true`. That workaround still works but is no longer necessary; a shared directory needs no special handling. {% endhint %} ## Extracting Ahead of Time To place the libraries on disk before any service starts, for example when building a container image, run the extractor directly: ```bash NATIVE_DIR=/opt/talon/native java -Djava.library.path=$NATIVE_DIR \ -Dnv.native.libdir=$NATIVE_DIR \ -cp nvx-core-all--.jar \ com.neeve.NativeKernel ``` Services can then be started with `nv.native.suppressextraction=true`, which skips extraction entirely and writes nothing to the directory. This is worth doing when the extraction directory is read-only at runtime, or when you want image build time rather than service start time to carry the cost. It is no longer necessary merely because instances share a directory. {% hint style="warning" %} Pre-extracted libraries are not checked against the platform JAR at startup, because suppressing extraction suppresses the check along with it. Upgrading the platform without re-running the extractor leaves the previous version's libraries in place. Make the extraction step part of whatever produces the image, so the two cannot drift. {% endhint %} ## Configuring the Environment The libraries have to be *found* once they are on disk. Extraction only writes the files; it does not by itself make them loadable, because the platform loads them with `System.loadLibrary`, which goes through the JVM's normal library search rather than an absolute path. **Either `java.library.path` or `LD_LIBRARY_PATH` must point at the extraction directory.** Setting one is enough; the two take different routes to the same result. ### What each one does `java.library.path` is a JVM property. It is what `System.loadLibrary` searches, and it governs finding the *first* library only - the JVM plays no part in resolving what that library depends on in turn. `LD_LIBRARY_PATH` is read by the operating system's dynamic loader. On Linux the JVM also seeds `java.library.path` from it at startup, which is why setting `LD_LIBRARY_PATH` alone is sufficient: it satisfies the JVM's lookup *and* the loader's. ### Behaviour of each combination | Setting | First library found by | Sibling libraries resolved by | Works? | | ------------------------ | -------------------------------------------------------------------- | ----------------------------------------- | ------ | | `java.library.path` only | the explicit property | the libraries' own `RUNPATH` of `$ORIGIN` | yes | | `LD_LIBRARY_PATH` only | `java.library.path`, which the JVM seeds from it | `LD_LIBRARY_PATH` | yes | | both | `java.library.path` - an explicit `-D` replaces the seeded value | `LD_LIBRARY_PATH` first, then `RUNPATH` | yes | | neither | nothing - the default path does not include the extraction directory | n/a | **no** | With both set, they are not in conflict: the explicit `-Djava.library.path` decides which directory the JVM searches for `libnvxkrnl`, while `LD_LIBRARY_PATH` still governs the loader's resolution of that library's dependencies. If the two point at different directories you can load `libnvxkrnl` from one and its siblings from the other, which is rarely what you want - keep them the same. ```bash # either of these is sufficient java -Djava.library.path=/opt/talon/native -Dnv.native.libdir=/opt/talon/native ... export LD_LIBRARY_PATH=/opt/talon/native java -Dnv.native.libdir=/opt/talon/native ... ``` If you override `nv.native.libdir`, point whichever variable you use at that same directory. {% hint style="info" %} `LD_LIBRARY_PATH` used to be **required**, not optional. The libraries referenced one another by bare name, so `java.library.path` on its own found the first library and then failed on a sibling. They now record a `RUNPATH` of `$ORIGIN` - "the directory this library was loaded from" - so each library locates its siblings relative to itself and `java.library.path` alone is enough. Existing deployments need no change. `RUNPATH` is `DT_RUNPATH`, which the dynamic loader searches *after* `LD_LIBRARY_PATH`, so anything that sets `LD_LIBRARY_PATH` today resolves libraries in exactly the order it always has. {% endhint %} To confirm what a library records: ```bash readelf -d libnvxkrnl.so | grep -E 'RPATH|RUNPATH' # 0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN] ``` Note that this applies to the libraries as a set. If you copy individual libraries out of the extraction directory and separate them, `$ORIGIN` no longer points at their siblings and loading fails again - keep them together. ## Troubleshooting Set `nv.native.debug=true` to trace the sequence to `stderr`. This is the first thing to turn on when a library fails to load, because it distinguishes an extraction problem from a loading problem. A normal first start looks like this: ``` [nv.native] Initializing X native runtime... [nv.native] ...bundled native libs found: jar:file:/opt/talon/lib/nvx-core-all.jar!/nvx-native/ [nv.native] extracting /home/talon/.nvx/native/libnvxkrnl.so [nv.native] ...attempting to load native libraries. [nv.native] ...native libraries loaded [nv.native] ...initialization complete: X native runtime is AVAILABLE. ``` A subsequent start against an unchanged platform skips the extraction step: ``` [nv.native] ...bundled native libs found: jar:file:/opt/talon/lib/nvx-core-all.jar!/nvx-native/ [nv.native] ...native libs already extracted for 3.16.70:a3f21c0b9d4e17ff, skipping extraction. ``` ### Reading the Output | Line | Meaning | | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `...bundled native libs found: ` | The libraries were located. The URL shows which JAR or directory they came from, which is the fastest way to spot the wrong artifact on the classpath. | | `extracting ` | This process is doing the extraction. One line per library. | | `...native libs already extracted for , skipping extraction.` | The directory already holds these libraries. Normal on every start after the first. | | `...native library extraction suppressed.` | `nv.native.suppressextraction` is set. Nothing was written or checked. | | `...no bundled native libraries found.` | Nothing on the classpath carries them. Usually the wrong artifact, or a slim JAR where the all-inclusive one was intended. | | `...failed to load native libraries []` | Extraction succeeded but the JVM could not load the result. Look at `java.library.path`, at whether the libraries match the platform and architecture, and - if the error names a *different* library than the one being loaded - at whether the libraries have been separated from one another. See [Configuring the Environment](#configuring-the-environment). | | `...initialization complete: X native runtime is UNAVAILABLE.` | Startup continues without native support unless `nv.native.exitifnonativelibs` is set. | ### Common Situations **A library fails to load and no `extracting` lines appear.** Either the directory was already up to date, in which case the problem is the library or `java.library.path` rather than extraction, or extraction is suppressed and the pre-extracted libraries are stale. Compare the marker against a run with a cleared directory. **Extraction happens on every start.** The fingerprint is changing, which means the libraries on the classpath are changing. Expected against a SNAPSHOT build; unexpected against a release, and worth checking whether the classpath is being assembled differently each time. **The service starts but behaves as though the platform were unavailable.** Look for `X native runtime is UNAVAILABLE`. Set `nv.native.exitifnonativelibs=true` where a service genuinely cannot run without native support, so this fails at startup rather than presenting later as unexplained behaviour. ## Related Topics * [Configuration Reference](/talon/reference/configuration.md#native-library-properties) - `nv.native.libdir`, `nv.native.suppressextraction`, `nv.native.exitifnonativelibs`, `nv.native.debug` * [Trace Logging](/talon/operating-applications/analysis-and-troubleshooting/trace-logging.md) - Configuring trace output for the rest of the platform