Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Native Libraries

How Talon's bundled native libraries get onto disk, how to share them between instances, and how to diagnose a failure to load them.

Overview

Parts of the platform are implemented natively. Those libraries ship inside the platform JAR, and the JVM cannot load a library from inside an archive, so Talon extracts them to a directory on disk at startup and loads them from there.

For most deployments this is invisible and needs no configuration. It becomes worth understanding in three situations:

  • Several instances start at once against the same extraction directory, typically because they share a home directory or a mounted volume.

  • The libraries need to be on disk before the process starts, for instance baked into a container image.

  • A library fails to load and you need to know whether extraction is the cause.

The directory is set by nv.native.libdir and defaults to ${user.home}/.nvx/native. See Configuration Reference for that and the other three properties named on this page.

How Extraction Works

At startup, before it loads anything, the platform:

  1. Locates the bundled libraries on the classpath.

  2. Takes an exclusive lock on .nvx-extract.lock in the extraction directory, waiting if another process holds it.

  3. Compares the libraries on the classpath against the record in .nvx-extract.complete. If they match, extraction is skipped.

  4. Otherwise clears the directory, extracts, and writes .nvx-extract.complete as the final step.

  5. Releases the lock and loads the libraries.

Both bookkeeping files live in the extraction directory alongside the libraries themselves. Neither is removed by the clearing step: the lock file is what another process may be waiting on, and on Windows it cannot be deleted while that process holds it open.

Why the Record Is a Fingerprint

.nvx-extract.complete holds a version string and a fingerprint of the library content, and it is the fingerprint that decides whether to re-extract.

Recording only the version would be simpler and wrong. A SNAPSHOT build, or a locally rebuilt platform JAR, changes the libraries without changing the version string. A version-keyed record would treat the directory as already up to date and the process would load the previous build's libraries, which fails later and somewhere else. Fingerprinting the content means a changed library is always re-extracted, whatever the version says.

Why the Record Is Written Last

A process that dies part way through extraction leaves a partial set of libraries and no record. The next start finds no record, so it clears and extracts again rather than trusting what it finds. There is no state in which a partial extraction looks complete.

Sharing an Extraction Directory

Several instances may share one nv.native.libdir safely, including when they start simultaneously. The lock serializes them: the first to acquire it extracts, the rest find a matching record and skip. No instance observes a directory another is still writing to.

This was not always the case. Earlier releases cleared and re-extracted on every start with no locking, so concurrent starts could interleave and one process could load a partially written library. The recommended workaround was a one-time standalone extraction followed by nv.native.suppressextraction=true. That workaround still works but is no longer necessary; a shared directory needs no special handling.

Extracting Ahead of Time

To place the libraries on disk before any service starts, for example when building a container image, run the extractor directly:

Services can then be started with nv.native.suppressextraction=true, which skips extraction entirely and writes nothing to the directory.

This is worth doing when the extraction directory is read-only at runtime, or when you want image build time rather than service start time to carry the cost. It is no longer necessary merely because instances share a directory.

Pre-extracted libraries are not checked against the platform JAR at startup, because suppressing extraction suppresses the check along with it. Upgrading the platform without re-running the extractor leaves the previous version's libraries in place. Make the extraction step part of whatever produces the image, so the two cannot drift.

Configuring the Environment

The libraries have to be found once they are on disk. Extraction only writes the files; it does not by itself make them loadable, because the platform loads them with System.loadLibrary, which goes through the JVM's normal library search rather than an absolute path.

Either java.library.path or LD_LIBRARY_PATH must point at the extraction directory. Setting one is enough; the two take different routes to the same result.

What each one does

java.library.path is a JVM property. It is what System.loadLibrary searches, and it governs finding the first library only - the JVM plays no part in resolving what that library depends on in turn.

LD_LIBRARY_PATH is read by the operating system's dynamic loader. On Linux the JVM also seeds java.library.path from it at startup, which is why setting LD_LIBRARY_PATH alone is sufficient: it satisfies the JVM's lookup and the loader's.

Behaviour of each combination

Setting
First library found by
Sibling libraries resolved by
Works?

java.library.path only

the explicit property

the libraries' own RUNPATH of $ORIGIN

yes

LD_LIBRARY_PATH only

java.library.path, which the JVM seeds from it

LD_LIBRARY_PATH

yes

both

java.library.path - an explicit -D replaces the seeded value

LD_LIBRARY_PATH first, then RUNPATH

yes

neither

nothing - the default path does not include the extraction directory

n/a

no

With both set, they are not in conflict: the explicit -Djava.library.path decides which directory the JVM searches for libnvxkrnl, while LD_LIBRARY_PATH still governs the loader's resolution of that library's dependencies. If the two point at different directories you can load libnvxkrnl from one and its siblings from the other, which is rarely what you want - keep them the same.

If you override nv.native.libdir, point whichever variable you use at that same directory.

LD_LIBRARY_PATH used to be required, not optional. The libraries referenced one another by bare name, so java.library.path on its own found the first library and then failed on a sibling. They now record a RUNPATH of $ORIGIN - "the directory this library was loaded from" - so each library locates its siblings relative to itself and java.library.path alone is enough.

Existing deployments need no change. RUNPATH is DT_RUNPATH, which the dynamic loader searches after LD_LIBRARY_PATH, so anything that sets LD_LIBRARY_PATH today resolves libraries in exactly the order it always has.

To confirm what a library records:

Note that this applies to the libraries as a set. If you copy individual libraries out of the extraction directory and separate them, $ORIGIN no longer points at their siblings and loading fails again - keep them together.

Troubleshooting

Set nv.native.debug=true to trace the sequence to stderr. This is the first thing to turn on when a library fails to load, because it distinguishes an extraction problem from a loading problem.

A normal first start looks like this:

A subsequent start against an unchanged platform skips the extraction step:

Reading the Output

Line
Meaning

...bundled native libs found: <url>

The libraries were located. The URL shows which JAR or directory they came from, which is the fastest way to spot the wrong artifact on the classpath.

extracting <path>

This process is doing the extraction. One line per library.

...native libs already extracted for <marker>, skipping extraction.

The directory already holds these libraries. Normal on every start after the first.

...native library extraction suppressed.

nv.native.suppressextraction is set. Nothing was written or checked.

...no bundled native libraries found.

Nothing on the classpath carries them. Usually the wrong artifact, or a slim JAR where the all-inclusive one was intended.

...failed to load native libraries [<error>]

Extraction succeeded but the JVM could not load the result. Look at java.library.path, at whether the libraries match the platform and architecture, and - if the error names a different library than the one being loaded - at whether the libraries have been separated from one another. See Configuring the Environment.

...initialization complete: X native runtime is UNAVAILABLE.

Startup continues without native support unless nv.native.exitifnonativelibs is set.

Common Situations

A library fails to load and no extracting lines appear. Either the directory was already up to date, in which case the problem is the library or java.library.path rather than extraction, or extraction is suppressed and the pre-extracted libraries are stale. Compare the marker against a run with a cleared directory.

Extraction happens on every start. The fingerprint is changing, which means the libraries on the classpath are changing. Expected against a SNAPSHOT build; unexpected against a release, and worth checking whether the classpath is being assembled differently each time.

The service starts but behaves as though the platform were unavailable. Look for X native runtime is UNAVAILABLE. Set nv.native.exitifnonativelibs=true where a service genuinely cannot run without native support, so this fails at startup rather than presenting later as unexplained behaviour.

Related Topics

  • Configuration Reference - nv.native.libdir, nv.native.suppressextraction, nv.native.exitifnonativelibs, nv.native.debug

  • Trace Logging - Configuring trace output for the rest of the platform

PreviousDeployment
NextAnalysis & Troubleshooting

Last updated