The Code Generator

The com.neeve.tools.AdmCodeGenerator class, included in Talon ADM module (nvx-rumi-adm-<version.jar), is the main code generator that facilitates code generation. The AdmCodeGenerator creates classes from a model, organizing them into a namespace-qualified directory structure based on a specified root directory. Optional parameters can be provided.

Talon also offers plugin for build tools, such as Maven, to integrate code generation with the build lifecycle of Maven based applications.

Encoding Types

All classes created by the code generator ensure efficient serialization for storage and network transfer. Talon supports various serialization formats, known as encoding types. The code generator suppirts a parameter to specify the desired encoding type for which the classes are to be generated.

The following are the encoding types currently supported by the ADM code generator:

• Json: Javascript Object Notation serialization; not very efficient but highly readable

• Protobuf: Google's protobuf serialization format, which can be useful for portability purposes providing a reasonable balance between performance and interoperability. The generated classes for this encoding type internally use the classes generated by the Google Protobuf IDL compiler.

• Xbuf2: Talon's protobuf implementation. The classes generated by this encoding type support zero garbage operation, do not use the Google Protobuf IDL compiler and are 100% wire compatible with Google's protobuf wire format.

Deprecated Encoding Types

The Xbuf encoding format, which was a precursor to the Xbuf2 encoding format has been deprecated. Users of the Xbuf encoding format should migrate to the Xbuf2 encoding format. See Choosing an Encoding Type for more information.

Encoding Types Scheduled For Deprecation

Going forward, classes generated for all ADM encoding types will support serialization to and deserialization from the JSON serialized format. Therefore, the JSON encoding format will no longer be available starting with the next major X Platform release.

Starting with the next major release, The Xbuf2 encoding type will fully replace the Protobuf encoding type i.e. there will no capability to generate classes that produce and use classes generated by the Google Protobuf IDL compiler.

See Choosing an Encoding Type for more information on choosing an appropriate encoding.

Mixing Encoding Types

You can mix different encoding types across models at runtime, provided each model has a unique factory id ((since the factory id is used to determine which generated factory to use to decode a message). However, it is not legal to mix and match encodings within a model via import. For example, if you generate one model with Protobuf, it is not possible to use types generated using that model in a model generated with Xbuf2.

Schema Location

The XML schema for the ADM language (for the input models) - xadml.xsd - can be found at the root of nvx-rumi-adm-<version>.jar

Generated Class Namespace

Each model is associated with a namespace. This namespace is the default package for generated classes from a code generator. Users can specify a different namespace when invoking the code generator, allowing generation of sources with varied encodings of the same model by rerunning the generator with distinct namespaces.

Imported Model Resolution

To handle multi-module and multi-project builds, the ADM XML Parser, and Maven Build plugin search for model imports on the classpath using the imported model's fully qualified namespace. The model xml (and .proto for Xbuf2/Protobuf) will be copied to the generated source and target classes folder in fully qualified form for inclusion in the project's jar.

Import resolution searches both current classpath and OS file system when resolving an import. Recommended is to use relative classpath strings whenever possible. Consider the following example project with two models, model.xml imports other_model.xml:

${basedir}/src/some/package/model.xml [In xml we define namespace = "some.model.namespace"]
${basedir}/src/other/package/other_model.xml [In xml we define namespace = "some.other_model.namespace"]

The recommended way to define import would be one of the following:

Import Through Classpath

<!--
Method #1: (recommended) Import from relative model namespace.
Imports from classpath relative to namespace of importing model,
in this case some.model.namespace.
This would try resolving from
some/model/namespace/../../other_model/namespace/other_model.xml.
Evaluating ../.. would give us resource some/other_model/namespace/other_model.xml
-->
<import model="../../other_model/namespace/other_model.xml" />
 
 
<!-- Method #2 (recommended): Import from full model namespace.
This will work if generated sources are compiled and added to classpath.
For example if we have project dependencies where one project imports a
model XML from another project, the original XML file is not available,
but the one in the produced jar is. Therefore, we are getting file from
classpath, where project dependency containing import is added to the classpath.
For maven this also works within the same project if code generation for imported
other_model.xml is run first. So, when we reach in pom the code generation of model.xml,
other_model.xml is already generated and xml is copied to
target/classes/some/other_model/namespace/other_model.xml.
target/classes is already in classpath so this will resolve successfully to other_model.xml-->
<import model="some/other_model/namespace/other_model.xml" />

The reason why this works is that the result of running code generation would be:

${basedir}/target/classes/some/model/namespace/model.xml
${basedir}/target/classes/some/model/namespace/Model.proto
${basedir}/target/classes/some/model/namespace/GeneratedModelClass1.class
${basedir}/target/classes/some/model/namespace/GeneratedModelClass2.class
...
${basedir}/target/classes/some/other_model/namespace/other_model.xml
${basedir}/target/classes/some/other_model/namespace/OtherModel.proto
${basedir}/target/classes/some/other_model/namespace/GeneratedOtherModelClass1.class
${basedir}/target/classes/some/other_model/namespace/GeneratedOtherModelClass2.class
... 

The requirement is that code generation first runs for other_model.xml so that it can be found in classes folder at the time of running model.xml.

The following modes of import model resolution are provided for advanced use cases but are discouraged.

From Project Source Folder Through OS Filesystem

<!-- Method #3: Import from absolute filesystem path: -->
<import model="absolute path to file other_model.xml in the OS filesystem" />
 
<!-- Method #4:
Import as path relative to the provided modelsDir provided to the code generator
${modelsDir}/other/package/other_model.xml.
-->
<import model="other/package/other_model.xml" />
<!-- Method #5:
Import as relative filesystem path. Path is relative to
model that does the import so this is evaluated to
${basedir}/src/some/package/../other/package/other_model.xml.
If we evaluate ../ we get ${basedir}/src/some/other/package/other_model.xml
-->
<import model="../other/package/other_model.xml" />

Generated Protobuf IDLs

The Xbuf2 and Protobuf code generators generate the following Protobof IDLs (.proto file).

• Main IDL

  • This IDL is generated from the contents of the input model

• Platform Bundled IDLs

  • descriptor.proto

  • AdmTypes.proto

Main IDL

The Main IDL is generated for use by the user as well as for internal use by the Protobuf code generator. The Xbuf2 code generator generates it solely for the user and does not use it internally.

The Main IDL is output in the generated source folder using the model's fully qualified namespace.

Platform Bundled IDLs

The code generator bundles the following IDLs in the packaged jars to support custom options used by ADM to enrich handling of enums, and to define additional data types used by ADM.

descriptor.proto	google/protobuf/descriptor.proto	Defines options available in Protobuf
AdmTypes.proto	com/neeve/adm/types/protobuf/AdmTypes.proto	Defines custom enum options and additional ADM data types

Build Tool Integration

Talon currently supports the use of following code generators

• com.neeve.tools.AdmCodeGenerator

• Maven Plugins (layered on the AdmCodeGenerator)

The following describes how these can be run with various build tools.

Java

java –cp nvx-rumi-kernel-<version>.jar:nvx-rumi-core-<version>.jar:nvx-rumi-io-<version>.jar:nvx-rumi-adm-<version>.jar:[:jars-containing-imports] com.neeve.tools.AdmCodeGenerator -e <Encoding Type> –f <model-file>.xml -o target/generated-sources/nvx-adm

ANT

<target name="run-adm-generator">
    <java classname="com.neeve.tools.AdmCodeGenerator" 
          fork="true" 
          failonerror="true" 
          jvm="${build.java.home}/bin/java" 
          classpathref="build.classpath">
        <arg value="-e"/>
        <arg value="Xbuf"/>
        <arg value="-f"/>
        <arg value="${basedir}/src/main/java/com/acme/messages/messages.xml"/>
        <arg value="-o"/>
        <arg value="-o target/generated-sources/nvx-adm"/>
        <arg value="-d generateEmbeddedEntityInterfaces=false"/>
    </java>
</target>

Maven

When working with Maven, the following are the various options on how one can use the ADM code generator

• Using the Maven Exec Plugin

• Using the Maven X Platform Plugins

  • The ADM Plugin

  • The Platform Plugin

Using the Maven Exec Plugin

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>exec-maven-plugin</artifactId>
    <version>1.3.2</version>
    <executions>
        <execution>
            <id>generate-messages</id>
            <goals>
                <goal>exec</goal>
            </goals>
            <phase>generate-sources</phase>
            <configuration>
                <executable>java</executable>
                <arguments>
                    <argument>-classpath</argument>
                    <classpath/>
                    <argument>com.neeve.tools.AdmCodeGenerator</argument>
                    <argument>-e</argument>
                    <argument>Xbuf2</argument>
                    <argument>-o</argument>
                    <argument>${project.build.directory}/generated-sources/myapp/messages</argument>
                    <argument>-f</argument>
                    <argument>${project.basedir}/src/main/models/com/mycompany/myapp/messages/messages.xml</argument>
                </arguments>
            </configuration>
        </execution>
    </executions>
</plugin>

Using the Maven ADM Plugin

Plugin Goal	Description
generate, adm-generate	Generates code to a generated sources folder which would be included in the built jar.
generateTest, adm-generateTest	Generates code to a generated test sources folder which would be included in the built test jar

<plugins>
  <plugin>
    <groupId>com.neeve</groupId>
    <artifactId>nvx-adm-maven-plugin</artifactId>
    <version>${nvx.adm.version}</version>
    <executions>
      <execution>
        <id>Messages</id>
        <phase>generate-sources</phase>
        <goals>
        <goal>generate</goal>
        </goals>
        <configuration>
          <modelFile>${basedir}/src/main/java/com/mycompany/messages/messages.xml</modelFile>
          <encodingType>Xbuf2</encodingType>
          <directives>
            <generateEmbeddedEntityInterfaces>false</generateEmbeddedEntityInterfaces>
          </directives>
        </configuration>
      </execution>
     </executions>
   </plugin>
</plugins>

Using the Maven Platform Plugin

If your project is using the nvx-platform-bom and you want to generate code with the same version of the platform you are using, you may also use the nvx-platform-maven-plugin. The advantage of this approach is that you can use the same version of the plugin as the platform bom. Because maven BOMs don't cover plugin versions, using the nvx-adm-maven-plugin would mean that the nvx-adm-maven-plugin version would have to be specified separately.

<plugins>
  <plugin>
    <groupId>com.neeve</groupId>
    <artifactId>nvx-platform-maven-plugin</artifactId>
    <version>${nvx.platform.version}</version>
  </plugin>
</plugins>

Gradle, Ivy & Others

To create classes using the ADM code generator with build tools like Gradle, Ivy & others, invoke com.neeve.tools.AdmCodeGenerator using the appropriate mechanisms offered by these tools.

Code Generator Options

The following options are available for the ADM code generation:

Command Line	Maven Plugins (ADM & Platform)	Description	Default
-f, --file	modelFile	The input file specified either as OS path or URL	-
-m, --modelsdir	modelsDirectory	Base directory to use when searching for imported models that are not found on the project / generator's classpath.	-
-o, --outdir	projectOutputDirectory	Base output directory for the generated files. For maven plugin defaults to target/generated-[test]-sources/nvx-adm	-
-c, --classesdir	classesOutputDirectory	Classes output folder (to which generated resoures should be copied). May be specified multiple times to copy to multiple directories.	-
-e, --encoding	encodingType	Encoding type of content underlying the generated classes (Xbuf | Protobuf | Json)	Protobuf
-x, --xpcompat	protoXbufGenerationCompatibility	Wire compatibility between protobuf and xbuf generated classes (None | Xbuf | Protobuf	Protobuf
-y, --emptyifnullarray	generateArrayGetterEmptyIfNull	Instructs the code generator return empty arrays instead of null for unset array fields	false
-n, --namespace	namespace	Namespace override of model parsed from the input file (overrides namespace in model file if supplied)	-
-p, --protodir	N/A	An additonal directory in which to search for imported .proto files.	-
-d, --directive	directives	A key=value pair specifying a code generation directive. (May be specified multiple times). See Directives below.	-
-b, --buildinfo	buildInfo	String with build-time information such as project version, timestamp or machine. This is added to the AdmGenerated annotation of generated classes.	-
-i, --incremental	incrementalBuild	Trigger incremental code generation - run only if something changed since last run	false (true for maven plugin)
N/A	useBasicDeltaDetection	When running an incremental build, basic delta detection indicates that model's are rebuilt based on whether source model timestamp. With advanced delta detection dependencies are examined as well.	false
-u, --bundledir	modelBundleOutputDirectory	Directory to which to output model XML and IDL files if applicable	-
N/A	generateModelBundle	Indicates whether models with all their dependencies and IDLs should be output to modelBundleOutputDirectory	false
N/A	codegenListenerClassName	Class name of external listener to receive events from code generator. See ADM Code Generation Events.	-
N/A	codegenListenerProperties	Additional properties to pass to the constructed code gen listener.	-

Code Generator Directives

Some advanced properties can be passed to the code generator as directives. The following is a list of supported directives:

Directive	Description	Default
requireExplicitCollectionKeys	Directive indicating that map collections' contained entities must define an explicit field field to store the key for the entity when it is in the map. When this directive is false, it is possible that as the model evolves the implicitly generated key field could change and cause existing keys to be ignored on upgrade. 💡 It is recommend that new projects set this directive to true. SINCE 3.11	false
generateEmbeddedEntityInterfaces	Directive indicating that the generator should create interfaces for embedded entities. This can be disabled for applications with stringent performance requirements to reduce the overhead associated with multi-morphic vtable lookups.	true
generateEmbeddedEntitiesNonFinal	When this directive is set to true the generated entity class is not declared as final nor are its accessors. This feature can be useful for applications that need to mock embedded entities in test frameworks such as CGLIB, but is not recommended for production use for performance reasons. SINCE 3.8.189	false
generateDefaultGetters	Whether or not to generate default getters that accept a value to return when the field is not set. Not typically recommended	false
generateThrowOnUnsetGetters	Whether or not to generate getXXXOrThrow() or accessors that will throw an ERogFieldNotSetException when the field has not been set. This provides an alternative to calling hasXXX for a field to test if the field is unset. Usage of this directive is not recommended; hasXXX is the recommended approach to testing if a field is not set. Exception throwing is more expensive, and the generated getXXXOrThrow method introduces extra invocation overhead and a larger code size.	false
generateRequiredFieldValidators	Whether or not validation logic is generated in the types validators for required fields. Enabling this leads large generated code size, and validation checks are expensive, so this is not recommended for performance sensitive applications.	false
generateFluentSetters	Directive indicating that fluent style setters should be generated for fields. This can be enabled to generate fluent accessors on generated types. This can be useful for writing concise test code, but is more overhead, so it's usage is not typically recommended.	false
generateAllStringsPoolable	Directive indicating that all Strings fields in the model should be generated as poolable types regardless of the value of the field's poolable attribute.	false
pooledStringFieldTypeNameSuffixPolicy	Can specify None, Always or OnConflict to instruct the code generator as to how to handle naming conflicts that arise from a pooled string field type name generated from a field name are suffixed to avoid a name clash.	"None"
pooledStringFieldTypeNameSuffix	Specifies the suffice to use to resolve pooled string type name conflicts with Always or OnConflict suffixing policies.	"String"
generateProtobufClasses	Specified that protobuf classes should be generated using the protoc code generator in addition to the encoding type specific generated classes. This directive only applies to Xbuf and Xbuf2 encoding types.	"String"

Code Generation Events

ADM code generation is run by using com.neeve.tools.AdmCodeGenerator class. It is possible to supply a listener to the AdmCodeGenerator instance to subscribe to events that are fired at certain points in code generation run. The events are in the AdmCodeGenerator.CodeGenerateEventType enumeration as follows.

Event Types
/**
 * Code generation steps that external listener can listen for.
 */
public enum CodeGenerateEventType {
    /**
     * When code generation is about to run.
     */
    START,
    /**
     * After model has been parsed
     */
    MODEL_PARSED,
    /**
     * When code generation finished.
     */
    END,
    /**
     * When code generation is skipped because nothing changed since previous run
     * Can happen when incrementalBuild is on.
     */
    SKIP
}

An event listener can be supplied via the CODEGEN_EVENT_LISTENERS parameter to an instance of AdmCodeGenerator. The Listener interface is defined in AdmCodeGenerator as follows:

**
 * Event listener for code generation events.
 */
public interface CodegenListener {
    /**
     * When code generation event is triggered this method will be called.
     */
    void codeGenerateEvent(CodeGenerateEvent e);
}

The Maven plugin exposes this capability via the codegenListenerClassName configuration parameter. This parameter accepts the fully qualified name of the class that implements the listener interface. While the AdmCodeGenerator can accept multiple listeners, the Maven plugin accepts only one class and will create only one instance of that listener class per execution. The class shpuld be in project's build classpath (either in project being built or in one of its dependencies).

Each event is dispatched with an instance of AdmCodeGenerator.CodeGenerateEvent that contains the data for the event. The following are the key methods in this class

/**
 * Holds event data for code generation events.
 */
public class CodeGenerateEvent {
    /**
     * @return the eventType
     */
    public final CodeGenerateEventType getEventType() {
        return eventType;
    }
    /**
     * @return the model. Only available if eventType is {@link CodeGenerateEventType#MODEL_PARSED} or {@link CodeGenerateEventType#END}.
     */
    public final AdmModel getModel() {
        return model;
    }
    /**
     * @return the errorAggregator. Only available if eventType is {@link CodeGenerateEventType#MODEL_PARSED} or {@link CodeGenerateEventType#END}.
     */
    public final AdmSourceCodeErrorAggregator getErrorAggregator() {
        return errorAggregator;
    }
}

Sample: Running Additional Model Validations

Developers may tap into listener mechanism to perform additional model validations as given in example below. The example demonstrates how to use the listener mechanism to enforce globally unique field ids (meaning both the model for which code is generated and all its imports recursively).

package mypackage;
 
import java.util.HashSet;
import java.util.Set;
import com.neeve.adm.AdmField;
import com.neeve.adm.AdmModel;
import com.neeve.adm.AdmModelImport;
import com.neeve.adm.AdmSourceCodeErrorAggregator;
import com.neeve.tools.AdmCodeGenerator;
import com.neeve.tools.AdmCodeGenerator.CodeGenerateEvent;
public class CodegenListener implements AdmCodeGenerator.CodegenListener {
    @Override
    public void codeGenerateEvent(CodeGenerateEvent e) {
        switch (e.getEventType()) {
            case START:
                System.err.println("CodegenListener START");
                break;
            case SKIP:
                System.err.println("CodegenListener SKIP");
                break;
            case MODEL_PARSED:
                System.err.println("CodegenListener MODEL_PARSED " + e.getModel().getFullName());
                validateFieldIdsUnique(e.getModel(), e.getErrorAggregator(), new HashSet<Short>(), new HashSet<String>());
                // e.getErrorAggregator().add("Test external error", AdmSourceCodeErrorAggregator.Severity.ERROR, e.getModel().getCodeSource());
                break;
            case END:
                System.err.println("CodegenListener END " + e.getModel().getFullName());
                // e.getErrorAggregator().add("Test external error", AdmSourceCodeErrorAggregator.Severity.ERROR, e.getModel().getCodeSource());
                break;
        }
    }
    private void validateFieldIdsUnique(AdmModel model, AdmSourceCodeErrorAggregator aggregator, Set<Short> fields, Set<String> modelsProcessed) {
        // iterate imports and run validation for them if not already run. We will run bottom -> top validation so we assume that imported models have correct
        // field ids and the importing model doesn't if it defines the same id as in one of the imports.
        for (AdmModelImport modelImport : model.getModelImports()) {
            if (!modelsProcessed.contains(modelImport.getModel().getFullName())) {
                validateFieldIdsUnique(modelImport.getModel(), aggregator, fields, modelsProcessed);
                modelsProcessed.add(modelImport.getModel().getFullName());
            }
        }
        // now we check fields of current model
        for (AdmField field : model.getFields()) {
            if (fields.contains(field.getId())) {
                // aggregator collects errors and they will be displayed at the end of code generation.
                aggregator.add("Duplicate field id",
                               AdmSourceCodeErrorAggregator.Severity.ERROR,
                               field.getCodeSource(), null);
                // note that classes derived from AdmModelElement
                // usually have source code (error line) information retrieved through getCodeSource()
                // pointing back to place in XML where their XML representation is.
            }
            else {
                fields.add(field.getId());
            }
        }
    }
}

The listener is provided to the code generator as follows:

<plugins>
    <!-- Generates X model from XML model file -->
    <plugin>
        <groupId>com.neeve</groupId>
        <artifactId>nvx-core-maven-plugin</artifactId>
        <version>${project.version}</version>
        <executions>
            <execution>
                <id>Model</id>
                <phase>generate-sources</phase>
                <goals>
                    <goal>generate</goal>
                </goals>
                <configuration>
                    <!-- The usual configuration params go here ... -->
 
                    <!-- The listener class that should be on the build classpath for this project -->
                    <codegenListenerClassName>mypackage.CodegenListener</codegenListenerClassName>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>

Incremental Code Regeneration

The Maven ADM/Platform Plugin and ADM Code generator take a source model's last modified timestamp or checksum into account. Code generation will be skipped if:

• The model file has been changed since the last build.

• Any import model file has changed (checked recursively in imports of imports...)

• Any input option for code generation has changed such as encoding type, namespace, directives etc.

The incremental code generation works by tracking above given changes in an XML file which may be found in output dir. The file has a name with a pattern like this:

.${model_filename}.xml_${md5checksum}.metadata. model_name is the name of the model file for which code was generated. md5checksum is a signature calculated from input options given to code generator so that if any of them changes, the resulting filename no longer represents same code generation. Stored in this file are input options given to code generator and list of models with a number that would be different every time model file is persisted to disk. These files do not go into the jar and may be deleted at any time (which they usually do when a clean build is triggered).