JSON Document Storage

VMware GemFire is an in-memory distributed Key-Value datastore. As a datastore, GemFire provides a real-time, consistent and distributed service for modern applications with data-intensive needs and low latency response requirements. Because of GemFire’s distributed peer-to-peer nature it can take advantage of multiples servers to pool memory, cpu and disk storage for improved performance, scalability and fault tolerance to build applications needing caching, management of in-flight data or the key-value database of record.

Goal

The goal of this tutorial is to introduce using and storing JSON documents in GemFire, in particular using the newly introduced in GemFire 10 the JsonDocumentFactory and JsonDocument classes.

Prerequisite Required Software

• VMware GemFire 10.0 or later
• Apache Maven
• Java Developer Kit (JDK) 11
• (optional) Integrated Development Environment (IDE) such as Microsoft Visual Studio Code (vscode) or JetBrains IntelliJ IDEA

Download Examples and Configure Environment

Download and install VMware GemFire from Broadcom Support Portal. Follow the installation instructions in the GemFire documentation.

Clone the GemFire examples repository from GitHub.

$ git clone [email protected]:gemfire/gemfire-examples.git

Set GemFire home environment variable to top of GemFire install directory. Note for this example GemFire is installed in the home directory of the user - adjust as nessagery for local environment and install directory location.

$ export GEMFIRE_HOME=${HOME}/gemfire

Configure PATH to GemFire bin directory for access to gfsh utility.

$ export PATH=${PATH}:${GEMFIRE_HOME}/bin

Validate Java 11 and Maven install.

$ java -version

openjdk version "11.0.17" 2022-10-18
OpenJDK Runtime Environment (build 11.0.17+8-post-Ubuntu-1ubuntu2)
OpenJDK 64-Bit Server VM (build 11.0.17+8-post-Ubuntu-1ubuntu2, mixed mode, sharing)

$ mvn --version

Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 11.0.17, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-amd64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.19.0-29-generic", arch: "amd64", family: "unix"

In some environments it may be helpful to configure the JAVA_HOME environmental variable if issues are encountered or multiple Java versions are installed, along with setting a local PATH adding the JDK install directory.

Configure Access to GemFire Maven Repository

The quick start tutorial requires access to the Broadcom Maven Repository for the GemFire product jars. Navigate to the Broadcom Support Portal. Login (or register if you have not already). Click Show All Releases and find “Click Green Token For Repository Access” (don’t click the blue text; click the green icon to the right of it).

Once sign-up is completed, add the following to the settings.xml file in .m2 directory within the home directory. Make sure to replace the email and password with those used during sign-up.

<settings>
    <servers>
        <server>
            <id>gemfire-release-repo</id>
            <username> <!-- Email sign-up--> </username>
            <password> <!-- Replace with your token--> </password>
        </server>
    </servers>
</settings>

The pom.xml file provided with the examples is already configured with a pointer to the VMware GemFire maven repository and makes use of the GemFire 10.1.1 beta 1 version of the product.

Start a Developer GemFire Cluster

Start a GemFire cluster with a locator, configured PDX and a server.

$ gfsh start locator --dir=/home/${USERNAME}/locator --name=locator

$ gfsh -e "connect" -e "configure pdx --read-serialized=true --disk-store"

$ gfsh -e "connect" -e "start server --name=server1 --dir=/home/${USERNAME}/server1"

Create Region with GFSH

As a best practice each region should only contain a single type of instance data. Hence make sure to create a new region for each kind of data expected to be stored in GemFire and don’t mix key types or values into an existing region with different data.

Create a “petrecords” region to hold the example data using gfsh.

$ gfsh -e "connect" -e "create region --name=petrecords --type=PARTITION_PERSISTENT"

Command Line Build

Use the following commands to build and run the client application at the terminal.

Build the client application with Maven and copy dependencies to target directory (note - commands should all be issued in the example directory that contains the pom.xml file).

$ mvn clean compile dependency:copy-dependencies package

Set the classpath and run the client with the Java 11 virtual machine.

$ java -cp target/GemFireClient-1.0-EXAMPLE.jar:target/dependency/*  com.vmware.gemfire.examples.quickstart.GemFireClient

Managing JSON Documents?

Starting in GemFire 10.0 and later, a new API for managing JSON documents is provided. JSON documents are converted to a JsonDocument type for serialization to the GemFire servers. The JsonDocument instance is an immutable type, that implements the org.apache.geode.cache.Document interface. A default JsonDocument uses the BSON format to internally store the JSON data. JsonDocument’s support server side operations such as query without conversion to another type.

To create a JsonDocument, use an instance of JsonDocumentFactory obtained from RegionService and use create(String json) method to parse the JSON string returning a JsonDocument instance.

A JsonDocument can be converted back to a JSON string with the toJson() method. Individual fields of original JSON can accessed via the getField(String fieldName). The getField(String) method returns a Java object, the following table shows the mapping from JSON field to Java type. For example calling getField(String) for data that was a JSON array would return a Java List instance.

For additional details on using JSON with GemFire see documentation

For class details see JavaDocs for JsonDocument and JsonDocumentFactory.

See BSON and JSON specifications for additional information on JSON.

Client Application

A client application will access and communicate with GemFire through ClientCache and Region instances.

ClientCache cache;
Region<Integer, JsonDocument> region;

Properties clientCacheProps = new Properties();
clientCacheProps.setProperty("log-level", "config");
clientCacheProps.setProperty("log-file", "client.log");

Cache Creation and Region

Create client cache with properties and configure locators pool.

cache = new ClientCacheFactory(clientCacheProps)
    .addPoolLocator("127.0.0.1", 10334).create();

Create a region proxy “petrecords” matching one created on the server.

region = cache.<Integer, JsonDocument>createClientRegionFactory(
        ClientRegionShortcut.PROXY).create("petrecords");

JSON Document Storage

Create a String from a JSON document.

String jsonPetRecord = "{" +
    " petname:\"Spot\"," +
    " idNum:0," +
    " breed:\"Poodle\"," +
    " owner:\"Bill\"," +
    " currentOnVaccines:true," +
    " issues:[\"needs special diet\",\"pulls on leash when walked\"]" +
"}";

Get the default JsonDocumentFactory from the Cache (RegionService).

JsonDocumentFactory jdf = cache.getJsonDocumentFactory();

Use JsonDocumentFactory to convert JSON to JsonDocument instances.

try {
    petRecord = jdf.create(jsonPetRecord);
} catch (JsonParseException e) {
    e.printStackTrace();
}

Put JSON document into GemFire servers.

region.put(0, petRecord);

Get key 1 from servers returning a JsonDocument. Output JSON.

pt = region.get(1);
System.out.println("JSON Pet Record: " + pt.toJson());

List the field names of the JsonDocument.

List list = (List) pt.getFieldNames();
System.out.println("JSON Fields: " + list.toString());

Fetch field “petname” from JsonDocument.

System.out.println("PetName Field: " + pt.getField("petname"));

Fetch the Array field “issues” and display the first element.

System.out.println("Issues Field: "
    + ((List) (pt.getField("issues"))).get(0));

Compare two JSON documents.

if (pt.equals(petRecord2)) {
    System.out.println("Documents are equal");
}

The client should output to stdout, the following messages once it runs with a little pre and post logging. There will also be a client.log file created with the client configuration and other useful logging for reviewing the runtime behavior.

JSON Pet Record: {"petname":"Firehouse","idNum":1,"breed":"Dalmatian","currentOnVaccines":true,"owner":"Joe","issues":["bites mail person"]}
JSON Fields: [petname, idNum, breed, currentOnVaccines, owner, issues]
PetName Field: Firehouse
Issues Field: bites mail person
Documents are equal

Close Client

Prior to exiting client close local GemFire cache and connection pool.

System.out.println("Closing Client");
cache.close();

Cleanup GemFire Processes

Shutdown the GemFire Locator and Server.

$ gfsh -e "connect" -e "shutdown --include-locators=true"