The NetX I/O application comes in two different forms:
- Desktop App (Mac OS and Windows)
- CLI Tool (Linux, Mac OS, Windows)
This article will familiarize you with the CLI (command-line interface) version. While the Desktop version makes it easy to set up multiple syncs, along with drag and drop importing, there are cases where you'll want to programmatically drive NetX I/O. For example, the CLI allows you to set up various sync point on a workgroup file server. It runs "headless" (no user interface), and can be easily wrapped in shell scripts and scheduled via cron (or others). The NetX I/O CLI is packages as a Jar file, and is executed with Java.
This outlines the steps needed to start running the NetX I/O command-line tool. This illustrates setting it up in "sync" mode, but if you want to use the tool in WatchedFolder or any other mode, the steps are almost the same.
Windows users: NetX I/O requires the Microsoft Visual C++ Redistributable to be installed before it can run. For instructions on how to obtain this package, see https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads.
Step 1: Make sure JDK is installed locally
You can test this on Linux or Mac by typing: "java -version" to check the Java version. Like this:
$ java -version java version "1.8.0_20" Java(TM) SE Runtime Environment (build 1.8.0_20-b26) Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)
If the version printed is anything other than "1.8.x", you will need to download a supported version of Java from Oracle: https://www.oracle.com/java/technologies/javase/javase8-archive-downloads.html
Step 2: Download the NetX I/O jar file
You can download the appropriate version (eg "netxio-4.0.xxx.jar") from our Service Portal, found in the /Modules/NetX I/O folder.
Step 3: Create a work location folder
Create a folder on your workstation for NetX I/O to write files. For the sake of illustration, let's say you're on a Mac, and you create a NetXIO folder inside your home directory:
Step 4: Copy over the jar file
Copy the netxio-3.0.xxx.jar file into your work location folder. Eg:
Step 5: Create your sync folder
The storage location you created in Step 3 is simply for managing administrative files. Now you need to choose or create a new folder that will be the root of all the synced sub-folders and files. Again, to illustrate, we'll add one off the Desktop; here for example:
/Users/me/Desktop/My Project Files
Really, you can call this anything you like, and it can live anywhere. Just choose something convenient to use. Do NOT nest the work location folder inside of your sync folder.
Step 6: Create (or choose) your sync folder (in NetX)
You can choose any folder that is not a top-level folder. However, please choose wisely: it's better choose a folder whose sub-structure is not very large. You can, but the sync times, bandwidth, and local storage requirements will obviously increase. Make sure that you are choosing a folder that the account you are running NetX under has appropriate access permissions.
For our example, we'll choose this folder in NetX:
Step 7: Create (or choose) a user (in NetX)
NetX I/O logs into NetX using supplied credentials for a pre-existing user, and actions will be performed on behalf of that user. It's best to create a separate account for your NetX I/O user. If you were to log in to NetX and then log out as the same user logged in via NetX I/O, it disrupts the processes in I/O. If you're running NetX I/O in Sync mode, you must configure it to run with an Administrator-level user. For Watched Folder mode, a Producer-level user (or higher) is required. You will need to know the username and password for that user you want to log into NetX with.
Step 8: Start NetX I/O
On a Mac, you can open a terminal, and type the following:
java -Xms512M -Xmx1024M -cp "/Users/me/NetxIO/netxio-4.0.001.jar" com.netxposure.external.client.io.NetxIO -url https://acme.netx.net -username me -password xxx -netxSourceFolder "/Projects/Me/Sync Files" -localSourceDir "/Users/me/Desktop/My Project Files" -mode sync -localStorageDir "/Users/me/NetxIO" -useInsecureSSL -loop 15
Ensure the "cp" directive is placed before the class (com.netxposure.external.client.io.NetxIO), or it will be interpreted as a parameter to the class. Instead, this is intended for Java. Also, this process could potentially use a large amount of RAM (depending on the total sync size; it will need to hold the entire local state map in memory, twice to compare). So in this example the Java heap is set to start with 512MB (-Xms512M) and can utilize up to 1024MB (-Xmx1024M). This is just an example.
Most of this should now be familiar to you, including the library directory, the username, password, folder, directory and local storage. But to formalize this, here's each flag with corresponding details (please refer to the Usage article for more details and options):
|This parameter sets the minimum memory for NetX IO. 512MB of RAM is usually sufficient. This parameter MUST come before "com.netxposure.external.client.io.NetxIO".|
|This parameter sets the maximum memory for NetX IO. 1024MB of RAM is usually sufficient. This parameter MUST come before "com.netxposure.external.client.io.NetxIO".|
|This parameter sets the Java classpath and must point to the NetX library directory. This parameter MUST come before "com.netxposure.external.client.io.NetxIO".|
|This is the URL of your NetX server instance.|
|This is the username of your NetX user.|
|This is the password of your NetX user.|
|This is the folder (or "category") in NetX which files and folders will sync.|
"/Users/me/Desktop/My Project Files"
|This is the local directory where folders and files are synced with the NetX server.|
|This sets NetX I/O into "Watched Folder" mode.|
|Local storage for NetX I/O administrative files, including logs, clientId data file, and lostandfound folder. Please see storage capacity note below.|
|This is actually not usually recommended. But setting up certificates is out of scope for a quick start guide.|
|This sets NetX I/O to run the sync process every 15 minutes.|
If you start NetX I/O using the terminal, you may want to background that process, because not doing so and closing the terminal will quit NetX I/O. Additionally, you may also want to add this command to your workstation startup items. There are many ways to deploy this tool more robustly, but that's out of scope for this quick start. This article's purpose is simply to get you up and running quickly.
Be aware of the capacity of the underlying storage when configuring the "localStorageDir" setting. If imports encounter significant errors, failed import files will be copied to the "lost and found" directory under this path. If that storage facility runs out of space, that will lead to secondary (and critical) errors with NetX I/O.
If the user account running I/O does not have appropriate access permissions to the NetX folders it is trying to sync, an error will be received:
[NetX I/O] ERROR (com.netxposure.external.client.rpc.ApiException: JSON-RPC Error 4003: Object not found (invalid category_id (18)) (null))
Step 9: Start using the sync
Drop folders and/or files into your local sync folder, and/or import new files into NetX. Wait for the loop time, and then see the folders and files replicated on the other side. Enjoy!
You can obtain details on all the NetX I/O by running the CLI tool with the "-help" flag:
java -cp "/netx/8.11/webapps/ROOT/WEB-INF/lib/*" com.netxposure.external.client.io.NetxIO -help usage: help -blind For watchedFolder, import, and export modes, specifying 'blind' means the analysis step will be skipped. When enabled, it can be much faster if you are not concerned with avoiding uploading duplicates. This option also allows for versioning of files on the NetX server (watchedFolder and import modes only). Optional. -clientId <arg> Normally NetX IO automatically creates a Client ID, but if you want to control this for any reason (advanced), you can set it with this option. Optional. -config <arg> Local path to a config file, which can be used in place of other commandline options. This config file must be a properties formatted file (name=value) -databaseMap Use the database temp map? The default is to use an in-memory map; it's faster, but may require more memory. Optional. -debug Log debug messages? Optional. -deleteLimit <arg> If you are running NetX IO in 'sync' mode, and if you delete more than a certain number of files in a single sync cycle, NetX IO will automatically halt because it will assume it's a mistake on the local workstation. The default delete limit is '100'. Setting this to zero ('0') disables any deletion limit. Optional. -help Print help message. Optional. -localSourceDir <arg> Local source directory from which to look for files. Required. -localStorageDir <arg> To enabled 'sync' mode, or to have logs write to a file, you need to provide a local storage location where files can be written. Required for 'sync' and 'merge' modes. -loop <arg> Loop allows NetX IO to run continuously. Please set to a number, measured in minutes. NetX IO will pause between scans based on the configured minutes. Zero is the default, in which case the scan only runs once. Optional. -mode <arg> Options include: 'watchedFolder' [Only send files and folders from client to NetX; delete client files when done, and leave folders.]; 'sync' [This mode compares the client sync map with the NetX map, and reconciles differences between the client and NetX. Supports limited moves and renaming changes, as well as deletes and additions.]; 'merge' [Merge client files and folders with NetX files and folders; nothing will be deleted. This runs the for the first time in sync mode, since there are no client state map at that point.]; 'import' [Only send files and folders from client to NetX; leave client files and folders when done.]; 'export' [Only send files and folders from NetX to the client.]; 'analyze' [Emails an import report to the calling user's email. No other action is taken; no files or folders will be affected.]. Required. -netxSourceFolder <arg> Remote NetX folder where to store the imports (eg "/Category1/Category2"); if the value is a number, then this is assumed to be the corresponding category ID. Required. -noHeartbeat If you are running NetX IO in a loop, by default it will set up a heartbeat with the remote server; if that heartbeat stops, it will notify you via email. Use this flag to disable this function. Optional. -nonvolatile By default, NetX IO will automatically scan the source directory, wait 20 seconds, and then scan again to ensure all the bytes on all the files are static (files are not still being copied). Adding this flag turns off that safety check such that the sync will commence immediately. Optional. -password <arg> Password. Required. -replacementCharacter <arg> If illegal characters are encounter in either files or folders, then use this replacement character (single). Optional. -report <arg> Set the reporting period. Available values are: 'daily', 'weekly', 'monthly', and 'none'. Default is 'weekly'. Optional. -timeout <arg> Advanced. By default, all web service calls timeout after 3 minutes. Only change this if you know what you are doing. Value is measured in minutes. Only non-zero values are valid. Optional. -url <arg> NetX instance URL (eg "http://acme.netx.net"). Required. -useInsecureSSL If you are connecting with HTTPS and can't register your SSL certificate with NetX Sync, this will disable any certificate verification. Optional. -username <arg> Username (for 'watchedFolder' mode, this must be an Producer-level user; for 'sync' mode, this must be an Administrator). Required.
Merge and Sync modes are experimental and are not supported for use in mission-critical environments. Use at your own risk.
Running multiple NetX I/O processes on a single workstation
There are cases where you may want to run multiple instances of NetX I/O on a single workstation — and that's perfectly fine. However please ensure that each instance uses its own local storage and sync folders. Do not attempt to run multiple instances of NetX I/O CLI with a shared local storage directory!
If you're using a config file (see details below), and order to identify the various processes running presumably in the background, you can add a -D parameter to java. The "-D" is normally a mechanism for passing name+value pairs to the JVM. However, the JVM is happy to ignore directives it doesn't understand. So you can add something like "clientId=35" to the command-line argument so that if you need to query the process table (speaking specifically of Unix platforms here — Linux and Mac), you can see that argument there.
If you want to do this, you must add the "-D" flag before the reference to com.netxposure.external.client.io.NetxIO. Eg:
$ java -DclientId=35 -Xms512M -Xmx1024M -cp "/Users/me/NetxIO/netxio-4.0.001.jar" com.netxposure.external.client.io.NetxIO -config "/Users/me/NetxIO/client1.properties"
Doing this means that if you type this on a Mac:
$ ps ax | grep java
You might see something like this:
26916 s010 S+ 0:03.92 /usr/bin/java -DclientId=1 -Xms512M -Xmx1024M -cp "/Users/me/NetxIO/netxio-4.0.001.jar" com.netxposure.external.client.io.NetxIO -config "/Users/me/NetxIO/client1.properties"
Using a config file instead of command-line arguments
You can also set all your NetX I/O options within a configuration file. In this scenario, you only set the configuration file (the "-config" option) in the command-line. Like this:
java -Xms512M -Xmx1024M -cp "/Users/me/NetxIO/netxio-4.0.001.jar" com.netxposure.external.client.io.NetxIO -config "/Users/me/NetxIO/netxio.properties"
Here's what an example configuration file might look like:
url=http://127.0.0.1 username=admin password=xxx netxSourceFolder=/My Files localSourceDir=/Users/me/Desktop/NetxIO/files localStorageDir=/Users/me/Desktop/NetxIO/store mode=sync debug=true clientId=My Client Example 35
Note that option names are separated from their corresponding values with an equals sign (=). Also, do not wrap config values with quotes (like you might do on the command-line). Additionally, please note that command-line arguments that require no arguments — like "debug" in this example — can be set with a value of "true".
I/O lock file behavior
What happens with NetX I/O CLI if the last execution is still running when cron tries to execute again? The lock file should stop it from executing concurrently. NetX I/O writes a lock file IF there's a configured "localStorageDir". There, it will write a tmp file with a timestamp. And that will regulate how often another NetX I/O process — configured with the same localStorageDir — can run. For "loop" based tasks, the time-lock is twice as long as the loop. If not, it's 30 mins. This file is then periodically updated throughout the processing of the run. The reason for the timestamp is so that if NetX I/O terminates unexpectedly and the lock file is left there, later runs of other I/O processes can note that the lock file itself is "stale" and proceed regardless.