Administration Guide

Before installing, it is important to know that LeanXcale has a distributed architecture and it consists of several components:

There are other components used by the system, that are not relevant for the user and are not described here. For example, spread is a communication bus used by LeanXcale components.

Never install or run leanXcale as root.

To install, you need:

The install program itself requires a unix system with python 3.7 (or later).

Your sales representative will provide you with a valid license file. In case you have any issues with it, please send an email to sales@leanxcale.com.

The LeanXcale zip distribution can be found at:

There is a zip file per version. Download the latest version.

If needed, install the required software dependencies on your system:

Here, and in what follows, unix$ is the system prompt used in commands and examples.

Working on the directory where the distribution has been downloaded, unzip the LeanXcale zipped distribution:

Here, you type the command after the unix$ prompt, and the following lines are an example of the expected output.

Verify that the process completed successfully and there is a directory lxdist (with the distribution packages) and the lxinst program:

To install, run the lxinst program and, as an option, supply the name of the installation directory (/usr/local/leanxcale by default):

The command takes the zipped LeanXcale distribution from ./lxdist, and installs it at the target folder.

The detailed configuration file built by the install process is saved at lxinst.conf, as a reference.

The database has users. The user lxadmin is the administrator for the whole system and during the install you are asked to provide its password.

To complete the install, a license must be added, as explained in the next section.

As another example, this installs at the given partition /dev/sdc2, with both encryption and compression, mounting the installed system at /usr/local/leanxcale:

The final messages printed by the install program will remind you of how to list or remove the installed file systems.

The installation creates the lx command. Use it to execute commands to operate the installed system.

At the end of the installation, the install program prints suggestions for adjusting the PATH variable so that lx will be available as any other command. For example, as done with:

when the install directory is /usr/local/leanxcale.

The lx command can be found at the bin directory in the install directory. For example, at /usr/local/leanxcale/bin/lx when installing at /usr/local/leanxcale.

In what follows, we assume that lx can be found using the system PATH.

The first command used is usually the one to install a license file:

It is also possible to put the license file at ~/.lxlicense, and the install process will find it and install it on the target system(s).

When a preinstalled leanXcale docker image is available, disregard this section and proceed as described in the next one.

This section explains how to create containers from leanXcale images and how to install licenses and change the lxadmin password for them.

Provided the leanXcale image named lx:2, and the docker network lxnet, this command runs a leanXcale docker container:

In this command, the container name is lx1, the network used lxnext, and the image used lx:2. The port redirection -p…​ exports the SQL port to the underlying host.

It is important to know that:

The container name (lx1) can be used to issue commands. For example, this removes the container after stopping it:

The installed container includes the lx command. Use it to execute commands to operate the installed DB system.

It is possible to attach to the container and use the ''lx'' command as it can be done on a bare metal host install:

Here, we type docker attach lx1 on the host, and lx version on the docker container prompt.

Note that if you terminate the shell reached when attaching the docker container, it will stop. Usually, this is not desired.

It is possible to execute commands directly on the executed container. For example:

executes lx version on the container.

The program lxaws is used to list or remove AWS installs. This program is not built as such. To create it, copy lxinst to lxaws and use it.

Given a region, without any tags, it lists the tags installed:

Given a tag, it lists the tag resources as found on AWS:

It is also possible to supply just the base tag without the domain, as in

With flag -e prints commands to set environment variables with resources found, as an aid to run other scripts.

When more than one tag is asked for, or more than one instance/volume is found, variable names are made unique adding a number to the name, for example:

With flag -d, it removes the resources for the tags given. In this case, tags must be given explicitly in the command line.

To list, open, and close ports exported by the AWS install to the rest of the world, use lxaws flags -p (print ports), -o (open ports), and -c (close ports).

In all cases, the first argument is the tag for the install. The tag can be just the AWS install tag name, without .aws.leanxcale.com.

For example, this command lists the open ports:

Here, the protocol and port (or port range) is printed for each set of open ports. The CIDR printed shows the IP address range that can access the ports, and is 0.0.0.0/0 when anyone can access them.

Before with each port range, the name for the open port range is printed. This name is set by the default install, and can be set when opening ports as shown next.

To open a port range, use -o and give as arguments the tag for the install, the protocol, first and last port in range, the CIDR (or any if open to everyone), and a name to identify why this port range is open (no spaces). For example:

The new port will be shown as open if we ask for ports:

As another example:

To close a port, use -c and give as arguments the tag for the install, the protocol, a port within the range of interest, and the CIDR used to export the port. Note that any can be used here too instead of the CIDR 0.0.0.0/0 For example:

Here, we used the -v flag (verbose) to see what is going on.

As another example, this can be used to close the open port range 8888-10000 from the example above:

Peering connections can be used to bridge two VPCs at AWS.

One peer asks the other peer to accept a peering connection, the peer accepts the connection, and network routes and security group rules for access are configured.

Peering connections are handled using lxaws with the peering connection flags. If you do not have lxaws, copy lxinst to a file named lxaws and give it execution permissions.

These are the flags for peering connections:

To request a peering connection, supply as arguments

For example:

Here, we could have used xample.aws.leanxcale.com instead. The command prints the peering connection identifier, to be used for setting up networking and asking the peer administrator to accept the peering request.

To accept a peer request, supply as an argument the peering connection id, as in:

In either case, once the dialed peer accepts the request, networking must be set supplying as arguments

For example:

Our local CIDR block is 10.0.120.0/24. This must be given to the peer, so the peer system can setup routing for this block to our network, along with the VPC id and our security group id.

This information can be retrieved using lxaws as described in the previous section. For example:

Should it be necessary, the CIDR block used by the install can be set when installing the system (but not later), using the property awsnet, as in

Note that only the network address bytes used are given, instead of using 10.0.120.0/24.

Once done with a peering connection, it can be dismantled supplying both the tag and the peering connection identifier. The identifier is always given because, when accepting a peering request, the peering connection does not belong to us. But, using the command above you can retrieve such identifier easily.

When peering is no longer desired, the peering connection can be removed.

Removing the peering connection also removes the routing entries for it and the security group access rules added when it was setup.

The command lx is a Shell for running LeanXcale control programs. This simply fixes the environment for the installed host and runs the command given as an argument:

The command operates on the whole LeanXcale system, even when multiple hosts are used.

It is convenient to have lx in the PATH environment variable, as suggested in the install program output.

Command output usually includes information on a per-host basis reporting the progress of the used command.

Most commands follow the same conventions regarding options and arguments. We describe them here as a convenience.

Arguments specify what to operate (e.g., what to start, stop, etc.) may be empty to rely on the defaults (whole DB) or may specify a particular host and/or component name:

This may be repeated to specify different hosts and/or components.

The special host names db, repl, and repl2 may be used and stand for hosts without the nodb attribute, hosts for the first replica, and hosts for the second replica (hosts that are a mirror of other ones).

The start command starts LeanXcale:

Here, atlantis started a few processes and, once done, the start command checked out if the processes are indeed alive.

In case not all components can be started successfully, the whole LeanXcale system is halted by the start command.

To start a single host or component, use its name as an argument, like in:

Start does not wait for the system to be operational. To wait until the system is ready to handle SQL commands, the status command can be used with the -w (wait for status) flag, as in:

Without the -w flag, the command prints the current status, which can be stopped, failed, running, or waiting.

To start LeanXcale installed on Docker containers, you must start the containers holding the installed system components.

For example, consider the default docker install

The install process created a docker image named lx:2, installed for the docker host lx1, and the docker network lxnet.

To list the image we can

And, to list the networks

The created image is a single one for all containers. The name given when creating the container determines the host name used The install process specified the host names, and containers must be starte using the corresponding host name(s), so they know which leanXcale host they are for.

For example, to start the container for lx1:

In this command, the container name is lx1, the network used lxnext, and the image used lx:2. The port redirection -p…​ exports the SQL port to the underlying host.

Listing docker processes shows now the running container

It is important to know that:

The container name (lx1) can be used to issue commands. For example,

executes lx version on the lx1 container.

The status for the system can be seen in a similar way:

Note that the container will not start the DB if no valid license is found.

To start LeanXcale installed on AWS, you must start the AWS instances holding the installed system components.

Once started, the lx command is available at any of the installed system instances.

For example, after statring xample.aws.leanxcale.com, and provided the PEM file can be found at xample.pem, we can run this:

to see the installed version.

The command lx status reports the status for the system or waits for a given status. For example,

Or, to wait until the status is running:

To see the status for each one of the processes in the system, use lx procs. For example:

Before looking at the LeanXcale system status, it is important to look at the status of the docker containers running LeanXcale components.

When containers are running, the command lx status reports the status for the system or waits for a given status. For example,

executes lx status on the lx1 container. The status is reported for the whole system, and not just for that container.

To wait until the status is running:

To see the status for each one of the processes in the system, use lx procs. For example:

Before looking at the LeanXcale system status, it is important to look at the status of the AWS instances running LeanXcale components.

When instances are running, the command lx status reports the status for the system or waits for a given status. For example,

to see the system status.

To wait until the status is running:

To see the status for each one of the processes in the system, use lx procs. For example:

The stop command halts LeanXcale:

Stopping the LeanXcale containers should be done after stopping leanxcale. The reason is that docker might timeout the stop operation if the system is too busy updating the disk during the stop procedure.

To stop the database,

stops the components for the whole system (being lx1 an installed container).

We can double check this

Once this is done, we can stop the docker container.

We can also remove the container, but, note that doing this removes all data in the container as well.

To stop leanXcale on AWS, you must stop leanXcale before stopping the AWS instances running it. For example

This stops the system on all the instances it uses.

Flag -i performs an incremental backup, made with respect to the last total backup made on the same backup location.

This is done after the system is stopped. The system should not be running while doing a cold backup.

For example, from the external backup host in the running example:

Or, from the installed system backup in our example:

The system should not be running while doing a cold backup.

Remember to use flag -z or flag -C if the backups made were compressed or encrypted.

To perform a hot backup, backup the redo logs as an incremental dump by specifying lxqe:

Remember to use flag -z or flag -C if the backups made were compressed or encrypted. For example,

is the command if backups at this place are encrypted.

With flag -p, both lxbackup and lx backup list the known backups:

Those with a + in their names are incremental backups. Those encrypted are reported as such.

To remove a backup, it suffices to remove its directory. For example:

Here we used the flag “-d” for lx to change to $LXDIR before executing the remove command, which makes it easy to name the directory used for the dump.

Or, from our external backup example host:

Beware that if you remove a backup, you should remove those incremental backups that follow up to the next total backup.

To restore a backup, use the -r flag for lxbackup or lx backup and name the backup directory to restore.

For example, this restores the given full backup path:

Do this while the system is stopped.

To restore an incremental backup (and therefore, all previous incremental backups and the total backup made before them), supply an incremental backup path.

This restores the previous total backup made and the incremental backups that follow up to the given one.

To automate system backups, use crontab(8) or lxbackup to an external backup host or to run lx backup to an installed host, on the desired times.

This is the install program:

Given arguments specify where to install the DB. They may be paths for directories in the local host, or host names, or host and directory names using the syntax host:dir.

Instead of arguments, a configuration file may be given using the -f flag. The configuration file has been described before in this document.

Flag -k may be used to specify the key to download the distribution.

Flag '-K' must be used when installing on AWS to supply the path (without file name extension) to the pem/pub key files.

By default, the distribution is retrieved from the leanXcale artifactory. However, it is possible to supply one or more times the -d flag giving it the name for a .tgz file, a directory, or a URL. The special argument leanxcale stands for the official repository for the distribution.

Distribution packages will be retrieved by looking at those distribution sources. By convention, the first source is usally the ./lxdist directory, used to keep the distribution when downloaded.

For example, this installs just what is downloaded at ./lxdist, without downloading anything else:

Once downloaded, files are not downloaded again. Flag ''-c'' cleans the ''./lxdist'' directory to force a download of everything. If a different directory is specified by the user, it is not cleaned, although packages not found will be still downloaded there. To force a download of particular packages, remove the packages desired from the ''./lxdist'' directory (or the directory specified in the arguments).

For example, this downloads a fresh copy of the distribution and installs it:

And this tries first the ../lxdist directory and then the standard leanxcale repository:

This is actually the default when no distribution source is specified. Also, if no directory source is specified as a first option, lxdist is used as a directory to download components (other than files found on the local host).

The database has users (different from UNIX users). The user lxadmin is the administrator for the whole system. During the install you will be asked to type a password for lxadmin. To specify the password without being asked, you can set the password in the LXPASS environment variable:

When the users file ./lxowner exists, that file is used as the file describing the lxadmin user and its secret, instead of creating one using the user supplied password for lxadmin.

Do not use a existing file unless you know what you are doing. The reason is that this file must define lxadmin as the first user and that such user must have access to all resources, as configured.

Flag -i makes lxinst ignore system limits, to install on places with reduced disk or memory and in a hurry.

By default, the install performs a large install, trying to use all the machine resources for the service. Flag -s selects a small instead, and flag -m selects a medium install.

The install size affects components and sizes not specified by the user in command line arguments or in the configuration file. In a small install, at most 4 kdvs are added per host, and component memory is limited to 1GiB. In a medium install, at most 8 kvds components are added, and component memory is left with default values.

Flag -u is used to update a previous install. Still WIP, do not use it for now.

Shell for running LeanXcale control programs. This simply fixes the environment for the installed host and runs the command given as an argument:

When flag -d is used, the current working directory is set to the install directory at the host before running the given command.

Most commands follow the same conventions regarding options and arguments. We describe them here for convenience:

Flag -l is used when executing the command locally. This is used by the lx command framework, and should not be used in general by the end user.

Flag -D enables verbose diagnostics

Arguments specify what to operate (e.g., what to start, stop, etc.) may be empty to rely on the defaults (whole DB) or may specify a particular host and/or component name:

This may be repeated to specify different hosts and/or components.

The special host names db, repl, and repl2 may be used and stand for hosts without the nodb attribute, hosts for the first replica, and hosts for the second replica (hosts that are a mirror of other ones).

Add the given file(s) to the lib directory of the installed hosts, to add or update a library or jar in the installation:

Create, list, and restore backups from a installed host:

This command copies disk contents to a backup directory or restores them. See lxbackup for backups/restores on external hosts.

By default, the backup is $LXDIR/dump at the host used to issue backup commands. The convention is to use the first host to keep backups.

This directory can be a mount point or a link to a remote directory to keep backups at a different machine.

Directories under dump/ are named using the date, with .1, .2, etc. appended if more than one backup is created on the same date.

For incremental backups, a final + is added to the directory name.

For example, create a full, cold, backup when leanXcale is not running:

The printed path is the path for the directory keeping the backup, as used when restoring it.

Using flag -z makes the tool gzip backup files, preserving modification times despite compression. For example:

Using flag -C both encrypts and compresses the backup files. The flag argument is the path to the file keeping the file used to encrypt/decrypt.

To create an incremental (cold) backup:

It is possible to backup just one or a few components, supplying arguments that address them like in other commands, as described before. In this case, it is advisable to perform always backups for the same set of components. This command does not know if the last full backup is for the whole database or just for a few components. The last full backup is considered the full backup for the whole thing.

In general, the system should not be running while doing a backup. To perform a hot backup, backup just the query engine files as an incremental dump:

Incremental backups can be mixed for different components, as they overwrite some of the files saved in the full backup.

Create a full backup for just kvds100:

Create an incremental backup adding just kvds200:

List the backups known:

Those with a + in their names are incremental backups. Those encrypted are reported as such.

List the backups known for any kvds:

Restore the backup with the given name:

Here, if the name refers to an incremental backup, the restore recovers also the files found in previous incremental backups and in the total backup made before them.

Also, target directories to restore are identified by the current configuration of the system. That is, if kvds100 changed its location to a different host, the restore process will restore its disk at the new location, not at the location used to create the backup.

Restore just the files for kvds components from the given backup:

To restore a backup, it is usually desirable to format the disk for the involved components before using backup to restore their disks. The restore process copies restored files back, but does not remove anything else found on the disk for the component. For example:

To remove a backup, it suffices to remove its directory. For example:

Here we used the flag “-d” for lx to change to $LXDIR before executing the remove command, which makes it easy to name the directory used for the dump.

An important note is that file modification times are preserved in backup files. They are used to learn if a file must be copied or not in an incremental backup. If a backup is relocated to a different place, and later copied back into the standard location, preserve file modification times in the process.

Using flag -z makes the tool gzip backup files, preserving modification times despite compression.

When restoring encrypted backups, flag -C must be used to supply the path to the file keeping the secret used to decrypt.

Create, list, and restore backups from an external host:

This command can be copied along with the installed configuration to an external host, to perform external backups/restores. The external host must have ssh access to the installed hosts.

See lx backup for backups/restores on installed hosts.

Using lxbackup is exactly like using lx backup with a few differences:

For example, create a full backup for just kvds100:

The lxbackup command can be found at the installed $LXDIR/bin directory on any installed host. If you are not sure regarding the $LXDIR value, use this command to find it:

Flag -d for lx makes it change to the $LXDIR directory before running the given command.

The detailed configuration file to be used is created at lxinst.conf when installing. The configuration can be retrieved also by the lx config command. For example:

Therefore, in the running example, we can setup an external host named orion to perform backups in this way:

And then just:

This was for a full (cold) system backup. To perform a hot backup create an incremental backup with the query engine files:

Using flag -z in calls to lxbackup makes the tool gzip backup files, preserving modification times despite compression.

Using flag -C both encrypts and compresses the backup files. The flag argument is the path to the file keeping the secret used to encrypt/decrypt.

If any backup is encrypted, -C must be used also to restore, as in

Inspect or update the configuration:

By default, lx config prints the whole configuration, or that for elements given as arguments.

The arguments follow the conventional syntax used by most commands, but knows property names also:

For example,

prints all configuration.

prints the configuration for hosts atlantis and mariner (assuming those are configured host names).

prints the configuration for kvms components found at atlantis and lxqe components found at mariner

prints the configuration for any kvms component and the kvds101 one.

prints the addr attribute for any kvms component.

prints the awsdisk property from the (global) configuration.

Use flag -o to save the configuration (or the parts selected) to the given file.

Use flag -d to remove the selected configuration entries. Do not remove hosts or components, and use this with caution.

Use flag -s to update the selected properties with new values. In this case, the argument for a property includes both its name and the new value (e.g., mem=50m)

For example:

Disable transactions.

For example,

disables transactions for just the entire system, until the they are enabled.

While transactions are disabled, lx status shows the system status as waiting (for transactions to become available).

Before using this command, stop the watcher if the system was starting with flag -w or lx watch is running.

Otherwise, because new JDBC connections are not accepted while transactions are disabled, the watcher might consider that the system is not in good health, and then restart the system.

Enable transactions.

For example,

enables transactions for just the qe100 query engine.

Format the whole DB or the indicated hosts or components:

For example, format the kvds101 disk:

Ask for help:

Prints the list of known commands with quick usage information, or detailed usage information about the given command.

Checks the license status or installs a new license:

For example, ask for the current status:

or install a new file lxlicense with the desired license:

List and inspect logs for installed components:

For example, list the logs for kvds at atlantis, but only those after the time 230518.0551 (yymmdd.hhmm, or a prefix of this, can be used):

Print the last log for kvds101:

Grep all the logs for lines with fatal:

Copy all (not just the last one) kvds101 logs to /tmp:

Print the transaction log contents:

The command prints the log contents for the given log directory (or file). With flag -v, it prints the logged kv message, otherwise it prints just the first line of each logged message.

With flag -h (or -v) it prints the header information too.

It is suggested not to use this on a running log, just in case the program modifies the logger (although that should not happen).

List the processes for the DB:

For example, list the processes at host blade123:

Or to see the port ustage status:

Report system status and debug information.

This program collects information from the system and builds an archive to be sent to support.

As printed by the command output, the resulting tar file should be sent to support.

The archive includes:

When kvms is still running, the archive includes also:

Run a command on the selected installed hosts, using the lx environment to run it:

Arguments specify hosts where to run (as usual in the rest of commands), perhaps none (to imply all DB hosts), and then the keyword cmd must be given, followed by the command and arguments to run on each host.

For example, discover where this thing is installed:

Or, as an exceptional measure, kill all processes running:

Or, kill just those at blade123:

Or kill -9 any kvds at blade123:

Dump process stacks

The command understands the conventional syntax to select hosts and/or components. For example, to dump the stack of kvms components:

With flag -v, local variables are printed too.

For java processes, both the native stack and the java stacks are printed.

Starts the DB for operation:

For example, to start the kvds components at host atlantis and leave the rest of the installation alone:

With flag -w, start will run the watch service. This pings the DB to make sure it can answer queries, and, when that is not the case, try to stop and restart the system. See the Section 17.22 section for details.

By default, start will remove files on the installed directories with names suggesting they are core dumps. Flag -k prevents this from happening and can be used to keep core dump files for debugging.

Prints the status for the system or waits for a given status:

For example, to learn the status:

Or, to wait until the status is running:

The status can be any of:

Halts the DB or stops individual hosts or components:

For example, halt the DB:

Stop just the kvds servers:

If the watch service is running, make sure to stop it before stopping individual components. Otherwise it might decide to wait for lxmeta to stop and then restart the system.

Without arguments, stop will first stop the watch service and there is no extra caution needed.

Under flag -w, stop will not stop the watch process.

Flag -k may be used to kill components. Do not use this unless you know what you are doing. For example, to kill the lxqe100

Print the installed version:

The program reports the leanXcale version name, along with detailed version information for distribution packages installed:

Watch out the system and restart it if needed:

This program is started by lx start when the flag -w is given to it. It is usually a bad idea to execute this command explicitly.

It waits until the DB is running, doing nothing until that point. From that point on, if the DB ceases to be running, the program will wait for lxmeta to stop, and, then try to stop and start the whole system, and finally exit.

When restarting the system, the flag -w is used, to start a new watcher for the new system.

The implications are that a failure to start will not restart the system more times, and that stopping individual components requires to stop this program first, or it might take actions on its own.

This is a command for local use only:

Use lx run to run it at any/all of the installed hosts.

Without any flag, locates the DB process pids looking at the files reporting them, and then kills them.

With flag -a, locates any process in the system by process name (all the ones started by the DB use bin/…​ as a name), and all java processes starting with LX, and kills them.

A TERM signal is sent, and then a KILL signal after a few seconds.

Flag -9 can be used to send only a KILL signal.

If process/component names are given (eg, kvds, or lxqe100), only those processes are killed.

For example, send a kill signal to any kvds at blade123: