Administration Guide

Before installing, it is important to know that LeanXcale has a distributed architecture and it consists of several components:

There are other components used by the system, that are not relevant for the user and are not described here. For example, spread is a communication bus used by LeanXcale components.

Never install or run leanXcale as root.

To install, you need:

The install program itself requires a unix system with python 3.7 (or later).

Your sales representative will provide you with a valid license file. In case you have any issues with it, please send an email to sales@leanxcale.com.

The LeanXcale zip distribution can be found at:

There is a zip file per version. Download the latest version.

If needed, install the required software dependencies on your system:

Here, and in what follows, unix$ is the system prompt used in commands and examples.

Working on the directory where the distribution has been downloaded, unzip the LeanXcale zipped distribution:

Here, you type the command after the unix$ prompt, and the following lines are an example of the expected output.

Verify that the process completed successfully and there is a directory lxdist (with the distribution packages) and the lxinst program:

To install, run the lxinst program and, as an option, supply the name of the installation directory (/usr/local/leanxcale by default):

The command takes the zipped LeanXcale distribution from ./lxdist, and installs it at the target folder.

The detailed configuration file built by the install process is saved at lxinst.conf, as a reference.

The database has users. The user lxadmin is the administrator for the whole system and during the install you are asked to provide its password.

To complete the install, a license must be added, as explained in the next section.

As another example, this installs at the given partition /dev/sdc2, with both encryption and compression, mounting the installed system at /usr/local/leanxcale:

The final messages printed by the install program will remind you of how to list or remove the installed file systems.

The installation creates the lx command. Use it to execute commands to operate the installed system.

At the end of the installation, the install program prints suggestions for adjusting the PATH variable so that lx will be available as any other command. For example, as done with:

when the install directory is /usr/local/leanxcale.

The lx command can be found at the bin directory in the install directory. For example, at /usr/local/leanxcale/bin/lx when installing at /usr/local/leanxcale.

In what follows, we assume that lx can be found using the system PATH.

The first command used is usually the one to install a license file:

It is also possible to put the license file at ~/.lxlicense, and the install process will find it and install it on the target system(s).

To install at multiple hosts, you may indicate target host names and install paths in the command line, like in the following example:

or simply provide the target hostnames:

When using a configuration file, simply define multiple hosts:

and use such file to install:

To specify a number of components we can still use the command line, like in:

or

…​ # xample.conf host atlantis kvds x 2 host mariner kvds x 2 …​

For high availability (HA), it is possible to install using replicated components. As a convenience, replication is configured using mirror hosts where a host can be installed as a mirror of another to replicate its components. Either all hosts must have mirrors, or none of them.

To install using mirrors, for high availability, combine host names in the command line using “+”, like in:

or in

This configures mariner as a mirror for atlantis. Either all hosts must have mirrors, or none of them.

When using a configuration file, use the mirror property to specify that a host is a mirror for another. In our example:

…​ # xample.conf host atlantis kvds x 2 host mariner mirror atlantis …​

The mirror host should not specify anything other than its mirror property. It takes its configuration from the mirrored host.

When a preinstalled leanXcale docker image is available, disregard this section and proceed as described in the next one.

This section explains how to create containers from leanXcale images and how to install licenses and change the lxadmin password for them.

Provided the leanXcale image named lx:2, and the docker network lxnet, this command runs a leanXcale docker container:

In this command, the container name is lx1, the network used lxnext, and the image used lx:2. The port redirection -p…​ exports the SQL port to the underlying host.

It is important to know that:

The container name (lx1) can be used to issue commands. For example, this removes the container after stopping it:

The installed container includes the lx command. Use it to execute commands to operate the installed DB system.

It is possible to attach to the container and use the ''lx'' command as it can be done on a bare metal host install:

Here, we type docker attach lx1 on the host, and lx version on the docker container prompt.

Note that if you terminate the shell reached when attaching the docker container, it will stop. Usually, this is not desired.

It is possible to execute commands directly on the executed container. For example:

executes lx version on the container.

AWS installs creates a systemd system service for leanXcale. The service is installed as a disabled service.

This is meant to be used only for single host installs.

When multiple instances are used, instances must be started before using lx start to start the system, and lx stop must be used to stop the system before stopping the instances.

This section is here as a reference, and also because it might be useful on single-host installs.

For example:

In this case, and the rest of this section, we use commands running on the installed AWS instance.

To enable the service, we can:

To disable it again:

To start the service by hand:

And now we can see its status:

The DB status should now be as follows:

When more than a single instance is installed, the service must be enabled, disabled, started, and/or stopped on all instances involved.

Note that enabling the service makes the DB start when the instance starts, and stop before the instance stops.

This is done by each instance on its own. At start time, the instance runs

and at stop time, the instance runs

The last start and stop output is saved at /usr/local/leanxcale/log/start.log, for inspection.

It might be more convenient to leave the service disabled and, after starting the instances for the install, use lx start command to start the system.

The program lxaws is used to list or remove AWS installs, and to start and stop instances and find their status.

Given a region, without any tags, it lists the tags installed:

Given a tag, it lists the tag resources as found on AWS:

It is also possible to supply just the base tag without the domain, as in

With flag -e prints commands to set environment variables with resources found, as an aid to run other scripts.

When more than one tag is asked for, or more than one instance/volume is found, variable names are made unique adding a number to the name, for example:

With flag -d, it removes the resources for the tags given. In this case, tags must be given explicitly in the command line.

To list, open, and close ports exported by the AWS install to the rest of the world, use lxaws flags -p (print ports), -o (open ports), and -c (close ports).

In all cases, the first argument is the tag for the install. The tag can be just the AWS install tag name, without .aws.leanxcale.com.

For example, this command lists the open ports:

Here, the protocol and port (or port range) is printed for each set of open ports. The CIDR printed shows the IP address range that can access the ports, and is 0.0.0.0/0 when anyone can access them.

Before with each port range, the name for the open port range is printed. This name is set by the default install, and can be set when opening ports as shown next.

To open a port range, use -o and give as arguments the tag for the install, the protocol, first and last port in range, the CIDR (or any if open to everyone), and a name to identify why this port range is open (no spaces). For example:

The new port will be shown as open if we ask for ports:

As another example:

To close a port, use -c and give as arguments the tag for the install, the protocol, a port within the range of interest, and the CIDR used to export the port. Note that any can be used here too instead of the CIDR 0.0.0.0/0 For example:

Here, we used the -v flag (verbose) to see what is going on.

As another example, this can be used to close the open port range 8888-10000 from the example above:

To find out the instance statuses for a given AWS tag, use the -status flag for lxaws:

With just a tag name, it lists the instances for the given tag name.

Supplying extra arguments with a host name, IP, or instance ID, lists just the status for the given instances.

Flags -start and -stop in lxaws can be used to start/stop instances. Their use is similar to that of the -status flag. Given a tag, all instances are started/stopped. Given a tag and a host (or more hosts), matching instances are started/stopped.

For example:

Note that this does not start/stop the DB in the instances. The command just starts/stops the instances.

Once instances are started, connect to one of them and use lx start to start the DB.

before stopping, connect to one of the instances and use lx stop to stop the DB before stopping the instances.

Peering connections can be used to bridge two VPCs at AWS.

One peer asks the other peer to accept a peering connection, the peer accepts the connection, and network routes and security group rules for access are configured.

Peering connections are handled using lxaws with the peering connection flags. If you do not have lxaws, copy lxinst to a file named lxaws and give it execution permissions.

These are the flags for peering connections:

To request a peering connection, supply as arguments

For example:

Here, we could have used xample.aws.leanxcale.com instead. The command prints the peering connection identifier, to be used for setting up networking and asking the peer administrator to accept the peering request.

To accept a peer request, supply as an argument the peering connection id, as in:

In either case, once the dialed peer accepts the request, networking must be set supplying as arguments

For example:

Our local CIDR block is 10.0.120.0/24. This must be given to the peer, so the peer system can setup routing for this block to our network, along with the VPC id and our security group id.

This information can be retrieved using lxaws as described in the previous section. For example:

Should it be necessary, the CIDR block used by the install can be set when installing the system (but not later), using the property awsnet, as in

Note that only the network address bytes used are given, instead of using 10.0.120.0/24.

Once done with a peering connection, it can be dismantled supplying both the tag and the peering connection identifier. The identifier is always given because, when accepting a peering request, the peering connection does not belong to us. But, using the command above you can retrieve such identifier easily.

When peering is no longer desired, the peering connection can be removed.

Removing the peering connection also removes the routing entries for it and the security group access rules added when it was setup.

When using multiple hosts and replication, it is possible to start and stop individual hosts or replicas and force the system to run using just those.

For example,

stops the processes in replica-1.

To start it again, we can proceed in a similar way:

Stopping a replica while the system is running is strongly discouraged. Using it again requires restoring the replica state to make it work with the rest of the system.

In this example, if the whole system was running when lx stop repl1 was used, starting repl1 again will reintegrate it into the system if possible.

However, if we have a fully stopped system, and run

the system will run just the first replica. This will happen even if the second replica is not reachable and there is no way to ensure that the metadata in the first replica is up-to-date.

To ensure that the metadata is up-to-date in partial starts, use flag -c. This performs the same checks made when starting the whole system, and ensures that metadata is up to date, before attempting an start of the named replica or components.

The command lx status reports the status for the system or waits for a given status, as dsecribed for other installs in this document. For example,

To see the replication mirror status for the system, use flag -r

And, to see detailed information about components, use flag -p, perhaps in addition to -r:

In this example, all components are running with their mirror set as ok.

When some components failed, or part of the system was stopped, we can see a different output.

For example, after

We can see

The first thing to note here is that the replica is not ok, but single. This means we have single processes (without their mirrors) and the system is running in degraded mode.

Also, kvds100.r2 status with respect to replication is single. This means that it was running while its peer (kvds100.r1, in the first replica) was stopped.

This server will not be used again until it has been brought up to date with respect to the rest of the system.

Note how kvds100.r1 status with respect to replication is outdated. This means it passed away (failed or halted) while its peer was still in use.

The same happens to lxqe100.r2, but in this case both query engines could synchronize their mirrors after restarting lxqe100.r1 and nothing else was needed to permit the restarted process work with the rest of the system.

To inspect the status for a replicated system it is useful to look at DB metadata as stored on disk. On replicated systems the whole system is using a master metadata server (kvms), which synchronizes metadata with its mirror server.

Looking at the disk information may aid in diagnosing the state for the system when some replica is not running, or has been retired from service.

The dbmeta command can be used to do this. For example, after running

on a replicated system previously halted, we can see

The system has not been used, so the mirror status is still ok. The output for dbmeta returns what is known by the running kvms server:

We asked just for metadata of servers using the /srv resource path name.

The interesting part is that we can ask for metadata as known by both replicas:

It can be seen how replica-2 (that for kvms100.r2) is way out of date at least with respect to snapshots. This is not a surprise because it is stopped.

We can ask for the full metadata using

or for that for a particular table or index.

When there is a failure, the system continues to operate using the mirror processes that remain alive.

Here we describe example failures, and provide details about reparing specific failed components. Then we describe how to use lx restore to try to restore things in a more convenient way.

For example, if qe100.r2 fails (we killed it to make it so), this can be seen:

Further details are reported by flags -r (replication) and -p (process):

Component lxqe100.r1 is alive and running, and lxqe100.r2 is dead.

Using now the database produces a change in status:

This means that lxqe100.r1 is known to be single, i.e., it has been used while its mirror was dead or halted.

Also, lxqe100.r2 is known to be outdated, i.e., its mirror has been used while it was dead or halted.

The command lx recover inspects metadata and tries to restore the disk for failed components.

It can be used before starting the system, to let lx start start an already restored replicated system.

To make an example, with the example system running, we killed both kvds100.r1 and lxqe100.r2 and stopped the system after that. This is the resulting server metadata:

A dry run (flag -n) for lx recover shows this:

We can use lx copy and lx dbmeta -w to copy disks and clear flags, but it is more convenient to run this command without the dry run flag once we are decided to do so.

The recover command can take arguments to select what should be recovered, following the style of most other commands. For example:

After a recover, the system can be started normally.

The command lx is a Shell for running LeanXcale control programs. This simply fixes the environment for the installed host and runs the command given as an argument:

The command operates on the whole LeanXcale system, even when multiple hosts are used.

It is convenient to have lx in the PATH environment variable, as suggested in the install program output.

Command output usually includes information on a per-host basis reporting the progress of the used command.

Most commands follow the same conventions regarding options and arguments. We describe them here as a convenience.

Arguments specify what to operate (e.g., what to start, stop, etc.) may be empty to rely on the defaults (whole DB) or may specify a particular host and/or component name:

This may be repeated to specify different hosts and/or components.

The special host names db, repl, and repl2 may be used and stand for hosts without the nodb attribute, hosts for the first replica, and hosts for the second replica (hosts that are a mirror of other ones).

The start command starts LeanXcale:

Here, atlantis started a few processes and, once done, the start command checked out if the processes are indeed alive.

In case not all components can be started successfully, the whole LeanXcale system is halted by the start command.

By default not watcher or automatic restart is setup. Using flag -r asks start to start the system asking it to restart any QE that was running, failed, and was not restarted less than one minute ago.

using flag -w asks start to start lx watch. The watch tool will wait until the system becomes operational and, upon failures, try to restart the whole system.

To start a single host or component, use its name as an argument, like in:

Start does not wait for the system to be operational. To wait until the system is ready to handle SQL commands, the status command can be used with the -w (wait for status) flag, as in:

Without the -w flag, the command prints the current status, which can be stopped, failed, running, or waiting.

To start LeanXcale installed on Docker containers, you must start the containers holding the installed system components.

For example, consider the default docker install

The install process created a docker image named lx:2, installed for the docker host lx1, and the docker network lxnet.

To list the image we can

And, to list the networks

The created image is a single one for all containers. The name given when creating the container determines the host name used The install process specified the host names, and containers must be starte using the corresponding host name(s), so they know which leanXcale host they are for.

For example, to start the container for lx1:

In this command, the container name is lx1, the network used lxnext, and the image used lx:2. The port redirection -p…​ exports the SQL port to the underlying host.

Listing docker processes shows now the running container

It is important to know that:

The container name (lx1) can be used to issue commands. For example,

executes lx version on the lx1 container.

The status for the system can be seen in a similar way:

Note that the container will not start the DB if no valid license is found.

To start LeanXcale installed on AWS, you must start the AWS instances holding the installed system components.

This can be done by hand using the AWS console, or using the lxaws command.

For example, after installing using xample as an AWS tag, this command starts the instances:

Once instances are started, the lx command is available at any of them.

For example, provided the PEM file can be found at xample.pem, we can run this:

to see the installed version. Here xample1 is the DNS host name registered as the first host for the install AWS tag xample. In the same way, xample2 would be the name for the second host, and so on.

The command lx status reports the status for the system or waits for a given status. For example,

Or, to wait until the status is running:

To see the status for each one of the processes in the system, use lx procs. For example:

Before looking at the LeanXcale system status, it is important to look at the status of the docker containers running LeanXcale components.

When containers are running, the command lx status reports the status for the system or waits for a given status. For example,

executes lx status on the lx1 container. The status is reported for the whole system, and not just for that container.

To wait until the status is running:

To see the status for each one of the processes in the system, use lx procs. For example:

Before looking at the LeanXcale system status, it is important to look at the status of the AWS instances running LeanXcale components.

This can be done using the lxaws -status flag with the installed AWS tag name:

When instances are running, the command lx status reports the status for the system or waits for a given status. For example,

to see the system status.

To wait until the status is running:

To see the status for each one of the processes in the system, use lx procs. For example:

The stop command halts LeanXcale:

Stopping the LeanXcale containers should be done after stopping leanxcale. The reason is that docker might timeout the stop operation if the system is too busy updating the disk during the stop procedure.

To stop the database,

stops the components for the whole system (being lx1 an installed container).

We can double check this

Once this is done, we can stop the docker container.

We can also remove the container, but, note that doing this removes all data in the container as well.

To stop leanXcale on AWS, you must stop leanXcale before stopping the AWS instances running it. For example

This stops the system on all the instances it uses.

Then, the instances can be stopped. This can be done on the AWS console, or using the lxaws -stop flag with the installed AWS tag name:

To create an incremental backup use the flag -i of lxdisk backup or lxbackup.

The output reports which components got files backed up.

The incremental backup is encrypted if the total backup it refers to is encrypted too. No flag -e should be given.

The incremental backup can be performed while the system is running.

To list the backups known use the lxdisk list command, or lxbackups

Those with a + in their names are incremental backups.

Verbosity can be increased adding one or more -v flags:

It is possible to list a single backup by suppling its name and/or specific components using arguments as done with most lx commands:

or

To remove a backup, it suffices to remove its directory. For example:

Here we used the flag “-d” for lx to change to $LXDIR before executing the remove command, which makes it easy to name the directory used for the dump.

Or, from our external backup example host:

Beware that if you remove a backup, you should remove also those incremental backups that follow up to the next total backup.

To restore a backup, use the restore command.

Or perhaps

Do this while the system is stopped. By default, it selects the last backup made.

To restore a particular backup, supply its name:

This can be done also for incremental backups. When restoring an incremental backup, the restore takes data also from previous incremental backups and the corresponding total backup.

To restore only specific hosts or components, supply their names or types as done for other commands:

Or perhaps:

To restore a backup, it is usually desirable to format the disk for the involved components before using backup to restore their disks.

To automate system backups, use crontab(8) to run lx disk when backing up within a installed host, or to run lxdisk at an external host, on the desired times.

This is the install program:

Given arguments specify where to install the DB. They may be paths for directories in the local host, or host names, or host and directory names using the syntax host:dir.

Instead of arguments, a configuration file may be given using the -f flag. The configuration file has been described before in this document.

Flag -k may be used to specify the key to download the distribution.

Flag '-K' must be used when installing on AWS to supply the path (without file name extension) to the pem/pub key files.

By default, the distribution is retrieved from the leanXcale artifactory. However, it is possible to supply one or more times the -d flag giving it the name for a .tgz file, a directory, or a URL. The special argument leanxcale stands for the official repository for the distribution.

Distribution packages will be retrieved by looking at those distribution sources. By convention, the first source is usally the ./lxdist directory, used to keep the distribution when downloaded.

For example, this installs just what is downloaded at ./lxdist, without downloading anything else:

Once downloaded, files are not downloaded again. Flag ''-c'' cleans the ''./lxdist'' directory to force a download of everything. If a different directory is specified by the user, it is not cleaned, although packages not found will be still downloaded there. To force a download of particular packages, remove the packages desired from the ''./lxdist'' directory (or the directory specified in the arguments).

For example, this downloads a fresh copy of the distribution and installs it:

And this tries first the ../lxdist directory and then the standard leanxcale repository:

This is actually the default when no distribution source is specified. Also, if no directory source is specified as a first option, lxdist is used as a directory to download components (other than files found on the local host).

The database has users (different from UNIX users). The user lxadmin is the administrator for the whole system. During the install you will be asked to type a password for lxadmin. To specify the password without being asked, you can set the password in the LXPASS environment variable:

When the users file ./lxowner exists, that file is used as the file describing the lxadmin user and its secret, instead of creating one using the user supplied password for lxadmin.

Do not use a existing file unless you know what you are doing. The reason is that this file must define lxadmin as the first user and that such user must have access to all resources, as configured.

Flag -i makes lxinst ignore system limits, to install on places with reduced disk or memory and in a hurry.

By default, the install performs a large install, trying to use all the machine resources for the service. Flag -s selects a small instead, and flag -m selects a medium install.

The install size affects components and sizes not specified by the user in command line arguments or in the configuration file. In a small install, at most 4 kdvs are added per host, and component memory is limited to 1GiB. In a medium install, at most 8 kvds components are added, and component memory is left with default values.

Flag -u is used to update a previous install. Still WIP, do not use it for now.

Shell for running LeanXcale control programs. This simply fixes the environment for the installed host and runs the command given as an argument:

When flag -d is used, the current working directory is set to the install directory at the host before running the given command.

Most commands follow the same conventions regarding options and arguments. We describe them here for convenience:

Flag -l is used when executing the command locally. This is used by the lx command framework, and should not be used in general by the end user.

Flag -D enables verbose diagnostics

Arguments specify what to operate (e.g., what to start, stop, etc.) may be empty to rely on the defaults (whole DB) or may specify a particular host and/or component name:

This may be repeated to specify different hosts and/or components.

The special host names db, repl, and repl2 may be used and stand for hosts without the nodb attribute, hosts for the first replica, and hosts for the second replica (hosts that are a mirror of other ones).

Add the given file(s) to the lib directory of the installed hosts, to add or update a library or jar in the installation:

By default, backups are kept at $LXDIR/dump at the host used to issue backup commands. The convention is to use the first host to keep backups.

That is for internal backups (backups at a installed host).

For external backups, lxbackup (or lxdisk backup) is used at the host keeping the backup, and the backup top-level directory is given with flag -d or defaults to ./lxdump otherwise.

Directories under dump/ are named using the date as their name. When more than one backup is made on the same date, further digits are added to number succesive backups. A sorted list for the directory reports backups from oldest to newest.

For incremental backups, a final + is added to the directory name.

To create a full, cold, backup when leanXcale is not running:

The printed name is the name for the directory keeping the backup, as used when restoring it. In this case, 240809.

The files kept at the backup are compressed, and must uncompressed if copied by hand. The restore command takes care of this.

Using flag -e both encrypts and compresses the backup files. Flags belong to disk, note that backup is the argument given to disk.

The key used to encrypt the backed files is kept at $LXDIR/lib/lxkey.pem on the installed sites as set up by the installer.

To create an incremental backup use the flag -i.

The output reports which components got files backed up.

The incremental backup is encrypted if the total backup it refers to is encrypted too. No flag -e should be given.

The incremental backup can be performed while the system is running.

To create a backup while the system is running (known as an online backup), use flag -o for lx backup.

This is more expensive than doing a backup with the system down. The flag is required to prevent mistakes. If the system is down when an online backup is asked for, the backup is still made as a cold backup. That is, the flag says that it is Ok to backup while the system is online.

To list the backups known use the backups command:

For example:

Those with a + in their names are incremental backups.

Verbosity can be increased adding one or more -v flags.

It is possible to list a single backup by suppling its name and/or specific components using arguments as done with most lx commands. The name last refers to the last backup made. For example:

or

This command is actually lx disk list. It is named backups when used outside lx disk for clarity.

For external backups, lxdisk list can be used, or the lxdisk command copied to lxbackups and then used as an external version of lx backups.

The lx copy command can be used to copy disks from hosts, directories or components to other hosts, directories or components.

The special source or target repl can be used to copy to or from the replica of a component to restore a mirror from another.

For example

copies the files for kvds100 to /tmp/kvds100.

The same is achieved using

As another example:

copies the whole set of disks at replica-1 hosts into their mirrors.

The same thing can be achieved by copying one component at a time using

This command is like lx disk copy.

Create backups, restore them, recover replicas, and copy disk data.

This command is exactly the lx disk command, but packaged for use from external hosts. Like it happens with the internal command, it can be also used as lxbackup, lxrestore, etc., as a convenience, and adjusts its usage for the corresponding command.

The external host must have ssh access to the installed hosts.

See lx disk for more details an examples.

Using lxdisk is exactly like using lx disk with a few differences:

The configuration file used must be the one retrieved using lx config, and not the one written by the user to perform the install, because lx config reports addresses and details needed by the lxdisk command.

Once a backup has been created, the configuration used is saved along with the backup data and there is no need to use -f to supply it.

Encryption/decryption happens at the source/destination of data when creating/extracting backups. Therefore, there is no key kept at the host running lxdisk.

To prepare a host to use lxdisk, get the lxdisk command, the configuration, and the kvtar command and copy them to the host.

The lxdisk and kvtar commands can be found at the installed $LXDIR/bin directory on any installed host. If you are not sure regarding the $LXDIR value, use this command to find it:

Flag -d for lx makes it change to the $LXDIR directory before running the given command.

The detailed configuration file to be used is kept at $LXDIR/lib/lxinst.conf. The configuration can be retrieved using the lx config command. For example, this creates ./lxinst.conf with the detailed configuration:

As an example, we can setup an external host named orion to perform backups in this way:

And then just:

Or perhaps:

Create backups, restore them, recover replicas, and copy disk data.

This command handles disk data for components to backup them, restore them from a backup, recover replica disks, and copy disk files for hosts or components.

See lxdisk for a similar tool that can be used from external, non-installed, hosts to maintain and use backups from them.

Instead of using directly lx disk, it may be more convenient to use one of its variants: lx backup, lx restore, lx backups, lx recover, and lx copy.

They behave as lx disk when given the command of the same name (but for backups, which is lx disk list).

Refer to the sections on these commands for a description of the respective lx disk commands.

For example,

is like

Inspect or update the configuration:

By default, lx config prints the whole configuration, or that for elements given as arguments.

The arguments follow the conventional syntax used by most commands, but knows property names also:

For example,

prints all configuration.

prints the configuration for hosts atlantis and mariner (assuming those are configured host names).

prints the configuration for kvms components found at atlantis and lxqe components found at mariner

prints the configuration for any kvms component and the kvds101 one.

prints the addr attribute for any kvms component.

prints the awsdisk property from the (global) configuration.

Use flag -o to save the configuration (or the parts selected) to the given file.

Use flag -d to remove the selected configuration entries. Do not remove hosts or components, and use this with caution.

Use flag -s to update the selected properties with new values. In this case, the argument for a property includes both its name and the new value (e.g., mem=50m)

For example:

Issue control requests to lx components

This command issues control requests to running processes. Use with care and do not issue a control request unless you know what you are doing.

The comp argument selects the target for the control request or is a known command. It can be any of:

Different processes have different control requests. In general, ? can be used to learn the control requests supported.

For example,

This command calls kvcon to actually issue control requests, after adjusting a bit the target names as a convenience.

For example, adding no extra debug flags can be used to see current debug flags set:

For lxmeta (mm) and lxqe (qe) processes, lowercase debug flags correspond to log4j loggers and uppercase flags correspond to standard flags used everywhere in the system.

Repeating a flag increases the debug level. Popular debug flags are:

Popular debug flags for java are also:

Other usage examples follow:

List database data files:

The command lists DB data files as found on disk.

For example:

Each line in the output lists the machine and component where the file is kept before the file information.

Flag -v makes the program more verbose:

The information listed, and the table and index names, is taken from the on-disk files, as known by the data servers.

For example, db-APP-PAYMENTS5 refers to database db, schema APP, and table PAYMENTS5.

To ask for a particular table, use flag -t and give the table name as an argument in the format just described:

The region minimum and maximum values are listed also as known by the data servers (i.e., by kvds) and not by the user or the query engines.

It is possible to list only for particular hosts or components by supplying the usual names as arguments.

For example:

or

Print the transaction log contents:

The command prints the log contents for the given names. Names can be host and or component names to select the addressed components, or a log directory or a logger address.

By default the command prints the log header and the records in it.

Under flag -i, only the information in the log header is printed:

Under flag -r, only the records are printed.

For data records, only the first line of the recorded message is printed. Use flag -a to print the full record contents (eg, all tuples recorded).

Flags -s and/or -e may be used to focus the search on a timestamp interval, or, if flag -n is given, on a record number interval.

For example, print records with numbers from 10 to 14 for lxqe100:

Print DB metadata as found on disk.

This command prints the DB metadata as kept on disk.

For example, on a replicated install:

The command accept a kv metadata path to list, similar to kvcon list arguments. For example, to list server metadata as kept on disk:

Flag -w can be used to set or clear metadata server flags. For example:

sets the single flag for the named server, and

clears the flag.

After restoring a replica, the fail flag for the restored component is usually cleared, and the single flag for the component used as the restore source is cleared as well.

As another example, this lists the metadata information for a table:

Using flag -a reports all information:

Disable transactions.

For example,

disables transactions for just the entire system, until the they are enabled.

While transactions are disabled, lx status shows the system status as waiting (for transactions to become available).

Before using this command, stop the watcher if the system was starting with flag -w or lx watch is running.

Otherwise, because new JDBC connections are not accepted while transactions are disabled, the watcher might consider that the system is not in good health, and then restart the system.

Enable transactions.

For example,

enables transactions for just the qe100 query engine.

Format the whole DB or the indicated hosts or components:

For example, format the kvds101 disk:

Dump java process heaps and heap histograms.

The command understands the conventional syntax to select hosts and/or components. For example, save a base histogram to compare later and check out for leaks we can:

Later, we can compare with another histogram using

To print the current histogram (top 50 lines by default), run it without arguments:

Ask for help:

Prints the list of known commands with quick usage information, or detailed usage information about the given command.

Checks the license status or installs a new license:

For example, ask for the current status:

or install a new file lxlicense with the desired license:

List and inspect logs for installed components:

For example, list the logs for kvds at atlantis, but only those after the time 230518.0551 (yymmdd.hhmm, or a prefix of this, can be used):

Print the last log for kvds101:

Grep all the logs for lines with fatal:

Copy all (not just the last one) kvds101 logs to /tmp:

The lx move command can be used to move tables or regions from one data server to another while the system is down.

For example

moves the files for the named table to the given server. The table name uses the data server syntax, not the SQL syntax. This command is not meant for users.

This command is like lx disk move.

List the processes for the DB:

For example, list the processes at host blade123:

Or to see the port ustage status:

See also flag -p of lx status.

The command lx recover locates disks for failed replicas or replica components and restored them from their mirrors.

Arguments given restrict the repairs to them.

For example, after replica failures, we can run

To try to recover their disks before restarting.

Use flag -n for a dry run to see what would be repaired before running it, unless you are sure to recover failed replicas.

Report system status and debug information.

This program collects information from the system and builds an archive to be sent to support.

As printed by the command output, the resulting tar file should be sent to support.

The archive includes:

When kvms is still running, the archive includes also:

To restore a backup, use the restore command.

For example:

Do this while the system is stopped. By default, it selects the last backup made.

To restore a particular backup, supply its name:

This can be done also for incremental backups. When restoring an incremental backup, the restore takes data also from previous incremental backups and the corresponding total backup.

To restore only specific hosts or components, supply their names or types as done for other commands:

Or perhaps:

To restore a backup, it is usually desirable to format the disk for the involved components before using restore to restore their disks.

To remove a backup, it suffices to remove its directory. For example:

Here we used the flag “-d” for lx to change to $LXDIR before executing the remove command, which makes it easy to name the directory used for the dump.

When restoring encrypted backups, the disk command knows it has to decrypt, and it uses the key at $LXDIR/lib/lxkey.pem, the one used when making the backup.

If you did reinstall, update the key so it matches the backup key used.

For external backups, lxdisk restore can be used, or the lxdisk command copied to lxrestore and then used as an external version of lx restore. Al alternative is to use lxbackup with flag -r.

Run a command on the selected installed hosts, using the lx environment to run it:

Arguments specify hosts where to run (as usual in the rest of commands), perhaps none (to imply all DB hosts), and then the keyword cmd must be given, followed by the command and arguments to run on each host.

For example, discover where this thing is installed, by running pwd and using flag -d to set the current directory to the install directory on each host.

Or, as an exceptional measure, kill all processes running:

Or, kill just those at blade123:

Or kill -9 any kvds at blade123:

Dump process stacks

The command understands the conventional syntax to select hosts and/or components. For example, to dump the stack of kvms components:

With flag -v, local variables are printed too.

For java processes, both the native stack and the java stacks are printed.

Starts the DB for operation:

For example, to start the kvds components at host atlantis and leave the rest of the installation alone:

With flag -w, start will run the watch service. This pings the DB to make sure it can answer queries, and, when that is not the case, try to stop and restart the system. See the Section 20.32 section for details.

By default, start will remove files on the installed directories with names suggesting they are core dumps. Flag -k prevents this from happening and can be used to keep core dump files for debugging.

Under flag -r, start will ask any lxmeta process started to try to restart a failed QE that was running, unless it was restarted less than one minute ago.

Prints the status for the system or waits for a given status:

For example, to learn the status:

Or, to wait until the status is running:

The status can be any of:

It can be also running with failures when using replication and suffering failures.

Flag -r reports the replica status:

The replica status can be any of:

For example:

Flag -p reports the process status for each known component For example:

When using replicas, the preferred mirror process is listed first.

The mirror status for each process can be any of:

Halts the DB or stops individual hosts or components:

For example, halt the DB:

Stop just the kvds servers:

If the watch service is running, make sure to stop it before stopping individual components. Otherwise it might decide to wait for lxmeta to stop and then restart the system.

Without arguments, stop will first stop the watch service and there is no extra caution needed.

Under flag -w, stop will not stop the watch process.

Flag -k may be used to kill components. Do not use this unless you know what you are doing. For example, to kill the lxqe100

Print the installed version:

The program reports the leanXcale version name, along with detailed version information for distribution packages installed:

Watch out the system and restart it if needed:

This program is started by lx start when the flag -w is given to it. It is usually a bad idea to execute this command explicitly.

It waits until the DB is running, doing nothing until that point. From that point on, if the DB ceases to be running, the program will wait for lxmeta to stop, and, then try to stop and start the whole system, and finally exit.

When restarting the system, the flag -w is used, to start a new watcher for the new system.

The implications are that a failure to start will not restart the system more times, and that stopping individual components requires to stop this program first, or it might take actions on its own.

This is a command for local use only:

Use lx run to run it at any/all of the installed hosts.

Without any flag, locates the DB process pids looking at the files reporting them, and then kills them.

With flag -a, locates any process in the system by process name (all the ones started by the DB use bin/…​ as a name), and all java processes starting with LX, and kills them.

A TERM signal is sent, and then a KILL signal after a few seconds.

Flag -9 can be used to send only a KILL signal.

If process/component names are given (eg, kvds, or lxqe100), only those processes are killed.

For example, send a kill signal to any kvds at blade123: