Administration Reference Guide

This is the install program:

Given arguments specify where to install the DB. They may be paths for directories in the local host, or host names, or host and directory names using the syntax host:dir.

Instead of arguments, a configuration file may be given using the -f flag. The configuration file has been described before in this document.

Flag -k may be used to specify the key to download the distribution.

Flag '-K' must be used when installing on AWS to supply the path (without file name extension) to the pem/pub key files.

By default, the distribution is retrieved from the leanXcale artifactory. However, it is possible to supply one or more times the -d flag giving it the name for a .tgz file, a directory, or a URL. The special argument leanxcale stands for the official repository for the distribution.

Distribution packages will be retrieved by looking at those distribution sources. By convention, the first source is usally the ./lxdist directory, used to keep the distribution when downloaded.

For example, this installs just what is downloaded at ./lxdist, without downloading anything else:

Once downloaded, files are not downloaded again. Flag ''-c'' cleans the ''./lxdist'' directory to force a download of everything. If a different directory is specified by the user, it is not cleaned, although packages not found will be still downloaded there. To force a download of particular packages, remove the packages desired from the ''./lxdist'' directory (or the directory specified in the arguments).

For example, this downloads a fresh copy of the distribution and installs it:

And this tries first the ../lxdist directory and then the standard leanxcale repository:

This is actually the default when no distribution source is specified. Also, if no directory source is specified as a first option, lxdist is used as a directory to download components (other than files found on the local host).

The database has users (different from UNIX users). The user lxadmin is the administrator for the whole system. During the install you will be asked to type a password for lxadmin. To specify the password without being asked, you can set the password in the LXPASS environment variable:

When the users file ./lxowner exists, that file is used as the file describing the lxadmin user and its secret, instead of creating one using the user supplied password for lxadmin.

Do not use a existing file unless you know what you are doing. The reason is that this file must define lxadmin as the first user and that such user must have access to all resources, as configured.

Flag -i makes lxinst ignore system limits, to install on places with reduced disk or memory and in a hurry.

By default, the install performs a large install, trying to use all the machine resources for the service. Flag -s selects a small instead, and flag -m selects a medium install.

The install size affects components and sizes not specified by the user in command line arguments or in the configuration file. In a small install, at most 4 kdvs are added per host, and component memory is limited to 1GiB. In a medium install, at most 8 kvds components are added, and component memory is left with default values.

Flag -u is used to update a previous install. Still WIP, do not use it for now.

Shell for running LeanXcale control programs. This simply fixes the environment for the installed host and runs the command given as an argument:

When flag -d is used, the current working directory is set to the install directory at the host before running the given command.

Most commands follow the same conventions regarding options and arguments. We describe them here for convenience:

Flag -l is used when executing the command locally. This is used by the lx command framework, and should not be used in general by the end user.

Flag -D enables verbose diagnostics

Arguments specify what to operate (e.g., what to start, stop, etc.) may be empty to rely on the defaults (whole DB) or may specify a particular host and/or component name:

This may be repeated to specify different hosts and/or components.

The special host names db, repl, and repl2 may be used and stand for hosts without the nodb attribute, hosts for the first replica, and hosts for the second replica (hosts that are a mirror of other ones).

Add the given file(s) to the lib directory of the installed hosts, to add or update a library or jar in the installation:

By default, backups are kept at $LXDIR/dump at the host used to issue backup commands. The convention is to use the first host to keep backups.

That is for internal backups (backups at a installed host).

For external backups, lxbackup (or lxdisk backup) is used at the host keeping the backup, and the backup top-level directory is given with flag -d or defaults to ./lxdump otherwise.

Directories under dump/ are named using the date as their name. When more than one backup is made on the same date, further digits are added to number succesive backups. A sorted list for the directory reports backups from oldest to newest.

For incremental backups, a final + is added to the directory name.

To create a full, cold, backup when leanXcale is not running:

The printed name is the name for the directory keeping the backup, as used when restoring it. In this case, 240809.

The files kept at the backup are compressed, and must uncompressed if copied by hand. The restore command takes care of this.

Using flag -e both encrypts and compresses the backup files. Flags belong to disk, note that backup is the argument given to disk.

The key used to encrypt the backed files is kept at $LXDIR/lib/lxkey.pem on the installed sites as set up by the installer.

To create an incremental backup use the flag -i.

The output reports which components got files backed up.

The incremental backup is encrypted if the total backup it refers to is encrypted too. No flag -e should be given.

The incremental backup can be performed while the system is running.

To create a backup while the system is running (known as an online backup), use flag -o for lx backup.

This is more expensive than doing a backup with the system down. The flag is required to prevent mistakes. If the system is down when an online backup is asked for, the backup is still made as a cold backup. That is, the flag says that it is Ok to backup while the system is online.

To list the backups known use the backups command:

For example:

Those with a + in their names are incremental backups.

Verbosity can be increased adding one or more -v flags.

It is possible to list a single backup by suppling its name and/or specific components using arguments as done with most lx commands. The name last refers to the last backup made. For example:

or

This command is actually lx disk list. It is named backups when used outside lx disk for clarity.

For external backups, lxdisk list can be used, or the lxdisk command copied to lxbackups and then used as an external version of lx backups.

The lx copy command can be used to copy disks from hosts, directories or components to other hosts, directories or components.

The special source or target repl can be used to copy to or from the replica of a component to restore a mirror from another.

For example

copies the files for kvds100 to /tmp/kvds100.

The same is achieved using

As another example:

copies the whole set of disks at replica-1 hosts into their mirrors.

The same thing can be achieved by copying one component at a time using

This command is like lx disk copy.

Create backups, restore them, recover replicas, and copy disk data.

This command is exactly the lx disk command, but packaged for use from external hosts. Like it happens with the internal command, it can be also used as lxbackup, lxrestore, etc., as a convenience, and adjusts its usage for the corresponding command.

The external host must have ssh access to the installed hosts.

See lx disk for more details an examples.

Using lxdisk is exactly like using lx disk with a few differences:

The configuration file used must be the one retrieved using lx config, and not the one written by the user to perform the install, because lx config reports addresses and details needed by the lxdisk command.

Once a backup has been created, the configuration used is saved along with the backup data and there is no need to use -f to supply it.

Encryption/decryption happens at the source/destination of data when creating/extracting backups. Therefore, there is no key kept at the host running lxdisk.

To prepare a host to use lxdisk, get the lxdisk command, the configuration, and the kvtar command and copy them to the host.

The lxdisk and kvtar commands can be found at the installed $LXDIR/bin directory on any installed host. If you are not sure regarding the $LXDIR value, use this command to find it:

Flag -d for lx makes it change to the $LXDIR directory before running the given command.

The detailed configuration file to be used is kept at $LXDIR/lib/lxinst.conf. The configuration can be retrieved using the lx config command. For example, this creates ./lxinst.conf with the detailed configuration:

As an example, we can setup an external host named orion to perform backups in this way:

And then just:

Or perhaps:

Create backups, restore them, recover replicas, and copy disk data.

This command handles disk data for components to backup them, restore them from a backup, recover replica disks, and copy disk files for hosts or components.

See lxdisk for a similar tool that can be used from external, non-installed, hosts to maintain and use backups from them.

Instead of using directly lx disk, it may be more convenient to use one of its variants: lx backup, lx restore, lx backups, lx recover, and lx copy.

They behave as lx disk when given the command of the same name (but for backups, which is lx disk list).

Refer to the sections on these commands for a description of the respective lx disk commands.

For example,

is like

Inspect or update the configuration:

By default, lx config prints the whole configuration, or that for elements given as arguments.

The arguments follow the conventional syntax used by most commands, but knows property names also:

For example,

prints all configuration.

prints the configuration for hosts atlantis and mariner (assuming those are configured host names).

prints the configuration for kvms components found at atlantis and lxqe components found at mariner

prints the configuration for any kvms component and the kvds101 one.

prints the addr attribute for any kvms component.

prints the awsdisk property from the (global) configuration.

Use flag -o to save the configuration (or the parts selected) to the given file.

Use flag -d to remove the selected configuration entries. Do not remove hosts or components, and use this with caution.

Use flag -s to update the selected properties with new values. In this case, the argument for a property includes both its name and the new value (e.g., mem=50m)

For example:

Issue control requests to lx components

This command issues control requests to running processes. Use with care and do not issue a control request unless you know what you are doing.

The comp argument selects the target for the control request or is a known command. It can be any of:

Different processes have different control requests. In general, ? can be used to learn the control requests supported.

For example,

This command calls kvcon to actually issue control requests, after adjusting a bit the target names as a convenience.

For example, adding no extra debug flags can be used to see current debug flags set:

For lxmeta (mm) and lxqe (qe) processes, lowercase debug flags correspond to log4j loggers and uppercase flags correspond to standard flags used everywhere in the system.

Repeating a flag increases the debug level. Popular debug flags are:

Popular debug flags for java are also:

Other usage examples follow:

List database data files:

The command lists DB data files as found on disk.

For example:

Each line in the output lists the machine and component where the file is kept before the file information.

Flag -v makes the program more verbose:

The information listed, and the table and index names, is taken from the on-disk files, as known by the data servers.

For example, db-APP-PAYMENTS5 refers to database db, schema APP, and table PAYMENTS5.

To ask for a particular table, use flag -t and give the table name as an argument in the format just described:

The region minimum and maximum values are listed also as known by the data servers (i.e., by kvds) and not by the user or the query engines.

It is possible to list only for particular hosts or components by supplying the usual names as arguments.

For example:

or

Print the transaction log contents:

The command prints the log contents for the given names. Names can be host and or component names to select the addressed components, or a log directory or a logger address.

By default the command prints the log header and the records in it.

Under flag -i, only the information in the log header is printed:

Under flag -r, only the records are printed.

For data records, only the first line of the recorded message is printed. Use flag -a to print the full record contents (eg, all tuples recorded).

Flags -s and/or -e may be used to focus the search on a timestamp interval, or, if flag -n is given, on a record number interval.

For example, print records with numbers from 10 to 14 for lxqe100:

Print DB metadata as found on disk.

This command prints the DB metadata as kept on disk.

For example, on a replicated install:

The command accept a kv metadata path to list, similar to kvcon list arguments. For example, to list server metadata as kept on disk:

Flag -w can be used to set or clear metadata server flags. For example:

sets the single flag for the named server, and

clears the flag.

After restoring a replica, the fail flag for the restored component is usually cleared, and the single flag for the component used as the restore source is cleared as well.

As another example, this lists the metadata information for a table:

Using flag -a reports all information:

Disable transactions.

For example,

disables transactions for just the entire system, until the they are enabled.

While transactions are disabled, lx status shows the system status as waiting (for transactions to become available).

Before using this command, stop the watcher if the system was starting with flag -w or lx watch is running.

Otherwise, because new JDBC connections are not accepted while transactions are disabled, the watcher might consider that the system is not in good health, and then restart the system.

Enable transactions.

For example,

enables transactions for just the qe100 query engine.

Format the whole DB or the indicated hosts or components:

For example, format the kvds101 disk:

Dump java process heaps and heap histograms.

The command understands the conventional syntax to select hosts and/or components. For example, save a base histogram to compare later and check out for leaks we can:

Later, we can compare with another histogram using

To print the current histogram (top 50 lines by default), run it without arguments:

Ask for help:

Prints the list of known commands with quick usage information, or detailed usage information about the given command.

Checks the license status or installs a new license:

For example, ask for the current status:

or install a new file lxlicense with the desired license:

List and inspect logs for installed components:

For example, list the logs for kvds at atlantis, but only those after the time 230518.0551 (yymmdd.hhmm, or a prefix of this, can be used):

Print the last log for kvds101:

Grep all the logs for lines with fatal:

Copy all (not just the last one) kvds101 logs to /tmp:

The lx move command can be used to move tables or regions from one data server to another while the system is down.

For example

moves the files for the named table to the given server. The table name uses the data server syntax, not the SQL syntax. This command is not meant for users.

This command is like lx disk move.

List the processes for the DB:

For example, list the processes at host blade123:

Or to see the port ustage status:

See also flag -p of lx status.

The command lx recover locates disks for failed replicas or replica components and restored them from their mirrors.

Arguments given restrict the repairs to them.

For example, after replica failures, we can run

To try to recover their disks before restarting.

Use flag -n for a dry run to see what would be repaired before running it, unless you are sure to recover failed replicas.

Report system status and debug information.

This program collects information from the system and builds an archive to be sent to support.

As printed by the command output, the resulting tar file should be sent to support.

The archive includes:

When kvms is still running, the archive includes also:

To restore a backup, use the restore command.

For example:

Do this while the system is stopped. By default, it selects the last backup made.

To restore a particular backup, supply its name:

This can be done also for incremental backups. When restoring an incremental backup, the restore takes data also from previous incremental backups and the corresponding total backup.

To restore only specific hosts or components, supply their names or types as done for other commands:

Or perhaps:

To restore a backup, it is usually desirable to format the disk for the involved components before using restore to restore their disks.

To remove a backup, it suffices to remove its directory. For example:

Here we used the flag “-d” for lx to change to $LXDIR before executing the remove command, which makes it easy to name the directory used for the dump.

When restoring encrypted backups, the disk command knows it has to decrypt, and it uses the key at $LXDIR/lib/lxkey.pem, the one used when making the backup.

If you did reinstall, update the key so it matches the backup key used.

For external backups, lxdisk restore can be used, or the lxdisk command copied to lxrestore and then used as an external version of lx restore. Al alternative is to use lxbackup with flag -r.

Run a command on the selected installed hosts, using the lx environment to run it:

Arguments specify hosts where to run (as usual in the rest of commands), perhaps none (to imply all DB hosts), and then the keyword cmd must be given, followed by the command and arguments to run on each host.

For example, discover where this thing is installed, by running pwd and using flag -d to set the current directory to the install directory on each host.

Or, as an exceptional measure, kill all processes running:

Or, kill just those at blade123:

Or kill -9 any kvds at blade123:

Dump process stacks

The command understands the conventional syntax to select hosts and/or components. For example, to dump the stack of kvms components:

With flag -v, local variables are printed too.

For java processes, both the native stack and the java stacks are printed.

Starts the DB for operation:

For example, to start the kvds components at host atlantis and leave the rest of the installation alone:

With flag -w, start will run the watch service. This pings the DB to make sure it can answer queries, and, when that is not the case, try to stop and restart the system. See the Section 1.32 section for details.

By default, start will remove files on the installed directories with names suggesting they are core dumps. Flag -k prevents this from happening and can be used to keep core dump files for debugging.

Under flag -r, start will ask any lxmeta process started to try to restart a failed QE that was running, unless it was restarted less than one minute ago.

Prints the status for the system or waits for a given status:

For example, to learn the status:

Or, to wait until the status is running:

The status can be any of:

It can be also running with failures when using replication and suffering failures.

Flag -r reports the replica status:

The replica status can be any of:

For example:

Flag -p reports the process status for each known component For example:

When using replicas, the preferred mirror process is listed first.

The mirror status for each process can be any of:

Halts the DB or stops individual hosts or components:

For example, halt the DB:

Stop just the kvds servers:

If the watch service is running, make sure to stop it before stopping individual components. Otherwise it might decide to wait for lxmeta to stop and then restart the system.

Without arguments, stop will first stop the watch service and there is no extra caution needed.

Under flag -w, stop will not stop the watch process.

Flag -k may be used to kill components. Do not use this unless you know what you are doing. For example, to kill the lxqe100

Print the installed version:

The program reports the leanXcale version name, along with detailed version information for distribution packages installed:

Watch out the system and restart it if needed:

This program is started by lx start when the flag -w is given to it. It is usually a bad idea to execute this command explicitly.

It waits until the DB is running, doing nothing until that point. From that point on, if the DB ceases to be running, the program will wait for lxmeta to stop, and, then try to stop and start the whole system, and finally exit.

When restarting the system, the flag -w is used, to start a new watcher for the new system.

The implications are that a failure to start will not restart the system more times, and that stopping individual components requires to stop this program first, or it might take actions on its own.

This is a command for local use only:

Use lx run to run it at any/all of the installed hosts.

Without any flag, locates the DB process pids looking at the files reporting them, and then kills them.

With flag -a, locates any process in the system by process name (all the ones started by the DB use bin/…​ as a name), and all java processes starting with LX, and kills them.

A TERM signal is sent, and then a KILL signal after a few seconds.

Flag -9 can be used to send only a KILL signal.

If process/component names are given (eg, kvds, or lxqe100), only those processes are killed.

For example, send a kill signal to any kvds at blade123:

Properties with upper-case names are exported as environment variables for the process(es) executing the component involved. This includes both global properties and host properties as well. For example, to enable debug flags DOM for the kv datastore:

These are other properties not described before.