LeanXcale v2.4 User’s Guide

Before installing, it is important to know that LeanXcale has a distributed architecture and it consists of several components:

There are other components used by the system, that are not relevant for the user and are not described here. For example, spread is a communication bus used by LeanXcale components.

The command lx is a Shell for running LeanXcale control programs. This simply fixes the environment for the installed host and runs the command given as an argument:

The command operates on the whole LeanXcale system, even when multiple hosts are used.

It is convenient to have lx in the PATH environment variable, as suggested in the install program output.

Command output usually includes information on a per-host basis reporting the progress of the used command.

Most commands follow the same conventions regarding options and arguments. We describe them here as a convenience.

Arguments specify what to operate (e.g., what to start, stop, etc.) may be empty to rely on the defaults (whole DB) or may specify a particular host and/or component name:

This may be repeated to specify different hosts and/or components.

The special host names db, repl, and repl2 may be used and stand for hosts without the nodb attribute, hosts for the first replica, and hosts for the second replica (hosts that are a mirror of other ones).

The start command starts LeanXcale:

Here, atlantis started a few processes and, once done, the start command checked out if the processes are indeed alive.

In case not all components can be started successfully, the whole LeanXcale system is halted by the start command.

By default not watcher or automatic restart is setup. Using flag -r asks start to start the system asking it to restart any QE that was running, failed, and was not restarted less than one minute ago.

using flag -w asks start to start lx watch. The watch tool will wait until the system becomes operational and, upon failures, try to restart the whole system.

To start a single host or component, use its name as an argument, like in:

Start does not wait for the system to be operational. To wait until the system is ready to handle SQL commands, the status command can be used with the -w (wait for status) flag, as in:

Without the -w flag, the command prints the current status, which can be stopped, failed, running, or waiting.

To start LeanXcale installed on Docker containers, you must start the containers holding the installed system components.

For example, consider the default docker install

The install process created a docker image named lx:2, installed for the docker host lx1, and the docker network lxnet.

To list the image we can

And, to list the networks

The created image is a single one for all containers. The name given when creating the container determines the host name used The install process specified the host names, and containers must be starte using the corresponding host name(s), so they know which leanXcale host they are for.

For example, to start the container for lx1:

In this command, the container name is lx1, the network used lxnext, and the image used lx:2. The port redirection -p…​ exports the SQL port to the underlying host.

Listing docker processes shows now the running container

It is important to know that:

The container name (lx1) can be used to issue commands. For example,

executes lx version on the lx1 container.

The status for the system can be seen in a similar way:

Note that the container will not start the DB if no valid license is found.

To start LeanXcale installed on AWS, you must start the AWS instances holding the installed system components.

This can be done by hand using the AWS console, or using the lxaws command.

For example, after installing using xample as an AWS tag, this command starts the instances:

Once instances are started, the lx command is available at any of them.

For example, provided the PEM file can be found at xample.pem, we can run this:

to see the installed version. Here xample1 is the DNS host name registered as the first host for the install AWS tag xample. In the same way, xample2 would be the name for the second host, and so on.

The command lx status reports the status for the system or waits for a given status. For example,

Or, to wait until the status is running:

To see the status for each one of the processes in the system, use lx procs. For example:

Before looking at the LeanXcale system status, it is important to look at the status of the docker containers running LeanXcale components.

When containers are running, the command lx status reports the status for the system or waits for a given status. For example,

executes lx status on the lx1 container. The status is reported for the whole system, and not just for that container.

To wait until the status is running:

To see the status for each one of the processes in the system, use lx procs. For example:

Before looking at the LeanXcale system status, it is important to look at the status of the AWS instances running LeanXcale components.

This can be done using the lxaws -status flag with the installed AWS tag name:

When instances are running, the command lx status reports the status for the system or waits for a given status. For example,

to see the system status.

To wait until the status is running:

To see the status for each one of the processes in the system, use lx procs. For example:

The stop command halts LeanXcale:

Stopping the LeanXcale containers should be done after stopping leanxcale. The reason is that docker might timeout the stop operation if the system is too busy updating the disk during the stop procedure.

To stop the database,

stops the components for the whole system (being lx1 an installed container).

We can double check this

Once this is done, we can stop the docker container.

We can also remove the container, but, note that doing this removes all data in the container as well.

To stop leanXcale on AWS, you must stop leanXcale before stopping the AWS instances running it. For example

This stops the system on all the instances it uses.

Then, the instances can be stopped. This can be done on the AWS console, or using the lxaws -stop flag with the installed AWS tag name:

When using multiple hosts and replication, it is possible to start and stop individual hosts or replicas and force the system to run using just those.

For example,

stops the processes in replica-1.

To start it again, we can proceed in a similar way:

Stopping a replica while the system is running is strongly discouraged. Using it again requires restoring the replica state to make it work with the rest of the system.

In this example, if the whole system was running when lx stop repl1 was used, starting repl1 again will reintegrate it into the system if possible.

However, if we have a fully stopped system, and run

the system will run just the first replica. This will happen even if the second replica is not reachable and there is no way to ensure that the metadata in the first replica is up-to-date.

To ensure that the metadata is up-to-date in partial starts, use flag -c. This performs the same checks made when starting the whole system, and ensures that metadata is up to date, before attempting an start of the named replica or components.

The command lx status reports the status for the system or waits for a given status, as dsecribed for other installs in this document. For example,

To see the replication mirror status for the system, use flag -r

And, to see detailed information about components, use flag -p, perhaps in addition to -r:

In this example, all components are running with their mirror set as ok.

When some components failed, or part of the system was stopped, we can see a different output.

For example, after

We can see

The first thing to note here is that the replica is not ok, but single. This means we have single processes (without their mirrors) and the system is running in degraded mode.

Also, kvds100.r2 status with respect to replication is single. This means that it was running while its peer (kvds100.r1, in the first replica) was stopped.

This server will not be used again until it has been brought up to date with respect to the rest of the system.

Note how kvds100.r1 status with respect to replication is outdated. This means it passed away (failed or halted) while its peer was still in use.

The same happens to lxqe100.r2, but in this case both query engines could synchronize their mirrors after restarting lxqe100.r1 and nothing else was needed to permit the restarted process work with the rest of the system.

To inspect the status for a replicated system it is useful to look at DB metadata as stored on disk. On replicated systems the whole system is using a master metadata server (kvms), which synchronizes metadata with its mirror server.

Looking at the disk information may aid in diagnosing the state for the system when some replica is not running, or has been retired from service.

The dbmeta command can be used to do this. For example, after running

on a replicated system previously halted, we can see

The system has not been used, so the mirror status is still ok. The output for dbmeta returns what is known by the running kvms server:

We asked just for metadata of servers using the /srv resource path name.

The interesting part is that we can ask for metadata as known by both replicas:

It can be seen how replica-2 (that for kvms100.r2) is way out of date at least with respect to snapshots. This is not a surprise because it is stopped.

We can ask for the full metadata using

or for that for a particular table or index.

When there is a failure, the system continues to operate using the mirror processes that remain alive.

Here we describe example failures, and provide details about reparing specific failed components. Then we describe how to use lx restore to try to restore things in a more convenient way.

For example, if qe100.r2 fails (we killed it to make it so), this can be seen:

Further details are reported by flags -r (replication) and -p (process):

Component lxqe100.r1 is alive and running, and lxqe100.r2 is dead.

Using now the database produces a change in status:

This means that lxqe100.r1 is known to be single, i.e., it has been used while its mirror was dead or halted.

Also, lxqe100.r2 is known to be outdated, i.e., its mirror has been used while it was dead or halted.

Recovering from lxmeta failures is trivial because the component simply rebuilds its state from the running system.

Recovering from kvms failures requires making sure that when the system starts, the new master has the most recent metadata on disk.

kvms servers kept the metadata synchronized and there is nothing special to be done to recover them from a failure as long as they can reach the current master server or the disk data for the new master is up to date.

For example, if the master kvms dies, we see this as the status:

The system continues to operate using kvms100.r2.

To recover at this point, it suffices to restart the failed kvms:

It takes its state from kvms100.r2, which is the current master, and the system is ok.

However, if the system stops before updating kvms100.r1 disk with the, possibly newer metadata, it can be that the next restart it will be the new master and also have old data, leading to problems.

When used to start the whole system, lx start takes care of updating the metadata on disk for kvms components with that from the previous master (the one with newest timestamps in it).

When starting components by hand or by individual hosts, the kvms data should be updated on disk for the replicas with the data from the newest one.

That is, unless the server keeping the previous kvms master is unreachable, running

suffices to use up to date metadata despite previous kvms failures.

But, running explicitly kvms100.r1 after its failure, without updating its disk metadata, with the rest of the system stopped or unreachable, will lead to an execution with old system metadata, leading to problems.

This is only important when there are failures or explicit stops for parts of the system. Otherwise, all metadata is synchronized and there should be no problem.

Doing a system stop, and a system start, should recover failed query engines without doing anything else. But this is not always possible or desirable.

In the case of query engine failures, we can recover them while the system is running by starting the failed query engine.

Asking for the status for replica and processes yields:

If the system has been running for too long while a query engine was stopped, it may be better to recover the failed query engine disk state from the mirror alive.

This command recreates the disk for the replica of lxqe100.r2 from the disk of lxqe100.r1. The argument order is similar to cp, i.e., source and then destination.

Here, it is important to use repl as the (source or) target argument name, to let copy know that it is updating a replica.

Use lx copy with care. It blindly copies the disk from one component to another, and you might overwrite data in the process.

Flag -n makes the command report what it would do, and it is sensible to use it before actually copying anything. More details are printed using flag -v:

Here, log is the log data and logm is the mirror kept for the mirror QE, they are exchanged after the copy to make lxqe100.r2 a replica for lxqe100.r1.

In the case of a kvds failure, bringing it back into operation requires reparing it by updating its data with that from its mirror.

For example, using our example install and killing kvds100.r2 leads to this status:

Here, kvds100.r1 continued to run (has a single status) and kvds100.r2 failed while its mirror was running (has an outdated status). The whole system replication status is now single, because of this.

Starting kvds100.r2 by hand does not change things. Its state is probably out of date with respect to its mirror, and it is not integrated into the running system until restored.

Stopping the system still preserves the status for the single and failed server:

Restarting the system at this point will force a recovery from the DB log data, because there was a single kvds and its mirror must be updated before it can run user operations.

When the kvds server has been down for too long, it is better to restore its disk from its surviving mirror before restarting the system.

Consider the system after kvds100.r2 failed, and it was stopped using lx stop. Instead of using lx start directly, we can use lx copy to restore the obsolete and failed kvds100.r2:

Copying a kvds disk can take a long time.

The command lx recover inspects metadata and tries to restore the disk for failed components.

It can be used before starting the system, to let lx start start an already restored replicated system.

To make an example, with the example system running, we killed both kvds100.r1 and lxqe100.r2 and stopped the system after that. This is the resulting server metadata:

A dry run (flag -n) for lx recover shows this:

We can use lx copy and lx dbmeta -w to copy disks and clear flags, but it is more convenient to run this command without the dry run flag once we are decided to do so.

The recover command can take arguments to select what should be recovered, following the style of most other commands. For example:

After a recover, the system can be started normally.

To use dbeaver with leanXcale, just use the standard JDBC driver and configure the URI to access the installed system.

For example, download the JDBC for this version from Mvn central leanXcale drivers. Usually, you want the last driver. As of now, it is the leanXcaleJDBC driver 3.2. Then follow these steps:

At this point you can add the connection using the DB type just configured and the user and password as needed.

The lxqe log files include authentication information that can be checked out when in doubt.

Lines like

report that the user lxadmin (or whoever it was) was authenticated by the local (i.e., the DB) user entry.

When using LDAP, the line would be

This reports both the database used (db) and the user involved (USR1).

Authentication errors are reported in a similar way:

We folded the line to make it easier to read.

The audit log file contains auditing entries, including user authentication.

These can be enabled using the AUDIT SQL statement, although authentication auditing is always enabled and cannot be disabled.

For example, to locate authentication entries for the user lxadmin the flag -g (for grep) can be used as shown here

The user names include both the database and the user name, as a convenience. An exception to this rule is lxadmin.

Authentication failures are reported with lines like

If user or table auditing are enabled on a per-session basis, read, write, and change (alter) accesses are reported once per session. For example, after executing any of these:

the audit log file will include reports like:

More expensive auditing reports the statements executed (even if they fail), and can be enabled using, for example

Although it is more sensible to audit just READ, WRITE, or DDL statements instead of auditing all of them.

In this case, the audit log will contain lines like the following, and it may grow quickly.

When permission is denied to execute a statement, an audit record is added if any audit has been enabled.

The line has been folded for readability, although it is a single line in the audit log.

It is important to know that the lxqe log file reports failed permission checks even when auditing is disabled. There is no need to enable auditing just to search for failed permission checks.

To create an incremental backup use the flag -i of lxbackup (or the same flag with lx backup for an internal backup).

The output reports which components got files backed up.

The incremental backup is encrypted if the total backup it refers to is encrypted too. No flag -e should be given.

The incremental backup can be performed while the system is running.

To list the backups known use lxbackups (or lx backups for internal backups).

Those with a + in their names are incremental backups.

Verbosity can be increased adding one or more -v flags:

It is possible to list a single backup by suppling its name and/or specific components using arguments as done with most lx commands:

or

To remove a backup, it suffices to remove its directory. For example:

Here we used the flag “-d” for lx to change to $LXDIR before executing the remove command, which makes it easy to name the directory used for the dump.

Or, from our external backup example host:

Beware that if you remove a backup, you should remove also those incremental backups that follow, up to the next total backup.

To restore a backup, use the lxrestore command (or lx restore for internal backups).

Do this while the system is stopped. By default, it selects the last backup made.

To restore a particular backup, supply its name:

This can be done also for incremental backups. When restoring an incremental backup, the restore takes data also from previous incremental backups and from the previous total backup.

To restore only specific hosts or components, supply their names or types as done for other commands:

Or perhaps:

To restore a backup, it is usually desirable to format the disk for the involved components before using restore to restore their disks.

To automate system backups, use crontab(8) to run lx backup when backing up within a installed host, or to run lxbackup at an external host, on the desired times.