Administration Guide
1. Background
1.1. LeanXcale Components
Before installing, it is important to know that LeanXcale has a distributed architecture and it consists of several components:
-
lxqe: Query engine in charge of processing SQL queries.
-
kvds: Data server of the storage subsystem. There might be multiple instances.
-
kvms: Metadata server of the storage subsystem.
-
lxmeta: Metadata process for LeanXcale. It keeps metadata and services needed for other components.
-
stats: Optional monitoring subsystem to see resource usage and performance KPIs of LeanXcale database.
-
odata: Optional OpenDATA server to support a SQL REST API.
There are other components used by the system, that
are not relevant for the user and are not described here. For example,
spread
is a communication bus used by LeanXcale components.
2. Licenses
To check for license status or to install a new license you can use the lx license
command.
For a local installation, use just
unix$ lx license license expires: Mon Dec 30 00:00:00 2024
For docker installs, each container must include its own license. The DB in the container does not start the DB unless a valid license is found. But, the container must be running to check the license status and to install new licenses. Refer to the section on starting docker containers for help on that.
For example, to list the license status for the container lx1
we can run
unix$ docker exec -it lx1 lx license lx1 [ kvcon[1380]: license: no license file failed: failed: status 1 ] failed: status 1
To install a new license, just copy the license file to the container as shown here
unix$ docker cp ~/.lxlicense lx1:/usr/local/leanxcale/.lxlicense unix$ docker exec -it lx1 sudo chown lx /usr/local/leanxcale/.lxlicense
The license status should be ok now:
unix$ docker exec -it lx1 lx license license expires: Mon Dec 30 00:00:00 2024
3. Bare Metal and IaaS Installs
In here we describe how to install in bare metal and IaaS instances. Note that an IaaS (or virtualized) instance is equivalent to a bare metal one. IaaS that have been tried include AWS, Google, Azure and OCI.
3.1. Prerequisites
Never install or run leanXcale as root.
To install, you need:
-
2 cores & 2 GB of memory
-
A valid LeanXcale license file. Your sales representative will provide you with a valid license file. In case you have any issues with it, please send an email to sales@leanxcale.com.
-
The LeanXcale zip distribution. The LeanXcale zip distribution can be found at:
https://artifactory.leanxcale.com/artifactory/lxpublic
-
Ubuntu 20.04 LTS or 22.04 LTS with a standard installation, including
-
openjdk-11-jdk
-
python3
-
python3-dev
-
python3-pip
-
gdb
-
ssh tools as included in most UNIX distributions.
-
zfsutils-linux
-
libpam
-
-
Availability for ports used by leanXcale (refer to the [LeanXcale Components and Ports] section for details).
The install program itself requires a unix system with python 3.7 (or later).
Your sales representative will provide you with a valid license file. In case you have any issues with it, please send an email to sales@leanxcale.com.
The LeanXcale zip distribution can be found at:
https://artifactory.leanxcale.com/artifactory/lxpublic
There is a zip file per version. Download the latest version.
If needed, install the required software dependencies on your system:
unix$ sudo apt update unix$ sudo apt -y install python3 python3-dev python3-pip gdb openjdk-11-jdk curl
Here, and in what follows, unix$
is the system prompt used in
commands and examples.
3.2. Install
Working on the directory where the distribution has been downloaded, unzip the LeanXcale zipped distribution:
unix$ unzip lx.2.3.232129.zip Archive: lx.2.3.232129.zip creating: lxdist/ inflating: lxdist/lxinst.v2.3.port.tgz inflating: lxdist/lxavatica.v2.3.libs.tgz ...
Here, you type the command after the unix$
prompt, and the following
lines are an example of the expected output.
Verify that the process completed successfully and there is a directory
lxdist
(with the distribution packages) and the lxinst
program:
unix$ ls lx.2.3.232129.zip lxdist lxinst
To install, run the lxinst
program and, as an option, supply the
name of the installation directory (/usr/local/leanxcale
by
default):
unix$ ./lxinst /usr/local/leanxcale sysinfo localhost... install #cfgfile: argv /usr/local/leanxcale... cfgcomps… ... config: lxinst.conf install done. To run lx commands: localhost:/usr/local/leanxcale/bin/lx Or adjust your path: # At localhost:~/.profile: export PATH=/usr/local/leanxcale/bin:$PATH To start (with lx in your path): lx start
The command takes the zipped LeanXcale distribution from ./lxdist
, and
installs it at the target folder.
The detailed configuration file built by the install process is
saved at lxinst.conf
, as a reference.
The database has users. The user lxadmin
is
the administrator for the whole system and during the install you are asked
to provide its password.
To complete the install, a license must be added, as explained in the next section.
As another example, this installs at the given partition /dev/sdc2
, with both
encryption and compression, mounting the installed system at /usr/local/leanxcale
:
unix$ ./lxinst compress crypt /dev/sdc2:/usr/local/leanxcale sysinfo localhost... install #cfgfile: argv /usr/local/leanxcale... cfgcomps… ... config: lxinst.conf install done. To list installed file systems (on each host): zfs list -r lxpool -o name,encryption,compression To remove installed file systems (on each host): sudo zpool destroy lxpool To run lx commands: localhost:/usr/local/leanxcale/bin/lx Or adjust your path: # At localhost:~/.profile: export PATH=/usr/local/leanxcale/bin:$PATH To start (with lx in your path): lx start
The final messages printed by the install program will remind you of how to list or remove the installed file systems.
3.3. Using leanXcale Commands
The installation creates the lx
command. Use it to execute commands
to operate the installed system.
At the end of the installation, the install program prints suggestions
for adjusting the PATH
variable so that lx
will be available as any
other command. For example, as done with:
export PATH=/usr/local/leanxcale/bin:$PATH
when the install directory is /usr/local/leanxcale.
The lx
command can be found at the bin
directory in the install
directory. For example, at /usr/local/leanxcale/bin/lx
when
installing at /usr/local/leanxcale
.
In what follows, we assume that lx
can be found using the system PATH
.
The first command used is usually the one to install a license file:
unix$ lx license -f lxlicense
It is also possible to put the license file at ~/.lxlicense
, and the
install process will find it and install it on the target system(s).
3.4. Multiple Hosts and Replicated Installs
To install at multiple hosts, you may indicate target host names and install paths in the command line, like in the following example:
unix$ lxinst mariner:/opt/leanxcale atlantis:/usr/local/leanxcale
or simply provide the target hostnames:
unix$ lxinst mariner atlantis
When using a configuration file, simply define multiple hosts:
# xample.conf host mariner host atlantis
and use such file to install:
unix$ lxinst -f xample.conf
To specify a number of components we can still use the command line, like in:
unix$ lxinst atlantis kvds x 2 mariner kvds x 2
or
… # xample.conf host atlantis kvds x 2 host mariner kvds x 2 …
For high availability (HA), it is possible to install using replicated components. As a convenience, replication is configured using mirror hosts where a host can be installed as a mirror of another to replicate its components. Either all hosts must have mirrors, or none of them.
To install using mirrors, for high availability, combine host names in the command line using “+”, like in:
unix$ lxinst atlantis+mariner:/usr/local/leanxcale
or in
unix$ lxinst atlantis+mariner
This configures mariner
as a mirror for atlantis
. Either all
hosts must have mirrors, or none of them.
When using a configuration file, use the mirror
property to specify
that a host is a mirror for another.
In our example:
… # xample.conf host atlantis kvds x 2 host mariner mirror atlantis …
The mirror host should not specify anything other than its mirror property. It takes its configuration from the mirrored host.
4. Docker Installs
4.1. Docker Installs with no Preinstalled Image
When a preinstalled leanXcale docker image is available, disregard this section and proceed as described in the next one.
4.1.1. Prerequisites
To install, you need:
-
A valid LeanXcale license file
-
The LeanXcale zip distribution
-
A Linux system with
-
docker version 20.10.6 or later.
-
python 3.7 or later.
-
-
Access to the internet to permit docker to download standard docker images and software packages for them.
Your sales representative will provide you with a valid license file. In case you have any issues with it, please send an email to sales@leanxcale.com.
Download the latest LeanXcale zip distribution from:
https://artifactory.leanxcale.com/artifactory/lxpublic
There is a zip file per version.
4.1.2. Install
Working on the directory where the distribution has been downloaded, unzip the LeanXcale zipped distribution:
unix$ unzip lx.2.3.232129.zip Archive: lx.2.3.232129.zip creating: lxdist/ inflating: lxdist/lxinst.v2.3.port.tgz inflating: lxdist/lxavatica.v2.3.libs.tgz ...
Verify that the process completed successfully and there is a directory
lxdist
(with the distribution packages) and the lxinst
program:
unix$ ls lx.2.3.232129.zip lxdist lxinst
To create a docker image for leanXcale, use the following command:
unix$ lxinst docker sysinfo lx1... install #cfgfile: argv docker... ... install done docker images: REPOSITORY TAG IMAGE ID CREATED SIZE uxbase 2 434dfeaedf0c 3 weeks ago 1.06GB lx 2 6875de4f2531 4 seconds ago 1.33GB docker network: NETWORK ID NAME DRIVER SCOPE 471c52155823 lxnet bridge local to start: docker run -dit --name lx1 --network lxnet lx:2 lx1
The image created is named lx:2
. It is configured to run a container
with hostname lx1
on a docker network named lxnet
.
To list the image we can execute
unix$ docker images lx REPOSITORY TAG IMAGE ID CREATED SIZE lx 2 6875de4f2531 About a minute ago 1.33GB
And, to list the networks we can execute
unix$ docker network ls NETWORK ID NAME DRIVER SCOPE 471c52155823 lxnet bridge local
The created image is a single one for all containers. The name given
when creating the container determines the host name used (in this
example, lx1
).
Before using the system, a license file must be installed on each container created. This is explained later.
To remove the image when so desired, use this command:
unix$ docker rmi lx:2
4.2. Docker Installs with Preinstalled Image
4.2.1. Prerequisites
To install, you need:
-
A valid LeanXcale license file
-
The LeanXcale docker image
-
A Linux system with
-
docker version 20.10.6 or later.
-
python 3.7 or later.
-
Your sales representative will provide you with a valid license file. In case you have any issues with it, please send an email to sales@leanxcale.com.
Download the latest LeanXcale docker image from:
https://artifactory.leanxcale.com/artifactory/lxpublic/
There is a docker image file per version.
4.2.2. Install
Working on the directory where the image file has been downloaded, add the image to your docker system:
unix$ docker load --input lx.2.3.docker.tgz Loaded image: lx:2
Double check that the image has been loaded:
unix$ docker images lx:2 REPOSITORY TAG IMAGE ID CREATED SIZE lx 2 bd350f734448 28 hours ago 1.33GB
Create the docker network lxnet
:
unix$ docker network create --driver bridge lxnet
The image is a single one for all containers. The name given when
creating the container determines the host name used (in this example,
lx1
).
Before using the system, a license file must be installed on each container created. This is explained in the next section.
To remove the image when so desired, use this command:
unix$ docker rmi lx:2
4.3. Running leanXcale Docker Containers
This section explains how to create containers from leanXcale images and
how to install licenses and change the lxadmin
password for them.
Provided the leanXcale image named lx:2
, and the docker network
lxnet
, this command runs a leanXcale docker container:
unix$ docker run -dit --name lx1 --network lxnet -p0.0.0.0:14420:14420 lx:2 lx1 b28d30702b80028f8280ed6c55297b23203540387d3b4cfbd52bc78229593e27
In this command, the container name is lx1
, the network used
lxnext
, and the image used lx:2
. The port redirection -p…
exports the SQL port to the underlying host.
It is important to know that:
-
starting the container will start leanXcale if a valid license was installed;
-
stopping the container should be done after stopping leanxcale in it.
The container name (lx1
) can be used to issue commands. For example,
this removes the container after stopping it:
unix$ docker rm -f lx1
The installed container includes the lx
command. Use it to execute
commands to operate the installed DB system.
It is possible to attach to the container and use the ''lx'' command as it can be done on a bare metal host install:
unix$ docker attach lx1 lx1$ lx version ...
Here, we type docker attach lx1
on the host, and lx version
on
the docker container prompt.
Note that if you terminate the shell reached when attaching the docker container, it will stop. Usually, this is not desired.
It is possible to execute commands directly on the executed container. For example:
unix$ docker exec -it lx1 lx version
executes lx version
on the container.
4.3.1. Setting up a License and Admin Password
Starting the container starts leanxcale as well, but not if there is no license installed. In this case we must install a license file in the container.
To install a license file, it must be copied to the container as shown here:
unix$ docker cp ./lxlicense lx1:/usr/local/leanxcale/.lxlicense unix$ docker exec -it lx1 sudo chown lx /usr/local/leanxcale/.lxlicense
To change the password for the lxadmin
user, do this after starting
the database:
unix$ docker exec -it lx1 lx kvcon addusr lxadmin pass? *****
5. AWS Installs
AWS Installs require installing the AWS command line client, used by the installer. For example, on Linux:
unix$ sudo apt-get install awscli
You need a PEM file to be used for accessing the instance, once created. The PEM file and the public key for it can be created with this command:
unix$ ssh-keygen -t rsa -m PEM -C "tkey" -f tkey
Here, we create a file tkey
with the private PEM and a file
tkey.pub
with the public key. Before proceeding, rename the PEM file
to use .pem
:
unix$ mv tkey tkey.pem unix$ chmod 400 tkey.pem
When installing supply the path to the PEM file (without the .pem
extension) to flag -K
, so that lxinst
can find it and use its
base file name as the AWS key pair name).
Now, define your access and secret keys for the AWS account:
unix$ export AWS_ACCESS_KEY_ID=___your_key_here___ unix$ export AWS_SECRET_ACCESS_KEY=___your_secret_here___ ...
To install the public distribution, go to
https://artifactory.leanxcale.com/artifactory/lxpublic
and download the zip for the last version (for example, lx.2.3.232129.zip
).
Before extracting the zip file contents,
make sure that there is no lxdist
directory from previous
versions, or it will include packages that are not for this one.
unix$ rm -f lxdist/*
Extract the zip file:
unix$ unzip lx.2.3.232129.zip Archive: lx.2.3.232129.zip creating: lxdist/ inflating: lxdist/lxinst.v2.3.port.tgz inflating: lxdist/lxavatica.v2.3.libs.tgz ...
Verify that the process completed successfully and there is a directory
lxdist
(with the distribution packages) and the lxinst
program:
unix$ ls lx.2.3.232129.zip lxdist lxinst
To install in AWS, use lxinst
using the aws
property and the
-K
option to supply the key file/name to use. For example:
unix$ lxinst -K tkey aws ... aws instance i-03885ca519e8037a1 IP 44.202.230.8 ... aws lx1 instance id i-0d2287deeb3d45a82
Or, specify one or more properties to set the region, instance type, disk size, and AWS tag. The disk size is in GiB.
The tag is a name and should not include blanks or dots.
It will be used for .aws.leanxcale.com
domain names.
For example:
unix$ lxinst -K tkey aws awsregion us-west-1 awstype t3.large \ awsdisk 30 awstag 'client-tag' ... config: lxinst.conf install done. # To remove resources: ./lxinst.uninstall #To dial the instances: ssh -o StrictHostKeyChecking=no -i xkey.pem lx@18.232.95.2.2
It suffices to specify the key and give a tag, so this command is ok:
unix$ lxinst -K tkey awstag xample
When installing using an awstag
, domain names are set for installed
instances as a convenience.
Each instance is named with the tag and a number. For example, with
the tag xample
, the first instance is xample1.aws.leanxcale.com
,
the second instance in the same install (if any) is xample2
, etc.
Arguments follow the semantics of a configuration file. Therefore, if a host name is specified, it must come after global properties.
Try to follow the conventions and call lx1
the first host installed,
lx2
the second one, etc.
Also, do not specify directories and similar attributes, leave them to the AWS
install process.
If something fails, or the install completes, check for a lxinst.uninstall
command, created by the install process, to remove allocated resource when
so desired.
The detailed configuration file built by the install process is
saved at lxinst.conf
as a reference.
This is not a configuration file written by the user, but
a configuration file including all the install details.
This is an example:
#cfgfile lxinst.conf awstag client-tag awsvpc vpc-0147a6d8e9d6e6910 awssubnet subnet-0b738237bbce66037 awsigw igw-0a160524cf4edab30 awsrtb rtb-040894a1050b58a3f awsg sg-05ee9e0232599026d host lx1 awsinst i-0e0167a4233f76bb1 awsvol vol-01629a4b3694bb5fd lxdir /usr/local/leanxcale JAVA_HOME /usr/lib/jvm/java-1.11.0-openjdk-amd64 addr 10.0.120.100 kvms 100 addr lx1!14400 lxmeta 100 addr lx1!14410 lxqe 100 addr lx1!14420 mem 1024m kvds 100 addr lx1!14500 mem 1024m
In the installed system, the user running the DB is lx
, and
LeanXcale is added to the UNIX system as a standard service (disabled by
default). The instance is left running. You can stop it on your own if
that is not desired.
The command lxinst.uninstall
, created by the
installation, can be used to remove the created AWS resources:
unix$ lxinst.uninstall ...
To use an instance (and host) name other than lx1
, supply your
desired host name. And use a name, not something with dots or special
characters. For example:
unix$ lxinst -K tkey aws lxhost1 lxhost2 ...
creates a network and two instances named lxhost1
and lxhost2
,
and leaves them running.
The tradition is to use names lx1
, lx2
, etc. for the installed hosts.
Once installed, use lx
as in any other system:
unix$ ssh -o StrictHostKeyChecking=no -i tkey.pem lx@44.202.230.8 lx1$ lx stop -v
When installing using a tag, DNS names can be used. The installer output reminds this at the end of the install process:
unix$ lxinst -K tkey awstag xample lxhost1 lxhost2 ... open ports: 14420 install done. # To remove resources: ./lxinst.uninstall # To dial the instances: ssh -o StrictHostKeyChecking=no -i xxkey.pem lx@98.81.32.197 ssh -o StrictHostKeyChecking=no -i xxkey.pem lx@18.204.227.103 # or ssh -o StrictHostKeyChecking=no -i xxkey.pem lx@xample1.aws.leanxcale.com ssh -o StrictHostKeyChecking=no -i xxkey.pem lx@xample2.aws.leanxcale.com
By default, AWS installs use compression but not encryption.
To change this, use the compress
and/or the crypt
global property with
values yes
or no
as desired.
For example:
unix$ lxinst -K tkey aws crypt lxhost1 lxhost2 ...
installs with encryption enabled, and this disables compression:
unix$ lxinst aws -K tkey compress no lxhost1 lxhost2 ...
5.1. The lx System Service
AWS installs creates a systemd
system service for leanXcale.
The service is installed as a disabled service.
This is meant to be used only for single host installs.
When multiple instances are used, instances must be started before using lx start
to start the system, and lx stop
must be used to stop the system before stopping
the instances.
This section is here as a reference, and also because it might be useful on single-host installs.
For example:
unix$ ssh -o StrictHostKeyChecking=no -i tkey.pem lx@44.202.230.8 lx$ systemctl status lx ○ lx.service - LeanXcale Loaded: loaded (/etc/systemd/system/lx.service; disabled; Active: inactive (dead)
In this case, and the rest of this section, we use commands running on the installed AWS instance.
To enable the service, we can:
lx$ sudo su root@lx1# systemctl enable lx Created symlink /etc/systemd/system/multi-user.target.wants/lx.service → /etc/systemd/system/lx.service.
To disable it again:
lx$ sudo su root@lx1# systemctl disable lx Removed /etc/systemd/system/multi-user.target.wants/lx.service.
To start the service by hand:
lx$ sudo su root@lx1# systemctl start lx Removed /etc/systemd/system/multi-user.target.wants/lx.service.
And now we can see its status:
lx$ systemctl status lx ● lx.service - LeanXcale Loaded: loaded (/etc/systemd/system/lx.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2024-08-12 07:19:48 UTC; 1min 14s ago Process: 6822 ExecStart=sh -c /usr/local/leanxcale/bin/lx start -l -v |tee /usr/local/leanxcale/log/start.log (code=exited, status=0/SUCCESS) Tasks: 116 (limit: 9355) Memory: 3.6G CPU: 13.370s CGroup: /system.slice/lx.service ├─6828 bin/spread -c lib/spread.conf ├─6829 bin/kvlog -D log/spread ... Aug 12 07:19:48 lx1 sh[6824]: bin/kvds: started pid 6836 Aug 12 07:19:48 lx1 sh[6824]: bin/kvds -D -c 1536m ds101@lx1!14504 /usr/local/leanxcale/disk/kvds101 ... ...
The DB status should now be as follows:
lx$ lx status status: running
When more than a single instance is installed, the service must be enabled, disabled, started, and/or stopped on all instances involved.
Note that enabling the service makes the DB start when the instance starts, and stop before the instance stops.
This is done by each instance on its own. At start time, the instance runs
lx$ lx start -l
and at stop time, the instance runs
lx$ lx stop -l
The last start and stop output is saved at /usr/local/leanxcale/log/start.log
,
for inspection.
It might be more convenient to leave the service disabled and, after
starting the instances for the install, use lx start
command
to start the system.
5.2. Listing and handling AWS resources (lxaws)
The program lxaws
is used to list or remove AWS installs, and to
start and stop instances and find their status.
usage: lxaws [-h] [-e] [-v] [-d] [-D] [-r region] [-n] [-askpeer] [-yespeer] [-netpeer] [-delpeer] [-p] [-o] [-c] [-status] [-start] [-stop] [tag [tag ...]] lx AWS cmds positional arguments: tag aws tag and/or command args optional arguments: -h, --help show this help message and exit -e define env vars -v verbose -d remove resources -D enable debug diags -r region AWS region -n dry run -askpeer ask peer: tag owner reg vpc -yespeer accept peer:tag pcxid -netpeer set peer net: tag pcxid cidr sec -delpeer del peer: pcxid -p print open ports -o open ports: tag proto port0 portn cidr name -c close ports: tag proto port cidr -status show instance status: tag [host] -start start instances: tag [host] -stop stop instances: tag [host]
Given a region, without any tags, it lists the tags installed:
unix$ lxaws -r us-east-1 xtest.aws.leanxcale.com
Given a tag, it lists the tag resources as found on AWS:
unix$ lxaws -r us-east-1 xtest.aws.leanxcale.com #xtest.aws.leanxcale.com: vpc vpc-0bb89fa4f83fc69c6 subnet subnet-0b5fb20a5372f89da igw igw-08c3cdec1dc865b84 rtb rtb-0e40ace79169b2e08 assoc rtbassoc-0248017196d4be19c sec sg-028614274a930d0ef inst i-041b70633666af01b xtest1.aws.leanxcale.com 18.209.59.230 vol vol-04310af65774fc5e7
It is also possible to supply just the base tag without the domain, as in
unix$ lxaws -r us-east-1 xtest
With flag -e
prints commands to set environment variables with resources found,
as an aid to run other scripts.
unix$ lxaws -e -r us-east-1 xtest.aws.leanxcale.com #xtest.aws.leanxcale.com: export vpc='vpc-0bb89fa4f83fc69c6' export subnet='subnet-0b5fb20a5372f89da' export igw='igw-08c3cdec1dc865b84' export rtb='rtb-0e40ace79169b2e08' export assoc='rtbassoc-0248017196d4be19c' export sec='sg-028614274a930d0ef' export inst='i-041b70633666af01b' export addr='peer11.aws.leanxcale.com' export vol='vol-04310af65774fc5e7'
When more than one tag is asked for, or more than one instance/volume is found, variable names are made unique adding a number to the name, for example:
unix$ lxaws -e peer1 peer2 #peer1.aws.leanxcale.com: export vpc0='vpc-0a50a6e989aa9da9a' export subnet0='subnet-0d9fd3a7d03eca61b' export igw0='igw-0af4279169fd8cab6' export rtb0='rtb-0f5d93a83239c3ada' export assoc0='rtbassoc-0e7d0f74cd780e121' export sec0='sg-01afb3d3c985f7881' export inst0='i-072326e86bcc77e9f' export addr0='peer11.aws.leanxcale.com' export vol0='vol-08ed1c4acdc0eae61' #peer2.aws.leanxcale.com: export vpc1='vpc-023ce3e3c47bbb48f' export subnet1='subnet-0f9af7190d758d6d6' export igw1='igw-0ae3c860a69969a83' export rtb1='rtb-0d45f2059b4696cf4' export assoc1='rtbassoc-0a365cb4472f0b89e' export sec1='sg-04b122e86debcd735' export inst1='i-0b9cf8cff4b46d657' export addr1='peer21.aws.leanxcale.com' export vol1='vol-0f9f344a3a2b9bf38'
With flag -d
, it removes the resources for the tags given.
In this case, tags must be given explicitly in the command line.
unix$ lxaws -d -r us-east-1 xtest.aws.leanxcale.com
5.3. AWS Ports
To list, open, and close ports exported by the AWS install to the rest of the world,
use lxaws
flags -p
(print ports), -o
(open ports), and -c
(close ports).
In all cases, the first argument is the tag for the install.
The tag can be just the AWS install tag name, without .aws.leanxcale.com
.
For example, this command lists the open ports:
unix$ lxaws -p xample.aws.leanxcale.com port: web: tcp 80 0.0.0.0/0 port: ssh: tcp 22 0.0.0.0/0 port: comp: tcp 14420 0.0.0.0/0
Here, the protocol and port (or port range) is printed for each set of open ports.
The CIDR printed shows the IP address range that can access the ports,
and is 0.0.0.0/0
when anyone can access them.
Before with each port range, the name for the open port range is printed. This name is set by the default install, and can be set when opening ports as shown next.
To open a port range, use -o
and give as arguments the tag for the install,
the protocol, first and last port in range, the CIDR (or any
if open to everyone),
and a name to identify why this port range is open (no spaces).
For example:
unix$ lxaws -o xample.aws.leanxcale.com tcp 6666 6666 any opentest
The new port will be shown as open if we ask for ports:
unix$ lxaws -p xample.aws.leanxcale.com port: web: tcp 80 0.0.0.0/0 port: opentest: tcp 6666 0.0.0.0/0 port: ssh: tcp 22 0.0.0.0/0 port: comp: tcp 14420 0.0.0.0/0
As another example:
unix$ lxaws -o xample tcp 8888 10000 212.230.1.0/24 another unix$ lxaws -p xample port: web: tcp 80 0.0.0.0/0 port: ssh: tcp 22 0.0.0.0/0 port: another: tcp 8888-10000 212.230.1.0/24 port: comp: tcp 14420 0.0.0.0/0
To close a port, use -c
and give as arguments the tag for the install,
the protocol, a port within the range of interest, and the CIDR used to export the port.
Note that any
can be used here too instead of the CIDR 0.0.0.0/0
For example:
unix$ lxaws -vc xample tcp 6666 any searching aws... close ports tcp 6666-6666 to 0.0.0.0/0
Here, we used the -v
flag (verbose) to see what is going on.
As another example, this can be used to close the open port range 8888-10000
from the example above:
unix$ lxaws -c xample tcp 9000 2.2.230.1.0/24
5.4. AWS Instance status
To find out the instance statuses for a given AWS tag, use the -status
flag for lxaws
:
unix$ lxaws -status lxi24 #lxi24.aws.leanxcale.com: inst i-05e0708c0e4965ef0 lxi241.aws.leanxcale.com stopped inst i-02bbf1473c01ea6ae lxi242.aws.leanxcale.com 44.203.139.222 running
With just a tag name, it lists the instances for the given tag name.
Supplying extra arguments with a host name, IP, or instance ID, lists just the status for the given instances.
unix$ lxaws -status lxi24 lxi241 #lxi24.aws.leanxcale.com: inst i-05e0708c0e4965ef0 lxi241.aws.leanxcale.com stopped
5.5. AWS Instance start/stop
Flags -start
and -stop
in lxaws
can be used to start/stop instances.
Their use is similar to that of the -status
flag.
Given a tag, all instances are started/stopped.
Given a tag and a host (or more hosts), matching instances are started/stopped.
For example:
unix$ lxaws -stop lxi24 #lxi24.aws.leanxcale.com: inst i-05e0708c0e4965ef0 lxi241.aws.leanxcale.com stopped
Note that this does not start/stop the DB in the instances. The command just starts/stops the instances.
Once instances are started, connect to one of them and use lx start
to start the DB.
before stopping, connect to one of the instances and use lx stop
to stop the DB before
stopping the instances.
5.6. AWS VPC Peering Connections
Peering connections can be used to bridge two VPCs at AWS.
One peer asks the other peer to accept a peering connection, the peer accepts the connection, and network routes and security group rules for access are configured.
Peering connections are handled using lxaws
with the peering connection flags.
If you do not have lxaws
, copy lxinst
to a file named lxaws
and give it
execution permissions.
These are the flags for peering connections:
-askpeer ask peer: tag owner reg vpc -yespeer accept peer:tag pcxid -netpeer set peer net: tag pcxid cidr sec -delpeer del peer: pcxid
-
With flag
-askpeer
,lxaws
requests VPC peering connection. -
With flag
-yespeer
,lxaws
accepts a VPC peering request. -
With flag
-netpeer
,lxaws
sets up the routes and port access rules. -
With flag
-delpeer
,lxaws
removes a peering connection.
To request a peering connection, supply as arguments
-
the tag for the installed system where to setup a peer VPC.
-
the peer AWS owner id (user id).
-
the peer region
-
the peer VPC id
For example:
unix$ lxaws -askpeer xample 232967442225 us-east-1 vpc-0cf1a3b5c1232d172 peervpc pcx-06548783d83ddaba9
Here, we could have used xample.aws.leanxcale.com
instead.
The command prints the peering connection identifier,
to be used for setting up networking and asking
the peer administrator to accept the peering request.
To accept a peer request, supply as an argument the peering connection id, as in:
unix$ lxaws -yespeer pcx-06548783d83ddaba9
In either case, once the dialed peer accepts the request, networking must be set supplying as arguments
-
the tag for the installed system where to setup a peer VPC.
-
the peering connection identifier
-
the peer CIDR block
-
the peer security group id
For example:
unix$ lxaws -netpeer xample pcx-06548783d83ddaba9 10.0.130.0/24 sg-0f277658c2328a955
Our local CIDR block is 10.0.120.0/24
.
This must be given to the peer,
so the peer system can setup routing for this block to our network, along with
the VPC id and our security group id.
This information can be
retrieved using lxaws
as described in the previous section.
For example:
unix$ lxaws xample.aws.leanxcale.com #xample.aws.leanxcale.com: vpc vpc-0f69c4a92b0a78523 peervpc pcx-0e88c49635ed2e59e subnet subnet-0ca70fab7476c2a04 igw igw-0cd6a7bfa99981659 rtb rtb-06e4994a57d37a054 assoc rtbassoc-022b54b9a0216e9f5 sec sg-0df3a6ec01a4ee5ee inst i-0b8144548f0e0f1d8 peer11.aws.leanxcale.com 44.200.78.14 vol vol-0c40de22d908e40bd
Should it be necessary, the CIDR block used by the install can be set when installing
the system (but not later), using the property awsnet
, as in
unix$ lxinst -K mykey aws awsnet 10.0.120 awstag xample
Note that only the network address bytes used are given, instead of using 10.0.120.0/24
.
Once done with a peering connection, it can be dismantled supplying both the tag and the peering connection identifier. The identifier is always given because, when accepting a peering request, the peering connection does not belong to us. But, using the command above you can retrieve such identifier easily.
When peering is no longer desired, the peering connection can be removed.
unix$ lxaws -delpeer xample pcx-005c2b84b89377737
Removing the peering connection also removes the routing entries for it and the security group access rules added when it was setup.
6. HP Greenlake Installs
We currently support VM installs at HP Greenalake. For installing, first create the VM image and in this VM image, install like in bare metal. Follow the steps in Bare Metal Installs. The administration of the database instance will be the same as a bare metal. Just follow the bare metal sections in administration guide.
7. Distributed and Replicated Installs
In general, the system is used the same way it is used when installing using a single host. Refer to the section for the kind of install of interest to learn how to start, stop, and operate the system before reading this section.
As described in the reference manual, most commands take arguments to select particular replicas, hosts, or components. And this is the case for start and stop commands. On replicated installs it is important to start and stop the whole system.
Starting the whole system checks out that replicas are synchronized and takes care of updating outdated metadata of a previously failing or stopped replica.
If a replica is not reachable and start
cannot ensure that the system would start
with the most recent metadata, the system will not start.
On distributed and replicated installs it is possible to ask start
to proceed
with just a single replica or a single host or set of components.
This is done by calling start
with arguments that name just a replica (or perhaps
a host or a set of components).
On replicated installs, two useful names are repl1
and repl2
, to
ask a command to operate on the first or the second replica.
By convention, the first replica is the set of hosts configured that are not mirrors, and the second replica is the set of hosts that are mirrors of former hosts.
As an example, we use this configuration file
# lxinst.conf host blade110 kvds host blade161 mirror blade110
In this case, the first replica is just blade110
and the second replica
is just blade161
.
Installing the system on bare metal is done using
unix$ lxinst -f lxinst.conf
To start the system we execute
unix$ lx start start... blade110 [ bin/spread -c lib/spread.conf ... forked bin/spread... ... ] blade161 [ bin/spread -c lib/spread.conf ... forked bin/spread... ... ] unix$
We can ask for the system status or wait for a status as usual:
unix$ lx status status: running
To stop the system:
unix$ lx stop stop... blade110: [ stop: term lxqe100.r1 pid 460953 stop: term lxmeta100.r1 pid 460932 stop: term kvds100.r1 pid 460927 stop: term kvms100.r1 pid 460923 stop: term spread pid 460919 ] blade161: [ stop: term lxqe100.r2 pid 250984 stop: term lxmeta100.r2 pid 250959 stop: term kvds100.r2 pid 250955 stop: term kvms100.r2 pid 250950 stop: term spread pid 250946 ]
7.1. Partial Starts and Stops
When using multiple hosts and replication, it is possible to start and stop individual hosts or replicas and force the system to run using just those.
For example,
unix$ lx stop repl1 stop... blade110: [ stop: term lxqe100.r1 pid 443056 stop: term lxmeta100.r1 pid 443035 stop: term kvds100.r1 pid 443030 stop: term kvms100.r1 pid 443026 stop: term spread pid 443022 ]
stops the processes in replica-1.
To start it again, we can proceed in a similar way:
unix$ lx start repl1 blade110 [ bin/spread -c lib/spread.conf ... forked bin/spread... bin/spread: started pid 446756 ... ] blade110 [ kvds100.r1 pid 446764 alive disk open spread pid 446756 alive lxmeta100.r1 pid 446769 alive starting lxqe100.r1 pid 446790 alive kvms100.r1 pid 446760 alive ]
Stopping a replica while the system is running is strongly discouraged. Using it again requires restoring the replica state to make it work with the rest of the system.
In this example, if the whole system was running when lx stop repl1
was used,
starting repl1
again will reintegrate it into the system if possible.
However, if we have a fully stopped system, and run
unix$ lx start repl1
the system will run just the first replica. This will happen even if the second replica is not reachable and there is no way to ensure that the metadata in the first replica is up-to-date.
To ensure that the metadata is up-to-date in partial starts, use flag -c
.
This performs the same checks made when starting the whole system, and ensures that
metadata is up to date, before attempting an start of the named replica or components.
7.2. System Status and Replication
The command lx status
reports the status for the system
or waits for a given status, as dsecribed for other installs in this
document.
For example,
unix$ lx status status: running
To see the replication mirror status for the system, use
flag -r
unix$ lx status -r status: running replica: ok
And, to see detailed information about components, use flag -p
, perhaps
in addition to -r
:
unix$ lx status -rp status: running replica: ok kvds100.r1 mirror alive snap 1173999 running kvds100.r2 mirror alive snap 1173999 running kvms100.r1 mirror alive kvms100.r2 mirror alive lxmeta100.r1 mirror alive snap 1173999 running lxmeta100.r2 mirror alive running lxqe100.r1 mirror alive snap 929999 running lxqe100.r2 mirror alive snap 916999 running
In this example, all components are running with their mirror set as ok.
When some components failed, or part of the system was stopped, we can see a different output.
For example, after
unix$ lx stop repl1
We can see
unix$ lx status -rp status: running replica: single kvds100.r2 single alive snap 228678999 single running kvds100.r1 outdated stopped snap 228161999 kvms100.r1 mirror stopped kvms100.r2 mirror alive lxmeta100.r2 mirror alive snap 362692999 running lxmeta100.r1 mirror stopped snap 108718999 lxqe100.r2 mirror alive snap 362664999 running lxqe100.r1 mirror stopped snap 227825999
The first thing to note here is that the replica is not ok
, but single
.
This means we have single processes (without their mirrors) and the system
is running in degraded mode.
Also, kvds100.r2
status with respect to replication is single
.
This means that it was running while its peer (kvds100.r1
, in the first
replica) was stopped.
This server will not be used again until it has been brought up to date with respect to the rest of the system.
Note how kvds100.r1
status with respect to replication is outdated
.
This means it passed away (failed or halted) while its peer was still in use.
The same happens to lxqe100.r2
, but in this case both query engines
could synchronize their mirrors after restarting lxqe100.r1
and nothing else
was needed to permit the restarted process work with the rest of the system.
7.3. System Metadata and Replication
To inspect the status for a replicated system it is useful to look at
DB metadata as stored on disk.
On replicated systems the whole system is using a master metadata
server (kvms
), which synchronizes metadata with its mirror server.
Looking at the disk information may aid in diagnosing the state for the system when some replica is not running, or has been retired from service.
The dbmeta
command can be used to do this.
For example, after running
unix$ lx start repl1
on a replicated system previously halted, we can see
lx status -rp status: running replica: ok kvds100.r1 mirror alive snap 163191999 running kvds100.r2 mirror stopped snap 110005999 kvms100.r1 mirror alive kvms100.r2 mirror stopped lxmeta100.r2 mirror stopped snap 497103000 lxmeta100.r1 mirror alive snap 163670999 running lxqe100.r1 mirror alive snap 163422999 running lxqe100.r2 mirror stopped snap 109917999
The system has not been used, so the mirror status is still ok.
The output for dbmeta
returns what is known by the running kvms
server:
unix$ lx dbmeta /srv dbmeta... # kvms100.r1 kvms blade110!14400 kvms blade161!14400 kvds ds100.r1 at blade110!14500 snap 172793999 rts 112774999 lxmeta mm100.r2 at blade161!14410 snap 497103000 kvds ds100.r2 at blade161!14500 snap 110005999 rts 109069999 lxmeta mm100.r1 at blade110!14410 snap 172793999 lxqe qe100.r1 at blade110!14420 snap 173016999 lxqe qe100.r2 at blade161!14420 snap 109917999
We asked just for metadata of servers using the /srv
resource path name.
The interesting part is that we can ask for metadata as known by both replicas:
unix$ lx dbmeta -a /srv dbmeta... blade110: # kvms100.r1 kvms blade110!14400 kvms blade161!14400 kvds ds100.r1 at blade110!14500 snap 174236999 rts 112774999 lxmeta mm100.r2 at blade161!14410 snap 497103000 kvds ds100.r2 at blade161!14500 snap 110005999 rts 109069999 lxmeta mm100.r1 at blade110!14410 snap 174236999 lxqe qe100.r1 at blade110!14420 snap 174457999 lxqe qe100.r2 at blade161!14420 snap 109917999 blade161: # kvms100.r2 kvms blade161!14400 kvms blade110!14400 kvds ds100.r1 at blade110!14500 snap 110005999 rts 109069999 kvds ds100.r2 at blade161!14500 snap 110005999 rts 109069999 lxmeta mm100.r2 at blade161!14410 snap 497103000 lxmeta mm100.r1 at blade110!14410 snap 110005999 lxqe qe100.r1 at blade110!14420 snap 109920999 lxqe qe100.r2 at blade161!14420 snap 109917999
It can be seen how replica-2 (that for kvms100.r2
) is way out of date
at least with respect to snapshots.
This is not a surprise because it is stopped.
We can ask for the full metadata using
unix$ lx dbmeta -a
or for that for a particular table or index.
7.4. Failures
When there is a failure, the system continues to operate using the mirror processes that remain alive.
Here we describe example failures, and provide details about reparing
specific failed components. Then we describe how to use lx restore
to
try to restore things in a more convenient way.
For example, if qe100.r2
fails (we killed it to make it so), this can
be seen:
unix$ lx status status: running with failures
Further details are reported by flags -r
(replication) and -p
(process):
unix$ lx status -rp status: running with failures replica: ok kvds100.r1 mirror alive snap 232084999 running kvds100.r2 mirror alive snap 232084999 running kvms100.r1 mirror alive kvms100.r2 mirror alive lxmeta100.r1 mirror alive snap 232564999 running lxmeta100.r2 mirror alive running lxqe100.r1 mirror alive snap 232416999 running lxqe100.r2 mirror dead snap 228005999
Component lxqe100.r1
is alive and running, and lxqe100.r2
is dead.
Using now the database produces a change in status:
lxqe100.r1 single alive snap 267003999 single running lxqe100.r2 outdated dead snap 228005999
This means that lxqe100.r1
is known to be single, i.e., it has been used
while its mirror was dead or halted.
Also, lxqe100.r2
is known to be outdated
, i.e., its mirror has been used
while it was dead or halted.
7.5. Using the recover command
The command lx recover
inspects metadata and tries to restore the disk for
failed components.
It can be used before starting the system, to let lx start
start an already
restored replicated system.
To make an example, with the example system running, we killed both
kvds100.r1
and lxqe100.r2
and stopped the system after that.
This is the resulting server metadata:
unix$ lx dbmeta /srv dbmeta... # kvms100.r1 meta ts 15264000 kvms blade110!14400 kvms blade161!14400 lxmeta mm100.r2 at blade161!14410 kvds ds100.r1 at blade110!14500 fail snap 2101999 kvds ds100.r2 at blade161!14500 single snap 14804999 lxmeta mm100.r1 at blade110!14410 snap 15264000 lxqe qe100.r1 at blade110!14420 single snap 14736999 lxqe qe100.r2 at blade161!14420 snap 925999
A dry run (flag -n
) for lx recover
shows this:
unix$ lx recover -n #disk... copy blade110 kvds100.r2 blade161 kvds100.r1 clear kvds100.r2 flag single clear kvds100.r1 flag fail copy blade110 lxqe100.r1 blade161 lxqe100.r2 clear lxqe100.r1 flag single clear lxqe100.r2 flag fail
We can use lx copy
and lx dbmeta -w
to copy disks and clear flags,
but it is more convenient to run this command without the dry run flag
once we are decided to do so.
The recover
command can take arguments
to select what should be recovered, following
the style of most other commands. For example:
lx recover kvds100 #recover... copy blade110 kvds100.r2 blade161 kvds100.r1 clear kvds100.r2 flag single clear kvds100.r1 flag fail
After a recover, the system can be started normally.
8. High Availability (Active-Active Replication)
High availability is attained by means of active-active replication (also called synchronous replication). This means that each write is made to each pair of replicas as part of the same transaction.
This means that if one of the replicas fails the other one will remain operational without any loss of data as it would happen in a database with master-slave (also called asynchronous replication). If one replica fails, the system remains operational, and we say that the system is working in degraded mode (since now there is now high availability, and a failure of the replica still alive will mean that the database is down).
When the failed replica is recovered it will be in sync with the live replica (same data) and the system will become fully operational, that is, highly available again.
Backups are performed with the same command as to perform backups without high availability. The backup for a fully operational highly available system backups one of the replicas (since both are identical). The backup of a degraded system (one replica up and one replica down) will backup the data of the live replica. It should be noted that if the system is shutdown in degraded mode, one the live replica contains the up-to-date data. We call it single (as opposed to married when both replicas are alive). The backup in the case of a degraded system will backup the data of the single replica.
If the system was stopped in degraded mode, it can only be restarted with the single replica.
9. Monitoring Tool
Monitoring in LeanXcale is performed with Prometheus to collect metrics and Grafana as dashboard. Utilizing advanced tools such as Prometheus and Grafana allows administrators to gain complete visibility into the system’s health, quickly identify issues, and optimize performance in real-time.
Prometheus takes care of metrics collection and and its storage. Prometheus is an open-source monitoring system designed to collect and store real-time numerical metrics. Thanks to the integration with Prometheus the integration with other monitoring tools such as Datadog, Instana, or Site24x7 is straightforward. LeanXcale relying on Prometheus, exposes detailed metrics across various aspects of its operation, including:
Database Performance: Metrics related to the throughput of transactions and read and write operations. Resource Consumption: CPU, memory, and disk usage, at component, node, and cluster levels. Prometheus collects these metrics through endpoints exposed by LeanXcale and stores them in its time-series store. This enables querying and analyzing performance data over historical periods, facilitating the identification of patterns and trends.
Grafana is devoted to the visualization and analysis of metrics. Grafana is a powerful data visualization tool integrated with Prometheus, providing an interactive and graphical environment for metric analysis. With Grafana, LeanXcale administrators can:
Create Custom Dashboards: Users can build personalized dashboards that display the most relevant metrics according to their operational needs. These dashboards enable real-time monitoring of the system’s most critical metrics. Configurable Alerts: Grafana allows setting up alerts based on specific thresholds for any metric collected by Prometheus. When a metric exceeds a predefined value, Grafana can send notifications through multiple channels (email, Slack, etc.), enabling teams to respond quickly to potential issues. Historical Analysis: Users can leverage Grafana to perform in-depth analysis of historical metrics, allowing comparisons of performance over different periods, identifying trends, and conducting detailed diagnostics of any incidents. The combination of Prometheus and Grafana provides:
Real-Time Visibility: Administrators gain a clear and up-to-date view of the database’s status, helping maintain high levels of performance and availability. Proactive Diagnostics: With the ability to configure alerts and analyze historical data, administrators can identify and resolve issues before they severely impact end users. Scalability: Both Prometheus and Grafana are designed to handle large volumes of data, making them ideal for distributed and large-scale environments like LeanXcale.
10. LX Command
10.1. LeanXcale Commands
The command lx
is a
Shell for running LeanXcale control programs. This simply fixes the
environment for the installed host and runs the command given as an
argument:
usage: lx [-d] cmd...
The command operates on the whole LeanXcale system, even when multiple hosts are used.
It is convenient to have lx
in the PATH
environment variable, as
suggested in the install program output.
Command output usually includes information on a per-host basis reporting the progress of the used command.
Most commands follow the same conventions regarding options and arguments. We describe them here as a convenience.
Arguments specify what to operate (e.g., what to start, stop, etc.) may be empty to rely on the defaults (whole DB) or may specify a particular host and/or component name:
-
when only component names are given, and only those components will be involved (e.g.,
lxqe101
). -
when a component type name is given, components for that type are selected. (e.g.,
lxqe
). -
when a host name is given, any component following is narrowed to that host. If no components follow the host name, all components from the host are selected.
This may be repeated to specify different hosts and/or components.
The special host names db
, repl
, and repl2
may be used and
stand for hosts without the nodb
attribute, hosts for the first
replica, and hosts for the second replica (hosts that are a mirror of
other ones).
10.1.1. LeanXcale Commands on Bare Metal Installs
For bare metal installs, it suffices to have the lx
command in the PATH.
It can run on any of the installed hosts.
For example, on an installed host, lx version
prints the installed version:
unix$ lx version leanXcale v2.2 kv v2.2.2023-09-29.115f5fba70e3af8dc203953399088902c4534389 QE v2.2.2023-09-30.1e5933900582.26a7a5c3420cd3d5d589d1fa6cc libs v2.2.2023-09-29.67535752acf19e092a6eaf17b11ad17597897956 avatica v2.2.2023-09-27.0b0a786b36e8bc7381fb2bb01bc8b3ed56f49172 TM v2.2.2023-09-29.9a9b22cfdc9b924dbc3430e613cddab4ed667a57
10.1.2. LeanXcale Commands on Docker Installs
To use the lx
command on a docker install, an installed container must be
running, and the command must be called on it.
For example, assume that the container named lx1
is running on a Docker install.
The container could be started using the following command,
assuming the leanXcale image is named lx:2
, and the docker network used is
lxnet
:
unix$ docker run -dit --name lx1 --network lxnet -p0.0.0.0:14420:14420 lx:2 lx1 b28d30702b80028f8280ed6c55297b2e203540387d3b4cfbd52bc78229593e27
It is possible to attach to the container and use the ''lx'' command as it can be done on a bare metal host install:
unix$ docker attach lx1 lx1$ lx version ...
Here, we type docker attach lx1
on the host, and lx version
on
the docker container prompt.
Note that if you terminate the shell reached when attaching the docker container, it will stop. Usually, this is not desired.
It is possible to execute commands directly on the executed container. For example:
unix$ docker exec -it lx1 lx version
executes lx version
on the lx1
container.
In what follows, lx1
is used as the container name in the examples for docker
installs.
10.1.3. LeanXcale Commands on AWS Installs
Using lx
on AWS hosts is similar to using it on a bare-metal install.
The difference is that you must connect on the AWS instance to run the command
there.
For example, after installing xample1.aws.leanxcale.com
, and provided the
PEM file can be found at xample.pem, we can run this:
unix$ ssh -i xample.pem xample1.aws.leanxcale.com lx version
to see the installed version.
In what follows, xample.pem
is used as the PEM file name
and xample1.aws.leanxcale.com
is used as the installed instance name,
for all AWS install examples.
11. Starting the System
11.1. Bare Metal System Start
The start
command starts LeanXcale:
unix$ lx start start... atlantis [ cfgile: /ssd/leandata/xamplelx/lib/lxinst.conf... bin/spread -c lib/spread.conf ... forked bin/spread... bin/spread: started pid 1056053 bin/kvms -D 192.268.1.224!9999 /ssd/leandata/xamplelx/disk/kvms100/kvmeta ... forked bin/kvms... ... ] atlantis [ kvds103 pid 1056084 alive kvms100 pid 1056057 alive spread pid 1056053 alive kvds102 pid 1056075 alive kvds100 pid 1056062 alive kvds101 pid 1056066 alive ] unix$
Here, atlantis
started a few processes and, once done, the start
command checked out if the processes are indeed alive.
In case not all components can be started successfully, the whole LeanXcale system is halted by the start command.
By default not watcher or automatic restart is setup.
Using flag -r
asks start to start the system asking it to restart any QE that
was running, failed, and was not restarted less than one minute ago.
using flag -w
asks start to start lx watch
.
The watch tool will wait until the system becomes operational and, upon failures,
try to restart the whole system.
To start a single host or component, use its name as an argument, like in:
# start the given host unix$ lx start atlantis # start the named components unix$ lx start kvds # start the named components at the given host unix$ lx start atlantis kvds
Start does not wait for the system to be operational. To wait until the
system is ready to handle SQL commands, the status
command can be
used with the -w
(wait for status) flag, as in:
unix$ lx status -w running status: running
Without the -w
flag, the command prints the current status, which
can be stopped, failed, running, or waiting.
11.2. Docker System Start
To start LeanXcale installed on Docker containers, you must start the containers holding the installed system components.
For example, consider the default docker install
unix$ lxinst docker ... install done docker images: REPOSITORY TAG IMAGE ID CREATED SIZE uxbase 2 7c8262008dac 3 months ago 1.07GB lx 2 cafd60d35886 3 seconds ago 2.62GB docker network: NETWORK ID NAME DRIVER SCOPE a8628b163a21 lxnet bridge local to start: docker run -dit --name lx1 --network lxnet lx:2 lx1
The install process created a docker image named lx:2
, installed for the docker
host lx1
, and the docker network lxnet
.
To list the image we can
unix$ docker images lx REPOSITORY TAG IMAGE ID CREATED SIZE lx 2 75b8c9ffa245 About a minute ago 2.62GB
And, to list the networks
unix$ docker network ls NETWORK ID NAME DRIVER SCOPE a8628b163a21 lxnet bridge local
The created image is a single one for all containers. The name given when creating the container determines the host name used The install process specified the host names, and containers must be starte using the corresponding host name(s), so they know which leanXcale host they are for.
For example, to start the container for lx1
:
unix$ docker run -dit --name lx1 --network lxnet -p0.0.0.0:14420:14420 lx:2 lx1 b28d30702b80028f8280ed6c55297b2e203540387d3b4cfbd52bc78229593e27
In this command, the container name is lx1
, the network used
lxnext
, and the image used lx:2
. The port redirection -p…
exports the SQL port to the underlying host.
Listing docker processes shows now the running container
unix$ docker ps CONTAINER ID IMAGE COMMAND STATUS PORTS NAMES e81d9d01f40a lx:2 "/bin/lxinit lx1" Up 56 seconds 14410 lx1
It is important to know that:
-
starting the container will start leanXcale if a valid license was installed;
-
stopping the container should be done after stopping leanxcale in it.
The container name (lx1
) can be used to issue commands. For example,
unix$ docker exec -it lx1 lx version leanXcale v2.1 unstable kv v2.1.2.14-02-15.c26f496706918e610831c02e99da3676a1cffa47 lxhibernate v2.1.2.14-02-07.f65c5a628afede27c15c77df6fbbccd6d781d3ee TM v2.1.2.14-02-06.bfc9f92216481dd05f51900ac522e5ccfb6d2555 QE v2.1.2.14-02-15.4a8ff4200dc3d3656c8469b6f74c05a296fbdfb3 avatica v2.1.2.14-02-14.1c442ac9e630957ace3fdb5c4faf92bb85510099 ...
executes lx version
on the lx1
container.
The status for the system can be seen in a similar way:
unix$ docker exec -it lx1 lx status status: running
Note that the container will not start the DB if no valid license is found.
11.3. AWS System Start
To start LeanXcale installed on AWS, you must start the AWS instances holding the installed system components.
This can be done by hand using the AWS console, or using the lxaws
command.
For example, after installing using xample
as an AWS tag, this command
starts the instances:
unix$ lxaws -start xample
Once instances are started, the lx
command is available at any of them.
For example, provided the PEM file can be found at xample.pem, we can run this:
unix$ ssh -i xample.pem xample1.aws.leanxcale.com lx version
to see the installed version.
Here xample1
is the DNS host name registered as the first host for the install
AWS tag xample
. In the same way, xample2
would be the name for the second host, and
so on.
12. Checking System Status
12.1. Bare Metal System Status
The command lx status
reports the status for the system
or waits for a given status.
For example,
unix$ lx status status: waiting kvds100: recovering files kvds101: recovering files
Or, to wait until the status is running
:
unix$ lx status -v -w running status: waiting kvds100: recovering files kvds101: recovering files status: running unix$
To see the status for each one of the processes in the system, use lx procs
.
For example:
unix$ lx procs procs... atlantis [ kvds103 pid 1057699 alive running kvms100 pid 1057672 alive running spread pid 1057668 alive kvds102 pid 1057690 alive running kvds100 pid 1057677 alive running kvds101 pid 1057681 alive running ]
12.2. Docker System Status
Before looking at the LeanXcale system status, it is important to look at the status of the docker containers running LeanXcale components.
unix$ docker ps CONTAINER ID IMAGE COMMAND STATUS PORTS NAMES e81d9d01f40a lx:2 "/bin/lxinit lx1" Up 56 seconds 14410 lx1
When containers are running,
the command lx status
reports the status for the system
or waits for a given status.
For example,
unix$ docker exec -it lx1 lx status
executes lx status
on the lx1
container.
The status is reported for the whole system, and not just for that container.
To wait until the status is running
:
unix$ docker exec -it lx1 lx status -v -w running status: waiting kvds100: recovering files kvds101: recovering files status: running
To see the status for each one of the processes in the system, use lx procs
.
For example:
unix$ docker exec -it lx1 lx procs procs... atlantis [ kvds103 pid 1057699 alive running kvms100 pid 1057672 alive running spread pid 1057668 alive kvds102 pid 1057690 alive running kvds100 pid 1057677 alive running kvds101 pid 1057681 alive running ]
12.3. AWS System Status
Before looking at the LeanXcale system status, it is important to look at the status of the AWS instances running LeanXcale components.
This can be done using the lxaws
-status
flag with the installed AWS tag name:
unix$ lxaws -status xample #xample.aws.leanxcale.com: inst i-02bbf1473c01ea6ae xample2.aws.leanxcale.com stopped inst i-05e0708c0e4965ef0 xample1.aws.leanxcale.com 54.84.39.77 running
When instances are running,
the command lx status
reports the status for the system
or waits for a given status.
For example,
unix$ ssh -i xample.pem xample1.aws.leanxcale.com lx status
to see the system status.
To wait until the status is running
:
unix$ ssh -i xample.pem xample1.aws.leanxcale.com lx status -v -w running status: waiting kvds100: recovering files kvds101: recovering files status: running
To see the status for each one of the processes in the system, use lx procs
.
For example:
unix$ ssh -i xample.pem xample1.aws.leanxcale.com lx procs procs... atlantis [ kvds103 pid 1057699 alive running kvms100 pid 1057672 alive running spread pid 1057668 alive kvds102 pid 1057690 alive running kvds100 pid 1057677 alive running kvds101 pid 1057681 alive running ]
13. Stopping the System
13.1. Bare Metal System Stop
The stop
command halts LeanXcale:
unix$ lx stop stop... atlantis [ kvcon[1056801]: halt ] atlantis [ term 1056062 2056066 1056075 1056084 1056057 1056053... kill 1056062 2056066 1056075 1056084 1056057 1056053... ] unix$
13.2. Docker System Stop
Stopping the LeanXcale containers should be done after stopping leanxcale. The reason is that docker might timeout the stop operation if the system is too busy updating the disk during the stop procedure.
To stop the database,
unix$ docker exec -it lx1 lx stop
stops the components for the whole system (being lx1
an installed container).
We can double check this
unix$ docker exec -it lx1 lx status status: stopped
Once this is done, we can stop the docker container.
unix$ docker ps CONTAINER ID IMAGE COMMAND STATUS PORTS NAMES e81d9d01f40a lx:2 "/bin/lxinit lx1" Up 56 seconds 14410 lx1
unix$ docker stop lx1 lx1
unix$ docker ps unix$
We can also remove the container, but, note that doing this removes all data in the container as well.
unix$ docker rm lx1 lx1 unix$
13.3. AWS System Stop
To stop leanXcale on AWS, you must stop leanXcale before stopping the AWS instances running it. For example
unix$ ssh -i xample.pem xample1.aws.leanxcale.com lx stop
This stops the system on all the instances it uses.
Then, the instances can be stopped.
This can be done on the AWS console, or using
the lxaws
-stop
flag with the installed AWS tag name:
unix$ lxaws -stop xample #xample.aws.leanxcale.com: inst i-02bbf1473c01ea6ae xample2.aws.leanxcale.com stopping inst i-05e0708c0e4965ef0 xample1.aws.leanxcale.com stopping
14. Start & Stop Particularities on Different Installs
System start depends on how the system has been installed. For bare-metal installations, the administrator installing the system is responsible for adding a system service that brings LeanXcale into operation when the machine starts, and stops LeanXcale before halting the system.
For AWS installations, LeanXcale is added as a service, disabled by default. Do not use the service on multi-host installs, the service starts/stops the DB and that requires DB processes to be accessible. That might not be the case with multiple instances.
When the service is enabled, starting the instance starts the LeanXcale service, and stopping the instance stops LeanXcale before the instance stops.
Otherwise, starting leanXcale requires to dial into the installed instance
(one of them) and issue the lx start
command.
For Docker installations, starting a container starts the LeanXcale service on it, and, for safety, LeanXcale should be halted before halting the container (otherwise Docker might decide to time-out and stop the container before LeanXcale did fully stop).
15. Configuring the System
The lx config
command prints or updates the configuration used for the
LeanXcale system:
unix$ lx config cfgile: /usr/local/leanxcale/lib/lxinst.conf... #cfgfile /usr/local/leanxcale/lib/lxinst.conf host localhost lxdir /usr/local/leanxcale JAVA_HOME /usr/lib/jvm/java-1.11.0-openjdk-amd64 addr 127.0.0.1 odata 100 addr localhost!14004 kvms 100 addr 127.0.0.1!14400 lxmeta 100 addr 127.0.0.1!16500 lxqe 100 addr 127.0.0.1!16000 kvds 100 addr 127.0.0.1!15000 kvds 101 addr 127.0.0.1!15002 kvds 102 addr 127.0.0.1!15004 kvds 103 addr 127.0.0.1!15006
The configuration printed provides more details than the configuration file (or command line arguments) used to install.
NB: when installing into AWS or docker, the configuration printed
might lack the initial aws
or docker
property used to install it.
It is possible to ask for particular config entries, like done here:
unix$ lx config kvms addr
It is also possible to adjust configured values, like in
unix$ lx config -s lxqe mem=500m
used to adjust the lxqe
mem
property to be 500m
in all lxqe
components configured.
16. System Recovery
The lxmeta
component watches the status for other components and will stop
the system when there is a failure that cannot be recovered online.
Should the system crash or fail-stop, upon a system restart, lxmeta
will
guide the system recovery.
At start time, each system component checks out its on-disk information and decides to start either as a ready component or as a component needing recovery.
The lxmeta
process, guides the whole system start process following these steps:
-
Wait for all required components to be executing.
-
Look up the component status (ready/recovering).
-
If there are components that need recovery, their recovery process is executed.
-
After all components are ready, the system is made available by accepting queries.
The command lx status
can be used both to inspect the system status and the
recovering process, or to wait until the recovery process finishes and the system
becomes available for queries.
17. System Logs
Logs are kept on a per-system directory, named log
, kept at the install
directory for each system.
Log files have names similar to
kvms100.240214.1127.log
Here, the component name comes first, and then, the date and time when the log file was created. When a log file becomes too big, a new one is started for the given component.
It is convenient to use the lx logs
command to list and inspect logs.
It takes care of reaching the involved log files on the involved hosts.
For example, to list all log files:
unix$ lx logs logs... atlantis: [ log/kvds100.240214.1127.log 250.00K log/kvms100.240214.1127.log 35.39K log/lxmeta100.240214.1127.log 24.10K log/lxqe100.240214.1127.log 445.80K log/spread.240214.1127.log 963 log/start.log 426 ]
Here, atlantis
was the only system installed.
We can give host and/or component names as in many other commands to focus on those systems and/or components.
For example, to just just logs for kvms
processes:
unix$ lx logs kvms logs... atlantis: [ log/kvms100.240214.1127.log 35.39K ]
Or, to list only those for the kvms100
:
unix$ lx logs kvms100
To list logs for the atlantis
host:
unix$ lx logs atlantis
To list logs for kvms
components within atlantis
:
unix$ lx logs atlantis kvms
To list logs for kvms
components at atlantis
and kvds
at orion
:
unix$ lx logs atlantis kvms orion kvds
With flag -p
, logs are printed in the output instead of being listed.
unix$ lx logs -p lxmeta atlantis: [ log/lxmeta100.240214.1127.log 24.10K [ # pid 3351755 cmd bin/javaw com.leanxcale.lxmeta.LXMeta -a atlantis!14410 ...
Flag -g
greps the logs for lines with the given expression.
For example:
unix$ lx logs -g fatal
And, flag -c
copies the logs to the given directory
unix$ lx logs -c /tmp
When printing and copying the logs, only the last log file for each component
are used.
To operate on all the logs and not just on the last one, use flag -a
too:
unix$ lx logs -a -c /tmp
18. Backups
Backups can be made to a external location (recommended to tolerate
disk failures) or within an installed host. External backups (i.e.,
to an external location) are made using the lxbackup
tool, installed
at the bin
directory, which works with a given configuration file.
Internal backups (i.e., to a directory on installed hosts) are made
using the lx backup
tool instead.
usage: lxbackup [-h] [-v] [-D] [-n] [-f cfgfile] [-d dir] [-i] [-e] [-r] [-o] [-F] [what [what ...]] make a backup positional arguments: what host/comps optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -f cfgfile config for the install -d dir root backup dir -i incremental backups -e encrypt -r restore -o online backup -F force command
Using lxbackup
is exactly like using lx backup
with a few
differences:
-
Flag
-f
is mandatory on the first invocation and provides the installed configuration file. -
The default backup directory is not
$LXDIR/dump
, but./lxdump
. -
The command
kvtar
must be available at the host runninglxbackup
.
The external host must have ssh access to the installed hosts.
The configuration file used must be the one retrieved using lx config
, and
not the one written by the user to perform the install, because lx config
reports addresses and details needed by the command.
Once a backup has been created, the configuration used is saved along with the
backup data and there is no need to use -f
to supply it.
Encryption/decryption happens at the source/destination of data when
creating/extracting backups.
Therefore, there is no key kept at the host running lxbackup
.
To prepare a host to use lxbackup
, get the lxbackup
command, the configuration,
and the kvtar
command and copy them to the host.
You might want to copy also lxrestore
and lxbackups
.
These commands can be found at the installed $LXDIR/bin
directory on any installed host. If you are not sure regarding the
$LXDIR
value, use this command to find it:
unix$ lx -d pwd /usr/local/leanxcale unix$
Flag -d
for lx
makes it change to the $LXDIR
directory
before running the given command.
The detailed configuration file to be used is kept at $LXDIR/lib/lxinst.conf
.
The configuration can be retrieved using the
lx config
command.
For example, this creates ./lxinst.conf
with the detailed configuration:
unix$ lx config -o lxinst.conf saved lxinst.conf
As an example, we can setup an external host named
orion
to perform backups in this way:
unix$ lx -d pwd /usr/local/leanxcale unix$ lx config -o /tmp/lxinst.conf saved /tmp/lxinst.conf unix$ cd /usr/local/leanxcale/bin unix$ scp lxbackup lxbackups lxrestore kvtar /tmp/lxinst.conf orion:~
And then just:
orion$ lxbackup -f lxinst.conf
To create a full, cold, backup when leanXcale is not running:
orion$ lxbackup -f lxinst.conf #disk... new 240809 make atlantis disk/kvms100 /usr/local/leanxcale/dump/240809/kvms100 make atlantis disk/kvds100 /usr/local/leanxcale/dump/240809/kvds100 make atlantis disk/kvds101 /usr/local/leanxcale/dump/240809/kvds101 make atlantis disk/lxqe100/log /usr/local/leanxcale/dump/240809/lxqe100 unix$
Flag -f
must be used the first time at least. Once a backup has been made,
the configuration is saved and there is no need to supply it again.
The printed name is the name for the directory keeping the backup, as
used when restoring it. In this case, 240809
.
The files kept at the backup are compressed, and must uncompressed if copied by hand. The restore command takes care of this.
Using flag -e
both encrypts and compresses the backup files.
Flags belong to disk
, note that backup
is the argument given to disk
.
The key used to encrypt the backed files is kept at $LXDIR/lib/lxkey.pem
on the
installed sites as set up by the installer.
orion$ lxbackup -e #disk... new 24080901 make atlantis disk/kvms100 /usr/local/leanxcale/dump/24080901/kvms100 crypt make atlantis disk/kvds100 /usr/local/leanxcale/dump/24080901/kvds100 crypt make atlantis disk/kvds101 /usr/local/leanxcale/dump/24080901/kvds101 crypt make atlantis disk/lxqe100/log /usr/local/leanxcale/dump/24080901/lxqe100 crypt
To perform a backup while the system is running, supply flag -o
(for online).
18.1. Incremental Backups
To create an incremental backup use the flag -i
of lxbackup
(or the same flag with lx backup
for an internal backup).
orion$ lxbackup -i #disk... new 24080902+ incr: lxqe100 += lxqe100
The output reports which components got files backed up.
The incremental backup is encrypted if the total backup it refers to is
encrypted too. No flag -e
should be given.
The incremental backup can be performed while the system is running.
18.2. Listing Backups
To list the backups known use lxbackups
(or lx backups
for internal backups).
orion$ lxbackups 240810 ts 1419000 24081002+ ts 1419000 ...
Those with a +
in their names are incremental backups.
Verbosity can be increased adding one or more -v
flags:
orion$ lxbackups -v #disk... 240810 ts 1419000 240810/kvms100 ts 2551000 240810/kvds100 ts 1292000 ...
orion$ lxbackups -vv #disk... 240810 ts 1419000 240810/kvms100 ts 2551000 240810/kvms100/kvmeta sz 239 ts 2551000 mt 1723285636 240810/kvds100 ts 1292000 240810/kvds100/dbf00001.kv sz 393216 ts 1419000 mt 1723285636 ...
orion$ lxbackups -vvv #disk... 240810 ts 1419000 240810/kvms100 ts 2551000 240810/kvms100/kvmeta sz 239 mt 1723285636 localhost ts 1 2551000 bck 0 0 na 0 240810/kvds100 ts 1292000 240810/kvds100/dbf00001.kv sz 393216 mt 1723285636 localhost ts 1292000 1419000 bck 0 0 na 3 tname db-APP-PERSONS rmin: tpmin rmax: tpmax ...
It is possible to list a single backup by suppling its name and/or
specific components using arguments as done with most lx
commands:
orion$ lxbackups 24080901
or
orion$ lxbackups 24080901 lxqe
18.3. Removing Old Backups
To remove a backup, it suffices to remove its directory. For example:
unix$ lx -d rm -rf dump/230720
Here we used the flag “-d” for lx
to change to $LXDIR
before
executing the remove command, which makes it easy to name the directory
used for the dump.
Or, from our external backup example host:
orion$ rm -rf lxdump/230720.2
Beware that if you remove a backup, you should remove also those incremental backups that follow, up to the next total backup.
18.4. Restore
To restore a backup, use the lxrestore
command (or lx restore
for internal backups).
orion$ lxrestore -v #disk... check... restore 24080904... restore atlantis disk/kvds100 from /usr/local/leanxcale/dump/24080904/kvds100 crypt dbf00001.kv dbf00003.kv ...
Do this while the system is stopped. By default, it selects the last backup made.
To restore a particular backup, supply its name:
orion$ lxrestore 24080904
This can be done also for incremental backups. When restoring an incremental backup, the restore takes data also from previous incremental backups and from the previous total backup.
To restore only specific hosts or components, supply their names or types as done for other commands:
orion$ lxrestore lxqe
Or perhaps:
orion$ lxrestore 24080904 lxqe
To restore a backup, it is usually desirable to format the disk for the
involved components before using restore
to restore their disks.
19. Reporting Issues
To report an issue, use lx report
to gather system information.
This program collects information from the system and
builds an archive to be sent to support.
unix$ lx report report: lxreport.231009... version... procs... logs... stacks... stack lxmeta100... stack kvds103... stack kvms100... stack spread... stack kvds102... stack kvds100... stack kvds101... stack lxqe100... # send this file to support. -rw-rw-r-- 1 leandata leandata 54861 Oct 9 14:58 lxreport.231009.tgz
As printed by the command output, the resulting tar file should be sent to support.
The archive includes:
-
installed version numbers
-
underlying OS names are versions
-
complete disk usage for the installed systems
-
complete process list for the installed systems
-
memory usage for the installed systems
-
lx process list
-
logs for components (last log file only, for each one)
-
stacks for each component
-
stacks for each core file found
When kvms
is still running, the archive includes also:
-
statistics for the sytem
-
long list of kv resources
-
process list for each kvds
-
file list for each kvds
20. Command Reference Manual
This section details all the commands available and how to use them, starting with the install command.
20.1. lxinst
This is the install program:
usage: lxinst [-h] [-v] [-D] [-s] [-m] [-n] [-f cfgfile] [-d dist] [-k key] [-K awspem] [-u location] [-c] [-i] [where ...] leanXcale installer positional arguments: where [aws|docker|stats] dir, host, or host:dir options: -h, --help show this help message and exit -v verbose -D enable debug diags -s small install -m medium install -n dry run -f cfgfile use this configuration file -d dist distrib file, dir, or url -k key key to download the distribution. -K awspem AWS pem file name w/o .pem -u location update inst at location -c clean. download everything -i ignore system limits
Given arguments specify where to install the DB. They may be paths for
directories in the local host, or host names, or host and directory
names using the syntax host:dir
.
Instead of arguments, a configuration file may be given using the -f
flag.
The configuration file has been described before in this document.
Flag -k
may be used to specify the key to download the distribution.
Flag '-K' must be used when installing on AWS to supply the path (without file name extension) to the pem/pub key files.
By default, the distribution is retrieved from the leanXcale
artifactory. However, it is possible to supply one or more times the
-d
flag giving it the name for a .tgz
file, a directory, or a
URL. The special argument leanxcale
stands for the official
repository for the distribution.
Distribution packages will be retrieved by looking at those distribution
sources. By convention, the first source is usally the ./lxdist
directory, used to keep the distribution when downloaded.
For example, this installs just what is downloaded at ./lxdist
,
without downloading anything else:
unix$ lxinst -d ./lxdist -f lxinst.conf
Once downloaded, files are not downloaded again. Flag ''-c'' cleans the ''./lxdist'' directory to force a download of everything. If a different directory is specified by the user, it is not cleaned, although packages not found will be still downloaded there. To force a download of particular packages, remove the packages desired from the ''./lxdist'' directory (or the directory specified in the arguments).
For example, this downloads a fresh copy of the distribution and installs it:
unix$ lxinst -c -f lxinst.conf
And this tries first the ../lxdist
directory and then the standard
leanxcale repository:
unix$ lxinst -d ./lxdist -d leanxcale -f lxinst.conf
This is actually the default when no distribution source is specified.
Also, if no directory source is specified as a first option, lxdist
is used as a directory to download components (other than files found on
the local host).
The database has users (different from UNIX users). The user
lxadmin
is the administrator for the whole system. During the
install you will be asked to type a password for lxadmin
. To specify
the password without being asked, you can set the password in the
LXPASS
environment variable:
unix$ export LXKEY=nemo:APA...uQy unix$ export LXPASS=fds92f3c unix$ lxinst /usr/local/leanxcale ...
When the users file ./lxowner
exists, that file is used as the file
describing the lxadmin
user and its secret, instead of creating one
using the user supplied password for lxadmin
.
Do not use a existing file unless you know what you are doing. The
reason is that this file must define lxadmin
as the first user and
that such user must have access to all resources, as configured.
Flag -i
makes lxinst
ignore system limits, to install on places with
reduced disk or memory and in a hurry.
By default, the install performs a large install, trying to use all the
machine resources for the service.
Flag -s
selects a small instead, and flag -m
selects a medium install.
The install size affects components and sizes not specified by the user in
command line arguments or in the configuration file.
In a small install, at most 4 kdvs
are added per host, and component
memory is limited to 1GiB.
In a medium install, at most 8 kvds
components are added, and
component memory is left with default values.
Flag -u
is used to update a previous install.
Still WIP, do not use it for now.
20.2. lx
Shell for running LeanXcale control programs. This simply fixes the environment for the installed host and runs the command given as an argument:
usage: lx [-d] cmd...
When flag -d
is used, the current working directory is set to the
install directory at the host before running the given command.
Most commands follow the same conventions regarding options and arguments. We describe them here for convenience:
positional arguments: what host|comp... options: -h, --help show this help message and exit -l local run only -D enable debug diags
Flag -l
is used when executing the command locally. This is used by
the lx
command framework, and should not be used in general by the end
user.
Flag -D
enables verbose diagnostics
Arguments specify what to operate (e.g., what to start, stop, etc.) may be empty to rely on the defaults (whole DB) or may specify a particular host and/or component name:
-
when only component names are given, and only those components will be involved (e.g.,
lxqe101
). -
when a component type name is given, components for that type are selected. (e.g.,
lxqe
). -
when a host name is given, any component following is narrowed to that host. If no components follow the host name, all components from the host are selected.
This may be repeated to specify different hosts and/or components.
The special host names db
, repl
, and repl2
may be used and
stand for hosts without the nodb
attribute, hosts for the first
replica, and hosts for the second replica (hosts that are a mirror of
other ones).
20.3. lx addlib
Add the given file(s) to the lib directory of the installed hosts, to add or update a library or jar in the installation:
usage: addlib [-h] [-l] [-v] [-D] [-a host] [-x] [files [files ...]] add libs to installed lx positional arguments: files files to add optional arguments: -h, --help show this help message and exit -l local run only -v verbose -D enable debug diags -a host copy to this installed host(s) -x internal use only
20.4. lx backup
usage: backup [-h] [-v] [-D] [-n] [-d dir] [-F] [-i] [-e] [-r] [-o] [what [what ...]] make a backup positional arguments: what host/comps optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -d dir root backup dir -F force command -i incremental backups -e encrypt -r restore -o online backup
By default, backups are kept at $LXDIR/dump
at the host used to issue
backup commands. The convention is to use the first host to keep
backups.
That is for internal backups (backups at a installed host).
For external backups, lxbackup
is used at the host
keeping the backup, and
the backup top-level
directory is given with flag -d
or defaults to ./lxdump
otherwise.
Directories under dump/
are named using the date as their name.
When more than one backup is made on the same date, further digits are added
to number succesive backups.
A sorted list for the directory reports backups from oldest to newest.
For incremental backups, a final +
is added to the directory name.
To create a full, cold, backup when leanXcale is not running:
unix$ lx backup #backup... new 240809 make atlantis disk/kvms100 /usr/local/leanxcale/dump/240809/kvms100 make atlantis disk/kvds100 /usr/local/leanxcale/dump/240809/kvds100 make atlantis disk/kvds101 /usr/local/leanxcale/dump/240809/kvds101 make atlantis disk/lxqe100/log /usr/local/leanxcale/dump/240809/lxqe100 unix$
The printed name is the name for the directory keeping the backup, as
used when restoring it. In this case, 240809
.
The files kept at the backup are compressed, and must uncompressed if copied by hand. The restore command takes care of this.
Using flag -e
both encrypts and compresses the backup files.
Flags belong to disk
, note that backup
is the argument given to disk
.
The key used to encrypt the backed files is kept at $LXDIR/lib/lxkey.pem
on the
installed sites as set up by the installer.
unix$ lx backup -e #backup... new 24080901 make atlantis disk/kvms100 /usr/local/leanxcale/dump/24080901/kvms100 crypt make atlantis disk/kvds100 /usr/local/leanxcale/dump/24080901/kvds100 crypt make atlantis disk/kvds101 /usr/local/leanxcale/dump/24080901/kvds101 crypt make atlantis disk/lxqe100/log /usr/local/leanxcale/dump/24080901/lxqe100 crypt
To create an incremental backup use the flag -i
.
unix$ lx backup -i #backup... new 24080902+ incr: lxqe100 += lxqe100
The output reports which components got files backed up.
The incremental backup is encrypted if the total backup it refers to is
encrypted too. No flag -e
should be given.
The incremental backup can be performed while the system is running.
To create a backup while the system is running (known as an online backup), use
flag -o
for lx backup
.
unix$ lx backup -o #backup... new 24080902 ...
This is more expensive than doing a backup with the system down. The flag is required to prevent mistakes. If the system is down when an online backup is asked for, the backup is still made as a cold backup. That is, the flag says that it is Ok to backup while the system is online.
20.5. lxbackup
This is the external version of lx backup
, for use on external (non installed)
hosts to keep external backups.
usage: lxbackup [-h] [-v] [-D] [-n] [-f cfgfile] [-d dir] [-i] [-e] [-r] [-o] [-F] [what [what ...]] make a backup positional arguments: what host/comps optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -f cfgfile config for the install -d dir root backup dir -i incremental backups -e encrypt -r restore -o online backup -F force command
Using lxbackup
is exactly like using lx backup
with a few
differences:
-
Flag
-f
is mandatory on the first invocation and provides the installed configuration file. -
The default backup directory is not
$LXDIR/dump
, but./lxdump
. -
The command
kvtar
must be available at the host runninglxbackup
.
The external host must have ssh access to the installed hosts.
The configuration file used must be the one retrieved using lx config
, and
not the one written by the user to perform the install, because lx config
reports addresses and details needed by the command.
Once a backup has been created, the configuration used is saved along with the
backup data and there is no need to use -f
to supply it.
Encryption/decryption happens at the source/destination of data when
creating/extracting backups.
Therefore, there is no key kept at the host running lxbackup
.
To prepare a host to use lxbackup
, get the lxbackup
command, the configuration,
and the kvtar
command and copy them to the host.
You might want to copy also lxrestore
and lxbackups
.
These commands can be found at the installed $LXDIR/bin
directory on any installed host. If you are not sure regarding the
$LXDIR
value, use this command to find it:
unix$ lx -d pwd /usr/local/leanxcale unix$
Flag -d
for lx
makes it change to the $LXDIR
directory
before running the given command.
The detailed configuration file to be used is kept at $LXDIR/lib/lxinst.conf
.
The configuration can be retrieved using the
lx config
command.
For example, this creates ./lxinst.conf
with the detailed configuration:
unix$ lx config -o lxinst.conf saved lxinst.conf
As an example, we can setup an external host named
orion
to perform backups in this way:
unix$ lx -d pwd /usr/local/leanxcale unix$ lx config -o /tmp/lxinst.conf saved /tmp/lxinst.conf unix$ cd /usr/local/leanxcale/bin unix$ scp lxbackup lxbackups lxrestore kvtar /tmp/lxinst.conf orion:~
And then just:
orion$ lxbackup -f lxinst.conf
This is just an example. It is a good idea to copy those commands to your local bin directory on the external host.
20.6. lx backups
To list the backups known use the backups
command:
usage: backups [-h] [-v] [-D] [-n] [-d dir] [what [what ...]] list backups positional arguments: what [last|backupname] host/comps optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -d dir root backup dir
For example:
unix$ lx backups 240810 ts 1419000 24081001+ ts 1419000 ...
Those with a +
in their names are incremental backups.
Verbosity can be increased adding one or more -v
flags.
unix$ lx backups -v 240810 ts 1419000 240810/kvms100 ts 2551000 240810/kvds100 ts 1292000 240810/lxqe100 ts 1419000 240810/lxqe101 ts 1419000 24081001+ ts 0 24081002+ ts 1419000 24081002+/lxqe100 ts 1419000 ...
unix$ lx backups -vv 240810 ts 1419000 240810/kvms100 ts 2551000 240810/kvms100/kvmeta sz 239 ts 2551000 mt 1723285636 240810/kvds100 ts 1292000 240810/kvds100/dbf00001.kv sz 393216 ts 1419000 mt 1723285636 240810/kvds100/seqs sz 4 ts 0 mt 1723285599 240810/lxqe100 ts 1419000 ...
unix$ lx backups -vvv 240810 ts 1419000 240810/kvms100 ts 2551000 240810/kvms100/kvmeta sz 239 mt 1723285636 localhost ts 1 2551000 bck 0 0 na 0 240810/kvds100 ts 1292000 240810/kvds100/dbf00001.kv sz 393216 mt 1723285636 localhost ts 1292000 1419000 bck 0 0 na 3 tname db-APP-PERSONS rmin: tpmin rmax: tpmax ...
It is possible to list a single backup by suppling its name and/or
specific components using arguments as done with most lx
commands.
The name last
refers to the last backup made.
For example:
unix$ lx backups -v 24080901
or
unix$ lx backups -v 24080901 lxqe
20.7. lxbackups
This is the external version of lx backups
.
It is used exactly like lx backups
.
usage: lxbackups [-h] [-v] [-D] [-n] [-f cfgfile] [-d dir] [what [what ...]] list backups positional arguments: what [last|backupname] host/comps optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -f cfgfile config for the install -d dir root backup dir
Refer to lxbackup
for installation details.
20.8. lx copy
The lx copy
command can be used to copy disks from hosts, directories or
components to other hosts, directories or components.
usage: copy [-h] [-v] [-D] [-n] [-F] [-l] [what [what ...]] copy component disks positional arguments: what src... dst optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -F force command -l call from lx.cptree src/dst: repl|host|comp|path
The special source or target repl
can be used to copy to or from the replica
of a component to restore a mirror from another.
For example
unix$ lx copy kvds100 /tmp
copies the files for kvds100
to /tmp/kvds100
.
The same is achieved using
unix$ lx copy kvds100 /tmp/kvds100
As another example:
unix$ lx copy repl1 repl
copies the whole set of disks at replica-1 hosts into their mirrors.
The same thing can be achieved by copying one component at a time using
unix$ lx copy lxqe100.r1 repl
20.9. lx config
Inspect or update the configuration:
usage: config [-h] [-v] [-D] [-s value] [-d] [-n] [-o fname] [what [what ...]] Inspect the configuration positional arguments: what kvaddr|grafana|cons|[host] [comp|prop...] optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -s set property values -d delete -n dry run -o fname write the output to fname
By default, lx config
prints the whole configuration, or that for elements given as arguments.
The arguments follow the conventional syntax used by most commands, but knows property names also:
-
Giving a host name narrows the rest of the arguments to that host, until another host name is given
-
Giving a component kind (e.g.,
lxqe
) selects those components. -
Giving a component name with included id (e.g.,
lxqe101
) selects just that component -
Giving a property name (not a host and not a component) selects just that property.
For example,
unix$ lx config
prints all configuration.
unix$ lx config atlantis mariner
prints the configuration for hosts atlantis
and mariner
(assuming those are configured host names).
unix$ lx config atlantis kvms mariner lxqe
prints the configuration for kvms
components found at atlantis
and
lxqe
components found at mariner
unix$ lxconfig kvms kvds101
prints the configuration for any kvms
component and the kvds101
one.
unix$ lxconfig kvms addr
prints the addr
attribute for any kvms
component.
unix$ lxconfig awsdisk
prints the awsdisk
property from the (global) configuration.
Use flag -o
to save the configuration (or the parts selected) to the given
file.
Use flag -d
to remove the selected configuration entries.
Do not remove hosts or components, and use this with caution.
Use flag -s
to update the selected properties with new values.
In this case, the argument for a property includes both its name and the new
value (e.g., mem=50m
)
For example:
unix$ lx config -s lxqe mem=500m kvds mem=500m
20.10. lx ctl
Issue control requests to lx components
usage: ctl [-h] [-v] [-D] comp [ctl [ctl ...]] issue control requests positional arguments: comp component or command ctl ctl arg optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags comps: ds, ms (master), qe, mm, meta (all ms), all, compname commands: sync: sync the disk debug: add/remove/set debug flags stats: print statistics useful ctls: all: debug +flags|-flags|flags qe: pull start|stop|addrs
This command issues control requests to running processes. Use with care and do not issue a control request unless you know what you are doing.
The comp
argument selects the target for the control request or is a known command.
It can be any of:
-
a component name, like
lxqe100
-
a process name, like
qe100
-
a process address, like
orion!14440
-
all
, to address all processes taking control requests -
ms
orkvms
, to address the masterkvms
process -
meta
, to address allkvms
processes -
ds
orkvds
, to address allkvds
processes -
cli
, to ask for a control request to be forwarded to all runningkvms
clients. -
a specific command, like
sync
, in which case addressed components are implicit.
Different processes have different control requests.
In general, ?
can be used to learn the control requests supported.
For example,
unix$ lx ctl qe ? help [cmd] sync debug flags mdebug on/off dump conflicts|txns|cfg|ni print conflicts|txns|cfg|ni txdebug on|off up [txn|conn] down [txn|conn] run halt: disable txns abort qe pull [stop | start | qeaddr...] test crash|hup now|begin|commit|msgs|xfer|publish|end|rescue
This command calls kvcon
to actually issue control requests, after adjusting a bit
the target names as a convenience.
20.10.1. Debug Control Request
The debug
control request adds, clears, or sets debug flags.
This ctl is also understood as a direct command (addressing everyone).
For example, adding no extra debug flags can be used to see current debug flags set:
unix$ lx ctl debug + local: ms100: DMMOO ds100: DMMOO mm100: dDMMOO qe100: dDMMOO
For lxmeta
(mm
) and lxqe
(qe
) processes, lowercase debug flags
correspond to log4j
loggers and uppercase flags correspond to standard flags
used everywhere in the system.
Repeating a flag increases the debug level. Popular debug flags are:
-
D
: general debug -
O
: operations -
M
: metadata
Popular debug flags for java are also:
-
j
: lxjdbc debug. JDBC requests -
m
: lxmeta debug. Lxmeta processing. -
q
: lxqe debug. Query engine processing. -
n
: lxnet debug. Network operations. -
l
: lxlogger debug. DB Logger and records. -
s
: lxss debug. Snapshot server. -
c
: lxcm debug. Conflict manager. -
t
: lxtxns debug. Transactions. -
e
: lxenum debug. Enumerations.
20.10.2. Stats Control Request
To ask for statistics, use the stats
command:
unix$ lx ctl lxqe100 stats ... unix$ lx ctl stats ... unix$ lx ctl sync unix$ lx ctl debug +
20.10.3. Print or Dump Control Request
The dump
request asks a process to dump data structures (or some of them) to the
log.
The print
request is similar, but sends the output to the client issuing the request
(i.e., the output is shown by lx ctl
).
Without further arguments, the request may print the most important ones, or just remind its usage.
For query engines, these are the requests known:
unix$ lx ctl qe dump ? dump conflicts|txns|cfg|ni
They correspond to the conclict manager structures, the transactions, the configuration,
and the NetInfo
information.
For lxmeta
processes, we have:
unix$ lx ctl mm dump ? dump slots|cfg|ni
Here, slots
corresponds to timestamp slots.
For kvms
, the request has no arguments and dumps the metadata as known by that process.
For kvds
, the dump requests are:
unix$ lx ctl ds dump ? dump [all|files|fileblks|seq|proc|stack|tids [full|blks]|mem [kind]|reqs|fid [hdr]
And the print requests are:
unix$ lx ctl ds print ? print [files|seq|proc|stack|tids|mem [kind]|reqs
Without arguments, both requests print the table of server files used for tables.
An important request is proc
, that prints the process table for the server, and
another related one is stack
, which prints the stacks for the processes in the server.
20.11. lx dbdata
List database data files:
usage: dbdata [-h] [-v] [-D] [-l] [-a] [-n] [-r] [-t tbl] [-i idx] [-w prop] [what [what ...]] list database data files positional arguments: what path|cfg optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -a print all content -n dry run -r print rows -t tbl table name to list data files for -i idx idx name to list data files for -w prop set bck:ts
The command lists DB data files as found on disk.
For example:
unix$ lx dbdata blade110 kvds100 dbf00001.kv tree db-APP-PAYMENTS5 blade110 kvds100 dbf00003.kv deriv db-APP-PAYMENTS5_OA blade110 kvds100 dbf00005.kv tree db-APP-TBL2
Each line in the output lists the machine and component where the file is kept before the file information.
Flag -v
makes the program more verbose:
unix$ lx dbdata blade110 kvds100 dbf00001.kv tree db-APP-PAYMENTS5 1:2 0x8000000002 ts 596000 2692000 snapts 0 snapno 31 bckts 0 bckdts 0 0 syncs rmin: tpmin rmax: tpmax blade110 kvds100 dbf00003.kv deriv db-APP-PAYMENTS5_OA 3:2 0x18000000002 ts 596000 2692000 snapts 0 snapno 6 bckts 0 bckdts 0 0 syncs rmin: tpmin rmax: tpmax deriv from db-APP-PAYMENTS5 blade110 kvds100 dbf00005.kv tree db-APP-TBL2 5:2 0x28000000002 ts 2796000 2796000 snapts 0 snapno 4 bckts 0 bckdts 0 0 syncs rmin: tpmin rmax: tpmax
The information listed, and the table and index names, is taken from the on-disk files, as known by the data servers.
For example, db-APP-PAYMENTS5
refers to database db
, schema APP
, and
table PAYMENTS5
.
To ask for a particular table, use flag -t
and give the table name as an
argument in the format just described:
unix$ lx dbdata -t db-APP-PAYMENTS5
The region minimum and maximum values are listed also as known by the data
servers (i.e., by kvds
) and not by the user or the query engines.
Flag -r
prints the rows in the files too.
They are printed in the stored field order, not in the user SQL table field order.
unix$ lx dbdata -rt db-APP-PAYMENTS5 blade110 kvds100 dbf00001.kv tree db-APP-PAYMENTS5 t1236000 - 10| 1 t1244000 - 11| 1 t1254000 - 21| 3 ...
It is possible to list only for particular hosts or components by supplying the usual names as arguments.
For example:
unix$ lx dbdata blade110
or
unix$ lx dbdata kvds100
Flag -w
is meant to write properties for the data files.
For now, only the backup TS as stored in the file. Use a in:
unix$ lx dbdata -w bck:777 kvds100
Where the number after bck:
is the backup TS to write.
20.12. lx dblog
Print the transaction log contents:
usage: dblog [-h] [-v] [-D] [-l] [-i] [-f] [-r] [-a] [-n] [-m] [-c count] [-s ts] [-w prop] [-e ts] [what [what ...]] print database log files positional arguments: what host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -i log header only -f log file information only -r log records only -a print all record content -n options -s and -e refer to record nbs. -m print the mirror log -c count print at most count records -s ts start from this ts -w prop set bck:ts or prop -e ts end at this ts
The command prints the log contents for the given names. Names can be host and or component names to select the addressed components, or a log directory or a logger address.
By default the command prints the log header and the records in it.
Under flag -i
, only the information in the log header is printed:
unix$ lx dblog -i logs... blade105: [ hdr vers1 sealed 18 records 0:18 2831 bytes 1113002:1358002 ts 1 meta records 12 commit records 12 xfer records ]
Under flag -r
, only the records are printed.
For data records, only the first line of the recorded message is printed.
Use flag -a
to print the full record contents (eg, all tuples recorded).
Flags -s
and/or -e
may be used to focus the search on a timestamp interval, or,
if flag -n
is given, on a record number interval.
For example, print records with numbers from 10 to 14 for lxqe100
:
unix$ lx dblog -r -n -s 10 -e 15 lxqe100 logs... blade105: [ commit[10] t1311002 txn.0x14011a r1 boff 1370 xfer[11] t1311002 txn.0x14011a r1 boff 1759 data[12] t1348002 db-APP-PERSONS txn.0x1491a2 r1 boff 1808 Tadd tag 26 sz 286 flg 0x40 0x1491a2 0x0 0x1491a2 <db-APP-PERSONS> 0x0 0x1485e9 [0]<> [0]<> tpzero [2]<0509> tpzero tpzero commit[13] t1348002 txn.0x1491a2 r1 boff 1857 xfer[14] t1348002 txn.0x1491a2 r1 boff 2246 ]
Flag -w
should be used with care.
If writes the given flag to the log file headers on disk.
To update the backup timestamp to 1234
, for example, use:
unix$ lx dblog -w bck:1234
The timestamp 0
means the last one kept in the log.
Other arguments for -w
are rescued
, single
, couple
, and mirror status strings.
20.13. lx dbmeta
Print DB metadata as found on disk.
usage: dbmeta [-h] [-v] [-D] [-l] [-a] [-n] [-w prop] [-f [file]] [what [what ...]] print database meta files positional arguments: what path|cfg optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -a print all content -n dry run -w prop write flag -f [file] meta file
This command prints the DB metadata as kept on disk.
For example, on a replicated install:
unix$ lx dbmeta dbmeta... blade110: # kvms100.r1 meta ts 0 kvms blade110!14400 usr: none usr: lxadmin blade161: # kvms100.r2 meta ts 0 kvms blade161!14400 usr: none usr: lxadmin
The command accept a kv metadata path to list, similar to kvcon
list
arguments.
For example, to list server metadata as kept on disk:
unix$ lx dbmeta /srv dbmeta... blade110: # kvms100.r1 kvms blade110!14400 kvms blade161!14400 kvds ds100.r1 at blade110!14500 lxmeta mm100.r2 at blade161!14410 kvds ds100.r2 at blade161!14500 lxmeta mm100.r1 at blade110!14410 lxqe qe100.r1 at blade110!14420 lxqe qe100.r2 at blade161!14420
Flag -w
can be used to set or clear metadata server flags.
For example:
unix$ lx dbmeta -w single /srv/ds100.r1
sets the single flag for the named server, and
unix$ lx dbmeta -w nosingle /srv/ds100.r1
clears the flag.
After restoring a replica, the fail
flag for the restored component
is usually cleared, and the single
flag for the component used as the
restore source is cleared as well.
As another example, this lists the metadata information for a table:
unix$ lx dbmeta /db/APP/tbl/INDEX2T0001
Using flag -a
reports all information:
unix$ lx dbmeta -a /db/APP/tbl/INDEX2T0001 owner 'lxadmin' fmt#0x0: 16 flds 1 key 16 usr 15 delta 0 sflds 0 old ... more output removed ... 2 regs 2 repls reg [ds101:69:2] min: tpmin max: tpmax reg [ds100:69:2] min: tpmin max: tpmax
20.14. lx disable
Disable transactions.
usage: disable [qe|qename]
For example,
unix$ lx disable
disables transactions for just the entire system, until the they are enabled.
While transactions are disabled, lx status
shows the system
status as waiting
(for transactions to become available).
Before using this command,
stop the watcher if the system was starting with flag -w
or
lx watch
is running.
Otherwise, because new JDBC connections are not accepted while transactions are disabled, the watcher might consider that the system is not in good health, and then restart the system.
20.15. lx enable
Enable transactions.
usage: enable [qe|qename]
For example,
unix$ lx enable qe100
enables transactions for just the qe100
query engine.
20.16. lx fmt
Format the whole DB or the indicated hosts or components:
usage: fmt [-h] [-v] [-D] [-l] [-c] [what [what ...]] fmt the store positional arguments: what host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -c clean
For example, format the kvds101 disk:
unix$ lx fmt kvds101
Flag -c
removes the logs too.
20.17. lx heap
Dump java process heaps and heap histograms.
usage: heap [-h] [-v] [-D] [-l] [-d] [-s] [-c] [-r] [-o] [-n num] [what [what ...]] get heap dumps and histograms positional arguments: what host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -d save a heap dump -s save a heap histogram -c cmp with baseline heap histogram -r remove saved dump and histogram -o include non-live objects in histogram -n num number of lines
The command understands the conventional syntax to select hosts and/or components. For example, save a base histogram to compare later and check out for leaks we can:
unix$ lx heap -s lxqe heaps... blade105 [ saved log/lxqe100.r1.hist ] blade110 [ saved log/lxqe100.r2.hist ]
Later, we can compare with another histogram using
lx heap -c lxqe heaps... blade105 [ saved log/lxqe100.r1.hist2 lxqe100.r1: ] blade110 [ saved log/lxqe100.r2.hist2 lxqe100.r2: [B: 9919 objs 674512 bytes java.lang.String: 9696 objs 232704 bytes java.lang.Class: 1779 objs 212480 bytes [I: 923 objs 157680 bytes ]
To print the current histogram (top 50 lines by default), run it without arguments:
lx heap -c lxqe heaps... blade105 [ lxqe100.r1: num #instances #bytes class name (module) ------------------------------------------------------- 1: 29587 7329264 [B (java.base@11.0.22) 2: 2905 814536 [I (java.base@11.0.22) 3: 6303 756888 java.lang.Class (java.base@11.0.22) ...
20.18. lx help
Ask for help:
usage: help [cmd] Print usage information
Prints the list of known commands with quick usage information, or detailed usage information about the given command.
20.19. lx license
Checks the license status or installs a new license:
usage: license [-h] [-v] [-D] [-f file] inspect or update license files. optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -f file license file to install
For example, ask for the current status:
unix$ lx license license expires: Mon Dec 30 00:00:00 2024
or install a new file lxlicense
with the desired license:
unix$ lx license lxlicense
20.20. lx logs
List and inspect logs for installed components:
usage: logs [-h] [-D] [-g rexp] [-a] [-p] [-c dst] [-s start] [-e end] [what ...] list logs positional arguments: what host|comp... options: -h, --help show this help message and exit -D enable debug diags -g rexp grep rexp -a all logs, not the last one -p print the last log (all if -a) -c dst copy the last logs to this dir (all if -a) -s start start fname time (yymmdd.hhmm or prefix) -e end end fname time (yymmdd.hhmm or prefix)
For example, list the logs for kvds at atlantis, but only those after
the time 230518.0551
(yymmdd.hhmm, or a prefix of this, can be
used):
unix$ lx logs -s 230518.0551 atlantis kvds
Print the last log for kvds101:
unix$ lx logs -p kvds101
Grep all the logs for lines with fatal
:
unix$ lx logs -g fatal
Copy all (not just the last one) kvds101 logs to /tmp:
unix$ lx -a -c /tmp kvds101
20.21. lx move
The lx move
command can be used to move tables or regions from one data
server to another while the system is down.
usage: move [-h] [-v] [-D] [-n] [-F] [what [what ...]] move tables or regions positional arguments: what tbl [rid] dstds | srcds dstds optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -F force command
For example
unix$ lx move db-APP-PERSONS kvds100
moves the files for the named table to the given server. The table name uses the data server syntax, not the SQL syntax. This command is not meant for users.
20.22. lx procs
List the processes for the DB:
usage: procs [-h] [-v] [-D] [-p] [what ...] list processes positional arguments: what host|comp... options: -h, --help show this help message and exit -v verbose -D enable debug diags -p report ports in use
For example, list the processes at host blade123
:
unix$ lx procs blade123 procs... blade123 [ lxmeta100 pid 813750 alive running kvds100 pid 813734 alive running kvms100 pid 813729 alive kvds103 pid 813746 alive running kvds101 pid 813738 alive running spread pid 813725 alive kvds102 pid 813742 alive running lxqe100 pid 813773 alive running ]
Or to see the port ustage status:
unix$ lx procs -p blade123 blade123 [ spread 14444 busy kvms100 14400 busy lxmeta100 14410 idle lxqe100 14420 busy kvds100 14500 busy kvds101 14504 busy kvds102.14508 busy kvds103 14512 busy ]
See also flag -p
of lx status
.
20.23. lx recover
usage: recover [-h] [-v] [-D] [-n] [-d dir] [-F] [what [what ...]] recover replica disks positional arguments: what repl|host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -d dir root backup dir -F force command
The command lx recover
locates disks for failed replicas or replica
components and restored them from their mirrors.
Arguments given restrict the repairs to them.
For example, after replica failures, we can run
unix$ lx recover
To try to recover their disks before restarting.
Use flag -n
for a dry run to see what would be repaired before running it,
unless you are sure to recover failed replicas.
20.24. lx report
Report system status and debug information.
usage: report [-h] report system information for debugging positional arguments: what host|comp... options: -h, --help show this help message and exit -D enable debug diags
This program collects information from the system and builds an archive to be sent to support.
unix$ lx report report: lxreport.231009... version... procs... logs... stacks... stack lxmeta100... stack kvds103... stack kvms100... stack spread... stack kvds102... stack kvds100... stack kvds101... stack lxqe100... # send this file to support. -rw-rw-r-- 1 leandata leandata 54861 Oct 9 14:58 lxreport.231009.tgz
As printed by the command output, the resulting tar file should be sent to support.
The archive includes:
-
installed version numbers
-
underlying OS names are versions
-
complete disk usage for the installed systems
-
complete process list for the installed systems
-
memory usage for the installed systems
-
lx process list
-
logs for components (last log file only, for each one)
-
stacks for each component
-
stacks for each core file found
When kvms
is still running, the archive includes also:
-
statistics for the sytem
-
long list of kv resources
-
process list for each kvds
-
file list for each kvds
20.25. lx restore
To restore a backup, use the restore
command.
usage: restore [-h] [-v] [-D] [-n] [-d dir] [-F] [what [what ...]] restore a backup positional arguments: what last|backupname [repl|host|comp...] optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -d dir root backup dir -F force command
For example:
unix$ lx restore #disk... check... restore 24080904... restore atlantis disk/kvds100 from /usr/local/leanxcale/dump/24080904/kvds100 crypt dbf00001.kv dbf00003.kv ...
Do this while the system is stopped. By default, it selects the last backup made.
To restore a particular backup, supply its name:
unix$ lx restore 24080904
This can be done also for incremental backups. When restoring an incremental backup, the restore takes data also from previous incremental backups and the corresponding total backup.
To restore only specific hosts or components, supply their names or types as done for other commands:
unix$ lx restore lxqe
Or perhaps:
unix$ lx restore 24080904 lxqe
To restore a backup, it is usually desirable to format the disk for the
involved components before using restore
to restore their disks.
To remove a backup, it suffices to remove its directory. For example:
unix$ lx -d rm -rf dump/230720
Here we used the flag “-d” for lx
to change to $LXDIR
before
executing the remove command, which makes it easy to name the directory
used for the dump.
When restoring encrypted backups, the disk
command knows it has to decrypt,
and it uses the key at $LXDIR/lib/lxkey.pem
, the one used when making the backup.
If you did reinstall, update the key so it matches the backup key used.
For external backups, use lxrestore
instead.
20.26. lx run
Run a command on the selected installed hosts, using the lx
environment to run it:
usage: run [-h] [-v] [-D] [-d] ... run a command on the installed host(s). positional arguments: hosts host|comp... cmd... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -d cd $LXDIR before running
Arguments specify hosts where to run (as usual in the rest of commands),
perhaps none (to imply all DB hosts), and then the keyword cmd
must
be given, followed by the command and arguments to run on each host.
For example, discover where this thing is installed, by running pwd
and using
flag -d
to set the current directory to the install directory on each host.
unix$ lx run -d cmd pwd
Or, as an exceptional measure, kill all processes running:
unix$ lx run cmd killprocs
Or, kill just those at blade123
:
unix$ lx run blade123 cmd killprocs
Or kill -9
any kvds at blade123
:
unix$ lx run blade123 cmd killprocs -9 kvds
20.27. lx stack
Dump process stacks
usage: stack [-h] [-v] [-D] [-l] [what [what ...]] print stacks positional arguments: what host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only
The command understands the conventional syntax to select hosts and/or components.
For example, to dump the stack of kvms
components:
unix$ lx stack kvms localhost [ stack kvms100: [ Thread 12 (Thread 0x7f8fc57fa700 (LWP 9272)): #0 __libc_read (nbytes=40, buf=0x7f8fa8000bd8, fd=13) at linux/read.c:26 #1 __libc_read (fd=13, buf=0x7f8fa8000bd8, nbytes=40) at linux/read.c:24 ... ] ]
With flag -v
, local variables are printed too.
For java processes, both the native stack and the java stacks are printed.
20.28. lx start
Starts the DB for operation:
usage: start [-h] [-v] [-D] [-l] [-w] [-k] [-r] [what [what ...]] start the service positional arguments: what host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -w start watch -k keep core files -r restart failed QEs
For example, to start the kvds components at host atlantis and leave the rest of the installation alone:
unix$ lx start atlantis kvds
With flag -w
, start will run the watch service. This pings the DB to
make sure it can answer queries, and, when that is not the case, try to
stop and restart the system. See the Section 20.32 section
for details.
By default, start will remove files on the installed directories
with names suggesting they are core dumps.
Flag -k
prevents this from happening and can be used to keep core dump files
for debugging.
Under flag -r
, start will ask any lxmeta
process started to try to restart
a failed QE that was running, unless it was restarted less than one minute ago.
20.29. lx status
Prints the status for the system or waits for a given status:
usage: status [-h] [-v] [-D] [-w status] [-t tout] [-r] [-p] show or wait for a DB status optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -w status wait for the given status -t tout timeout for -w (secs) -r print replica status -p print process status
For example, to learn the status:
unix$ lx status status: waiting kvds100: recovering files kvds101: recovering files
Or, to wait until the status is running
:
unix$ lx status -v -w running status: waiting kvds100: recovering files kvds101: recovering files status: running unix$
The status can be any of:
-
stopped: no process is running.
-
failed: some processes did fail.
-
waiting: processes are running but there is no SQL service.
-
running: processes are running and SQL connections are available.
It can be also running with failures when using replication and suffering failures.
Flag -r
reports the replica status:
unix$ lx status -r status: stopped replica: ok unix$
The replica status can be any of:
-
no: no replication used.
-
ok: replication used and state ok.
-
single: replication used and suffering single processes (mirror failures).
For example:
unix$ lx status -r status: stopped replica: no unix$
Flag -p
reports the process status for each known component
For example:
unix$ lx status -p kvds100 alone stopped kvms100 alone stopped lxmeta100 alone stopped lxqe100 alone stopped unix$
When using replicas, the preferred mirror process is listed first.
unix$ lx status -rp status: stopped replica: ok kvds100.r2 mirror stopped snap 890191999 kvds100.r1 mirror stopped snap 352036999 kvms100.r1 mirror stopped kvms100.r2 mirror stopped lxmeta100.r1 mirror stopped snap 890191999 lxmeta100.r2 mirror stopped lxqe100.r1 mirror stopped snap 889261999 lxqe100.r2 mirror stopped snap 889260999 unix$
The mirror status for each process can be any of:
-
mirror: replicated and its mirror is ok.
-
alone: not replicated.
-
single: did survive its mirror process.
-
outdated: did stop or fail before its mirror process.
-
conflict: did survive its mirror and its mirror thinks the same.
20.30. lx stop
Halts the DB or stops individual hosts or components:
usage: stop [-h] [-v] [-D] [-l] [-w] [-k] [what [what ...]] stop the service positional arguments: what host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -w do not stop watch -k kill -9
For example, halt the DB:
unix$ lx stop
Stop just the kvds servers:
unix$ lx stop kvds
If the watch service is running, make sure to stop it before stopping individual components. Otherwise it might decide to wait for lxmeta to stop and then restart the system.
Without arguments, stop will first stop the watch service and there is no extra caution needed.
Under flag -w
, stop will not stop the watch process.
Flag -k
may be used to kill components.
Do not use this unless you know what you are doing.
For example, to kill the lxqe100
unix$ lx stop -k lxqe100 stop... atlantis: [ stop: kill -9 lxqe100 pid 791944... ]
20.31. lx version
Print the installed version:
usage: version [-h] print installed version optional arguments: -h, --help show this help message and exit
The program reports the leanXcale version name, along with detailed version information for distribution packages installed:
unix$ lx version leanXcale v2.2 kv v2.2.2023-09-29.115f5fba70e3af8dc203953399088902c4534389 QE v2.2.2023-09-30.1e5933900582.26a7a5c3420cd3d5d589d1fa6cc libs v2.2.2023-09-29.67535752acf19e092a6eaf17b11ad17597897956 avatica v2.2.2023-09-27.0b0a786b36e8bc7381fb2bb01bc8b3ed56f49172 TM v2.2.2023-09-29.9a9b22cfdc9b924dbc3430e613cddab4ed667a57 lxlbins v2.2.2023-09-29.79e7e04fb16b38d08c2d5df1fe08e103d49cb22a lxinst v2.2.2023-10-02.b341e6545913aee8e0b0daf255362.273b33ea6d calcite v2.2.2023-09-27.d3dfcf24285d38add3f4e29a9c2e9eacbcd0b913 lxodata v2.2.2023-09-23.b84fa4c7d2ca3e778edd9de29389b2aa6e1a9fb8
20.32. lx watch
Watch out the system and restart it if needed:
usage: watch [-h] [-v] [-D] watch out the db and stop/start it if needed optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags
This program is started by lx start
when the flag -w
is given to
it. It is usually a bad idea to execute this command explicitly.
It waits until the DB is running, doing nothing until that point. From
that point on, if the DB ceases to be running, the program will wait for
lxmeta
to stop, and, then try to stop and start the whole system,
and finally exit.
When restarting the system, the flag -w
is used, to start a new
watcher for the new system.
The implications are that a failure to start will not restart the system more times, and that stopping individual components requires to stop this program first, or it might take actions on its own.
20.33. killprocs
This is a command for local use only:
usage: killprocs [-h] [-9] [-a] [procs] kill processes
Use lx run
to run it at any/all of the installed hosts.
Without any flag, locates the DB process pids looking at the files reporting them, and then kills them.
With flag -a
, locates any process in the system by process name (all
the ones started by the DB use bin/…
as a name), and all java
processes starting with LX
, and kills them.
A TERM signal is sent, and then a KILL signal after a few seconds.
Flag -9
can be used to send only a KILL signal.
If process/component names are given (eg, kvds
, or lxqe100
),
only those processes are killed.
For example, send a kill signal to any kvds at blade123:
unix$ lx run blade123 cmd killprocs -9 kvds
21. Notice
LeanXcale system uses spread as communication bus under the following license:
Version 1.0 June 26, 2001 Copyright (c) 1993-2016 Spread Concepts LLC. All rights reserved. This product uses software developed by Spread Concepts LLC for use in the Spread toolkit. For more information about Spread, see http://www.spread.org