Administration Reference Guide
1. Command Reference Manual
This section details all the commands available and how to use them, starting with the install command.
1.1. lxinst
This is the install program:
usage: lxinst [-h] [-v] [-D] [-s] [-m] [-n] [-f cfgfile] [-d dist] [-k key] [-K awspem] [-u location] [-c] [-i] [where ...] leanXcale installer positional arguments: where [aws|docker|stats] dir, host, or host:dir options: -h, --help show this help message and exit -v verbose -D enable debug diags -s small install -m medium install -n dry run -f cfgfile use this configuration file -d dist distrib file, dir, or url -k key key to download the distribution. -K awspem AWS pem file name w/o .pem -u location update inst at location -c clean. download everything -i ignore system limits
Given arguments specify where to install the DB. They may be paths for
directories in the local host, or host names, or host and directory
names using the syntax host:dir
.
Instead of arguments, a configuration file may be given using the -f
flag.
The configuration file has been described before in this document.
Flag -k
may be used to specify the key to download the distribution.
Flag '-K' must be used when installing on AWS to supply the path (without file name extension) to the pem/pub key files.
By default, the distribution is retrieved from the leanXcale
artifactory. However, it is possible to supply one or more times the
-d
flag giving it the name for a .tgz
file, a directory, or a
URL. The special argument leanxcale
stands for the official
repository for the distribution.
Distribution packages will be retrieved by looking at those distribution
sources. By convention, the first source is usally the ./lxdist
directory, used to keep the distribution when downloaded.
For example, this installs just what is downloaded at ./lxdist
,
without downloading anything else:
unix$ lxinst -d ./lxdist -f lxinst.conf
Once downloaded, files are not downloaded again. Flag ''-c'' cleans the ''./lxdist'' directory to force a download of everything. If a different directory is specified by the user, it is not cleaned, although packages not found will be still downloaded there. To force a download of particular packages, remove the packages desired from the ''./lxdist'' directory (or the directory specified in the arguments).
For example, this downloads a fresh copy of the distribution and installs it:
unix$ lxinst -c -f lxinst.conf
And this tries first the ../lxdist
directory and then the standard
leanxcale repository:
unix$ lxinst -d ./lxdist -d leanxcale -f lxinst.conf
This is actually the default when no distribution source is specified.
Also, if no directory source is specified as a first option, lxdist
is used as a directory to download components (other than files found on
the local host).
The database has users (different from UNIX users). The user
lxadmin
is the administrator for the whole system. During the
install you will be asked to type a password for lxadmin
. To specify
the password without being asked, you can set the password in the
LXPASS
environment variable:
unix$ export LXKEY=nemo:APA...uQy unix$ export LXPASS=fds92f3c unix$ lxinst /usr/local/leanxcale ...
When the users file ./lxowner
exists, that file is used as the file
describing the lxadmin
user and its secret, instead of creating one
using the user supplied password for lxadmin
.
Do not use a existing file unless you know what you are doing. The
reason is that this file must define lxadmin
as the first user and
that such user must have access to all resources, as configured.
Flag -i
makes lxinst
ignore system limits, to install on places with
reduced disk or memory and in a hurry.
By default, the install performs a large install, trying to use all the
machine resources for the service.
Flag -s
selects a small instead, and flag -m
selects a medium install.
The install size affects components and sizes not specified by the user in
command line arguments or in the configuration file.
In a small install, at most 4 kdvs
are added per host, and component
memory is limited to 1GiB.
In a medium install, at most 8 kvds
components are added, and
component memory is left with default values.
Flag -u
is used to update a previous install.
Still WIP, do not use it for now.
1.2. lx
Shell for running LeanXcale control programs. This simply fixes the environment for the installed host and runs the command given as an argument:
usage: lx [-d] cmd...
When flag -d
is used, the current working directory is set to the
install directory at the host before running the given command.
Most commands follow the same conventions regarding options and arguments. We describe them here for convenience:
positional arguments: what host|comp... options: -h, --help show this help message and exit -l local run only -D enable debug diags
Flag -l
is used when executing the command locally. This is used by
the lx
command framework, and should not be used in general by the end
user.
Flag -D
enables verbose diagnostics
Arguments specify what to operate (e.g., what to start, stop, etc.) may be empty to rely on the defaults (whole DB) or may specify a particular host and/or component name:
-
when only component names are given, and only those components will be involved (e.g.,
lxqe101
). -
when a component type name is given, components for that type are selected. (e.g.,
lxqe
). -
when a host name is given, any component following is narrowed to that host. If no components follow the host name, all components from the host are selected.
This may be repeated to specify different hosts and/or components.
The special host names db
, repl
, and repl2
may be used and
stand for hosts without the nodb
attribute, hosts for the first
replica, and hosts for the second replica (hosts that are a mirror of
other ones).
1.3. lx add
Add the given files to the named dir at the installed locations.
usage: add [-h] [-v] [-D] [-a host] ... add files to dir on installed lx positional arguments: files files... dir optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -a host copy to this installed host(s)
For example,
unix$ lx add ./fwd ubin
adds the forwarding command found at ./fwd
to the ubin
directory on
the installed locations.
Directory names should be one of ubin
, ulib
, or tmp
.
Using bin
or lib
is dangerous, you have been warned.
1.4. lx addlib
Add the given libraries or package file(s) to the installed locations.
usage: addlib [-h] [-l] [-v] [-D] [-a host] [-x] [files [files ...]] add libs to installed lx positional arguments: files files to add optional arguments: -h, --help show this help message and exit -l local run only -v verbose -D enable debug diags -a host copy to this installed host(s) -x internal use only
If the files added are distribution packages, they are unpacked on top of each one of the installed locations.
1.5. lx backup
usage: backup [-h] [-v] [-D] [-n] [-d dir] [-F] [-i] [-e] [-r] [-o] [what [what ...]] make a backup positional arguments: what host/comps optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -d dir root backup dir -F force command -i incremental backups -e encrypt -r restore -o online backup
By default, backups are kept at $LXDIR/dump
at the host used to issue
backup commands. The convention is to use the first host to keep
backups.
That is for internal backups (backups at a installed host).
For external backups, lxbackup
is used at the host
keeping the backup, and
the backup top-level
directory is given with flag -d
or defaults to ./lxdump
otherwise.
Directories under dump/
are named using the date as their name.
When more than one backup is made on the same date, further digits are added
to number succesive backups.
A sorted list for the directory reports backups from oldest to newest.
For incremental backups, a final +
is added to the directory name.
To create a full, cold, backup when leanXcale is not running:
unix$ lx backup #backup... new 240809 make atlantis disk/kvms100 /usr/local/leanxcale/dump/240809/kvms100 make atlantis disk/kvds100 /usr/local/leanxcale/dump/240809/kvds100 make atlantis disk/kvds101 /usr/local/leanxcale/dump/240809/kvds101 make atlantis disk/lxqe100/log /usr/local/leanxcale/dump/240809/lxqe100 unix$
The printed name is the name for the directory keeping the backup, as
used when restoring it. In this case, 240809
.
The files kept at the backup are compressed, and must uncompressed if copied by hand. The restore command takes care of this.
Using flag -e
both encrypts and compresses the backup files.
Flags belong to disk
, note that backup
is the argument given to disk
.
The key used to encrypt the backed files is kept at $LXDIR/lib/lxkey.pem
on the
installed sites as set up by the installer.
unix$ lx backup -e #backup... new 24080901 make atlantis disk/kvms100 /usr/local/leanxcale/dump/24080901/kvms100 crypt make atlantis disk/kvds100 /usr/local/leanxcale/dump/24080901/kvds100 crypt make atlantis disk/kvds101 /usr/local/leanxcale/dump/24080901/kvds101 crypt make atlantis disk/lxqe100/log /usr/local/leanxcale/dump/24080901/lxqe100 crypt
To create an incremental backup use the flag -i
.
unix$ lx backup -i #backup... new 24080902+ incr: lxqe100 += lxqe100
The output reports which components got files backed up.
The incremental backup is encrypted if the total backup it refers to is
encrypted too. No flag -e
should be given.
The incremental backup can be performed while the system is running.
To create a backup while the system is running (known as an online backup), use
flag -o
for lx backup
.
unix$ lx backup -o #backup... new 24080902 ...
This is more expensive than doing a backup with the system down. The flag is required to prevent mistakes. If the system is down when an online backup is asked for, the backup is still made as a cold backup. That is, the flag says that it is Ok to backup while the system is online.
1.6. lxbackup
This is the external version of lx backup
, for use on external (non installed)
hosts to keep external backups.
usage: lxbackup [-h] [-v] [-D] [-n] [-f cfgfile] [-d dir] [-i] [-e] [-r] [-o] [-F] [what [what ...]] make a backup positional arguments: what host/comps optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -f cfgfile config for the install -d dir root backup dir -i incremental backups -e encrypt -r restore -o online backup -F force command
Using lxbackup
is exactly like using lx backup
with a few
differences:
-
Flag
-f
is mandatory on the first invocation and provides the installed configuration file. -
The default backup directory is not
$LXDIR/dump
, but./lxdump
. -
The command
kvtar
must be available at the host runninglxbackup
.
The external host must have ssh access to the installed hosts.
The configuration file used must be the one retrieved using lx config
, and
not the one written by the user to perform the install, because lx config
reports addresses and details needed by the command.
Once a backup has been created, the configuration used is saved along with the
backup data and there is no need to use -f
to supply it.
Encryption/decryption happens at the source/destination of data when
creating/extracting backups.
Therefore, there is no key kept at the host running lxbackup
.
To prepare a host to use lxbackup
, get the lxbackup
command, the configuration,
and the kvtar
command and copy them to the host.
You might want to copy also lxrestore
and lxbackups
.
These commands can be found at the installed $LXDIR/bin
directory on any installed host. If you are not sure regarding the
$LXDIR
value, use this command to find it:
unix$ lx -d pwd /usr/local/leanxcale unix$
Flag -d
for lx
makes it change to the $LXDIR
directory
before running the given command.
The detailed configuration file to be used is kept at $LXDIR/lib/lxinst.conf
.
The configuration can be retrieved using the
lx config
command.
For example, this creates ./lxinst.conf
with the detailed configuration:
unix$ lx config -o lxinst.conf saved lxinst.conf
As an example, we can setup an external host named
orion
to perform backups in this way:
unix$ lx -d pwd /usr/local/leanxcale unix$ lx config -o /tmp/lxinst.conf saved /tmp/lxinst.conf unix$ cd /usr/local/leanxcale/bin unix$ scp lxbackup lxbackups lxrestore kvtar /tmp/lxinst.conf orion:~
And then just:
orion$ lxbackup -f lxinst.conf
This is just an example. It is a good idea to copy those commands to your local bin directory on the external host.
1.7. lx backups
To list the backups known use the backups
command:
usage: backups [-h] [-v] [-D] [-n] [-d dir] [what [what ...]] list backups positional arguments: what [last|backupname] host/comps optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -d dir root backup dir
For example:
unix$ lx backups 240810 ts 1419000 24081001+ ts 1419000 ...
Those with a +
in their names are incremental backups.
Verbosity can be increased adding one or more -v
flags.
unix$ lx backups -v 240810 ts 1419000 240810/kvms100 ts 2551000 240810/kvds100 ts 1292000 240810/lxqe100 ts 1419000 240810/lxqe101 ts 1419000 24081001+ ts 0 24081002+ ts 1419000 24081002+/lxqe100 ts 1419000 ...
unix$ lx backups -vv 240810 ts 1419000 240810/kvms100 ts 2551000 240810/kvms100/kvmeta sz 239 ts 2551000 mt 1723285636 240810/kvds100 ts 1292000 240810/kvds100/dbf00001.kv sz 393216 ts 1419000 mt 1723285636 240810/kvds100/seqs sz 4 ts 0 mt 1723285599 240810/lxqe100 ts 1419000 ...
unix$ lx backups -vvv 240810 ts 1419000 240810/kvms100 ts 2551000 240810/kvms100/kvmeta sz 239 mt 1723285636 localhost ts 1 2551000 bck 0 0 na 0 240810/kvds100 ts 1292000 240810/kvds100/dbf00001.kv sz 393216 mt 1723285636 localhost ts 1292000 1419000 bck 0 0 na 3 tname db-APP-PERSONS rmin: tpmin rmax: tpmax ...
It is possible to list a single backup by suppling its name and/or
specific components using arguments as done with most lx
commands.
The name last
refers to the last backup made.
For example:
unix$ lx backups -v 24080901
or
unix$ lx backups -v 24080901 lxqe
1.8. lxbackups
This is the external version of lx backups
.
It is used exactly like lx backups
.
usage: lxbackups [-h] [-v] [-D] [-n] [-f cfgfile] [-d dir] [what [what ...]] list backups positional arguments: what [last|backupname] host/comps optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -f cfgfile config for the install -d dir root backup dir
Refer to lxbackup
for installation details.
1.9. lx copy
The lx copy
command can be used to copy disks from hosts, directories or
components to other hosts, directories or components.
usage: copy [-h] [-v] [-D] [-n] [-F] [-l] [what [what ...]] copy component disks positional arguments: what src... dst optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -F force command -l call from lx.cptree src/dst: repl|host|comp|path
The special source or target repl
can be used to copy to or from the replica
of a component to restore a mirror from another.
For example
unix$ lx copy kvds100 /tmp
copies the files for kvds100
to /tmp/kvds100
.
The same is achieved using
unix$ lx copy kvds100 /tmp/kvds100
As another example:
unix$ lx copy repl1 repl
copies the whole set of disks at replica-1 hosts into their mirrors.
The same thing can be achieved by copying one component at a time using
unix$ lx copy lxqe100.r1 repl
1.10. lx config
Inspect or update the configuration:
usage: config [-h] [-v] [-D] [-s value] [-d] [-n] [-o fname] [what [what ...]] Inspect the configuration positional arguments: what kvaddr|grafana|cons|[host] [comp|prop...] optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -s set property values -d delete -n dry run -o fname write the output to fname
By default, lx config
prints the whole configuration, or that for elements given as arguments.
The arguments follow the conventional syntax used by most commands, but knows property names also:
-
Giving a host name narrows the rest of the arguments to that host, until another host name is given
-
Giving a component kind (e.g.,
lxqe
) selects those components. -
Giving a component name with included id (e.g.,
lxqe101
) selects just that component -
Giving a property name (not a host and not a component) selects just that property.
For example,
unix$ lx config
prints all configuration.
unix$ lx config atlantis mariner
prints the configuration for hosts atlantis
and mariner
(assuming those are configured host names).
unix$ lx config atlantis kvms mariner lxqe
prints the configuration for kvms
components found at atlantis
and
lxqe
components found at mariner
unix$ lxconfig kvms kvds101
prints the configuration for any kvms
component and the kvds101
one.
unix$ lxconfig kvms addr
prints the addr
attribute for any kvms
component.
unix$ lxconfig awsdisk
prints the awsdisk
property from the (global) configuration.
Use flag -o
to save the configuration (or the parts selected) to the given
file.
Use flag -d
to remove the selected configuration entries.
Do not remove hosts or components, and use this with caution.
Use flag -s
to update the selected properties with new values.
In this case, the argument for a property includes both its name and the new
value (e.g., mem=50m
)
For example:
unix$ lx config -s lxqe mem=500m kvds mem=500m
1.11. lx ctl
Issue control requests to lx components
usage: ctl [-h] [-v] [-D] comp [ctl [ctl ...]] issue control requests positional arguments: comp component or command ctl ctl arg optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags comps: ds, ms (master), qe, mm, meta (all ms), all, compname commands: sync: sync the disk debug: add/remove/set debug flags stats: print statistics useful ctls: all: debug +flags|-flags|flags qe: pull start|stop|addrs
This command issues control requests to running processes. Use with care and do not issue a control request unless you know what you are doing.
The comp
argument selects the target for the control request or is a known command.
It can be any of:
-
a component name, like
lxqe100
-
a process name, like
qe100
-
a process address, like
orion!14440
-
all
, to address all processes taking control requests -
ms
orkvms
, to address the masterkvms
process -
meta
, to address allkvms
processes -
ds
orkvds
, to address allkvds
processes -
cli
, to ask for a control request to be forwarded to all runningkvms
clients. -
a specific command, like
sync
, in which case addressed components are implicit.
Different processes have different control requests.
In general, ?
can be used to learn the control requests supported.
For example,
unix$ lx ctl qe ? help [cmd] sync debug flags mdebug on/off dump conflicts|txns|cfg|ni print conflicts|txns|cfg|ni txdebug on|off up [txn|conn] down [txn|conn] run halt: disable txns abort qe pull [stop | start | qeaddr...] test crash|hup now|begin|commit|msgs|xfer|publish|end|rescue
This command calls kvcon
to actually issue control requests, after adjusting a bit
the target names as a convenience.
1.11.1. Debug Control Request
The debug
control request adds, clears, or sets debug flags.
This ctl is also understood as a direct command (addressing everyone).
For example, adding no extra debug flags can be used to see current debug flags set:
unix$ lx ctl debug + local: ms100: DMMOO ds100: DMMOO mm100: dDMMOO qe100: dDMMOO
For lxmeta
(mm
) and lxqe
(qe
) processes, lowercase debug flags
correspond to log4j
loggers and uppercase flags correspond to standard flags
used everywhere in the system.
Repeating a flag increases the debug level. Popular debug flags are:
-
D
: general debug -
O
: operations -
M
: metadata
Popular debug flags for java are also:
-
j
: lxjdbc debug. JDBC requests -
m
: lxmeta debug. Lxmeta processing. -
q
: lxqe debug. Query engine processing. -
n
: lxnet debug. Network operations. -
l
: lxlogger debug. DB Logger and records. -
s
: lxss debug. Snapshot server. -
c
: lxcm debug. Conflict manager. -
t
: lxtxns debug. Transactions. -
e
: lxenum debug. Enumerations.
1.11.2. Stats Control Request
To ask for statistics, use the stats
command:
unix$ lx ctl lxqe100 stats ... unix$ lx ctl stats ... unix$ lx ctl sync unix$ lx ctl debug +
1.11.3. Print or Dump Control Request
The dump
request asks a process to dump data structures (or some of them) to the
log.
The print
request is similar, but sends the output to the client issuing the request
(i.e., the output is shown by lx ctl
).
Without further arguments, the request may print the most important ones, or just remind its usage.
For query engines, these are the requests known:
unix$ lx ctl qe dump ? dump conflicts|txns|cfg|ni
They correspond to the conclict manager structures, the transactions, the configuration,
and the NetInfo
information.
For lxmeta
processes, we have:
unix$ lx ctl mm dump ? dump slots|cfg|ni
Here, slots
corresponds to timestamp slots.
For kvms
, the request has no arguments and dumps the metadata as known by that process.
For kvds
, the dump requests are:
unix$ lx ctl ds dump ? dump [all|files|fileblks|seq|proc|stack|tids [full|blks]|mem [kind]|reqs|fid [hdr]
And the print requests are:
unix$ lx ctl ds print ? print [files|seq|proc|stack|tids|mem [kind]|reqs
Without arguments, both requests print the table of server files used for tables.
An important request is proc
, that prints the process table for the server, and
another related one is stack
, which prints the stacks for the processes in the server.
1.12. lx dbdata
List database data files:
usage: dbdata [-h] [-v] [-D] [-l] [-a] [-n] [-r] [-t tbl] [-i idx] [-w prop] [what [what ...]] list database data files positional arguments: what path|cfg optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -a print all content -n dry run -r print rows -t tbl table name to list data files for -i idx idx name to list data files for -w prop set bck:ts
The command lists DB data files as found on disk.
For example:
unix$ lx dbdata blade110 kvds100 dbf00001.kv tree db-APP-PAYMENTS5 blade110 kvds100 dbf00003.kv deriv db-APP-PAYMENTS5_OA blade110 kvds100 dbf00005.kv tree db-APP-TBL2
Each line in the output lists the machine and component where the file is kept before the file information.
Flag -v
makes the program more verbose:
unix$ lx dbdata blade110 kvds100 dbf00001.kv tree db-APP-PAYMENTS5 1:2 0x8000000002 ts 596000 2692000 snapts 0 snapno 31 bckts 0 bckdts 0 0 syncs rmin: tpmin rmax: tpmax blade110 kvds100 dbf00003.kv deriv db-APP-PAYMENTS5_OA 3:2 0x18000000002 ts 596000 2692000 snapts 0 snapno 6 bckts 0 bckdts 0 0 syncs rmin: tpmin rmax: tpmax deriv from db-APP-PAYMENTS5 blade110 kvds100 dbf00005.kv tree db-APP-TBL2 5:2 0x28000000002 ts 2796000 2796000 snapts 0 snapno 4 bckts 0 bckdts 0 0 syncs rmin: tpmin rmax: tpmax
The information listed, and the table and index names, is taken from the on-disk files, as known by the data servers.
For example, db-APP-PAYMENTS5
refers to database db
, schema APP
, and
table PAYMENTS5
.
To ask for a particular table, use flag -t
and give the table name as an
argument in the format just described:
unix$ lx dbdata -t db-APP-PAYMENTS5
The region minimum and maximum values are listed also as known by the data
servers (i.e., by kvds
) and not by the user or the query engines.
Flag -r
prints the rows in the files too.
They are printed in the stored field order, not in the user SQL table field order.
unix$ lx dbdata -rt db-APP-PAYMENTS5 blade110 kvds100 dbf00001.kv tree db-APP-PAYMENTS5 t1236000 - 10| 1 t1244000 - 11| 1 t1254000 - 21| 3 ...
It is possible to list only for particular hosts or components by supplying the usual names as arguments.
For example:
unix$ lx dbdata blade110
or
unix$ lx dbdata kvds100
Flag -w
is meant to write properties for the data files.
For now, only the backup TS as stored in the file. Use a in:
unix$ lx dbdata -w bck:777 kvds100
Where the number after bck:
is the backup TS to write.
1.13. lx dblog
Print the transaction log contents:
usage: dblog [-h] [-v] [-D] [-l] [-i] [-f] [-r] [-a] [-n] [-m] [-c count] [-s ts] [-w prop] [-e ts] [what [what ...]] print database log files positional arguments: what host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -i log header only -f log file information only -r log records only -a print all record content -n options -s and -e refer to record nbs. -m print the mirror log -c count print at most count records -s ts start from this ts -w prop set bck:ts or prop -e ts end at this ts
The command prints the log contents for the given names. Names can be host and or component names to select the addressed components, or a log directory or a logger address.
By default the command prints the log header and the records in it.
Under flag -i
, only the information in the log header is printed:
unix$ lx dblog -i logs... blade105: [ hdr vers1 sealed 18 records 0:18 2831 bytes 1113002:1358002 ts 1 meta records 12 commit records 12 xfer records ]
Under flag -r
, only the records are printed.
For data records, only the first line of the recorded message is printed.
Use flag -a
to print the full record contents (eg, all tuples recorded).
Flags -s
and/or -e
may be used to focus the search on a timestamp interval, or,
if flag -n
is given, on a record number interval.
For example, print records with numbers from 10 to 14 for lxqe100
:
unix$ lx dblog -r -n -s 10 -e 15 lxqe100 logs... blade105: [ commit[10] t1311002 txn.0x14011a r1 boff 1370 xfer[11] t1311002 txn.0x14011a r1 boff 1759 data[12] t1348002 db-APP-PERSONS txn.0x1491a2 r1 boff 1808 Tadd tag 26 sz 286 flg 0x40 0x1491a2 0x0 0x1491a2 <db-APP-PERSONS> 0x0 0x1485e9 [0]<> [0]<> tpzero [2]<0509> tpzero tpzero commit[13] t1348002 txn.0x1491a2 r1 boff 1857 xfer[14] t1348002 txn.0x1491a2 r1 boff 2246 ]
Flag -w
should be used with care.
If writes the given flag to the log file headers on disk.
To update the backup timestamp to 1234
, for example, use:
unix$ lx dblog -w bck:1234
The timestamp 0
means the last one kept in the log.
Other arguments for -w
are rescued
, single
, couple
, and mirror status strings.
1.14. lx dbmeta
Print DB metadata as found on disk.
usage: dbmeta [-h] [-v] [-D] [-l] [-a] [-n] [-w prop] [-f [file]] [what [what ...]] print database meta files positional arguments: what path|cfg optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -a print all content -n dry run -w prop write flag -f [file] meta file
This command prints the DB metadata as kept on disk.
For example, on a replicated install:
unix$ lx dbmeta dbmeta... blade110: # kvms100.r1 meta ts 0 kvms blade110!14400 usr: none usr: lxadmin blade161: # kvms100.r2 meta ts 0 kvms blade161!14400 usr: none usr: lxadmin
The command accept a kv metadata path to list, similar to kvcon
list
arguments.
For example, to list server metadata as kept on disk:
unix$ lx dbmeta /srv dbmeta... blade110: # kvms100.r1 kvms blade110!14400 kvms blade161!14400 kvds ds100.r1 at blade110!14500 lxmeta mm100.r2 at blade161!14410 kvds ds100.r2 at blade161!14500 lxmeta mm100.r1 at blade110!14410 lxqe qe100.r1 at blade110!14420 lxqe qe100.r2 at blade161!14420
Flag -w
can be used to set or clear metadata server flags.
For example:
unix$ lx dbmeta -w single /srv/ds100.r1
sets the single flag for the named server, and
unix$ lx dbmeta -w nosingle /srv/ds100.r1
clears the flag.
After restoring a replica, the fail
flag for the restored component
is usually cleared, and the single
flag for the component used as the
restore source is cleared as well.
As another example, this lists the metadata information for a table:
unix$ lx dbmeta /db/APP/tbl/INDEX2T0001
Using flag -a
reports all information:
unix$ lx dbmeta -a /db/APP/tbl/INDEX2T0001 owner 'lxadmin' fmt#0x0: 16 flds 1 key 16 usr 15 delta 0 sflds 0 old ... more output removed ... 2 regs 2 repls reg [ds101:69:2] min: tpmin max: tpmax reg [ds100:69:2] min: tpmin max: tpmax
1.15. lx disable
Disable transactions.
usage: disable [qe|qename]
For example,
unix$ lx disable
disables transactions for just the entire system, until the they are enabled.
While transactions are disabled, lx status
shows the system
status as waiting
(for transactions to become available).
Before using this command,
stop the watcher if the system was starting with flag -w
or
lx watch
is running.
Otherwise, because new JDBC connections are not accepted while transactions are disabled, the watcher might consider that the system is not in good health, and then restart the system.
1.16. lx enable
Enable transactions.
usage: enable [qe|qename]
For example,
unix$ lx enable qe100
enables transactions for just the qe100
query engine.
1.17. lx fmt
Format the whole DB or the indicated hosts or components:
usage: fmt [-h] [-v] [-D] [-l] [-c] [what [what ...]] fmt the store positional arguments: what host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -c clean
For example, format the kvds101 disk:
unix$ lx fmt kvds101
Flag -c
removes the logs too.
1.18. lx heap
Dump java process heaps and heap histograms.
usage: heap [-h] [-v] [-D] [-l] [-d] [-s] [-c] [-r] [-o] [-n num] [what [what ...]] get heap dumps and histograms positional arguments: what host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -d save a heap dump -s save a heap histogram -c cmp with baseline heap histogram -r remove saved dump and histogram -o include non-live objects in histogram -n num number of lines
The command understands the conventional syntax to select hosts and/or components. For example, save a base histogram to compare later and check out for leaks we can:
unix$ lx heap -s lxqe heaps... blade105 [ saved log/lxqe100.r1.hist ] blade110 [ saved log/lxqe100.r2.hist ]
Later, we can compare with another histogram using
lx heap -c lxqe heaps... blade105 [ saved log/lxqe100.r1.hist2 lxqe100.r1: ] blade110 [ saved log/lxqe100.r2.hist2 lxqe100.r2: [B: 9919 objs 674512 bytes java.lang.String: 9696 objs 232704 bytes java.lang.Class: 1779 objs 212480 bytes [I: 923 objs 157680 bytes ]
To print the current histogram (top 50 lines by default), run it without arguments:
lx heap -c lxqe heaps... blade105 [ lxqe100.r1: num #instances #bytes class name (module) ------------------------------------------------------- 1: 29587 7329264 [B (java.base@11.0.22) 2: 2905 814536 [I (java.base@11.0.22) 3: 6303 756888 java.lang.Class (java.base@11.0.22) ...
1.19. lx help
Ask for help:
usage: help [cmd] Print usage information
Prints the list of known commands with quick usage information, or detailed usage information about the given command.
1.20. lx license
Checks the license status or installs a new license:
usage: license [-h] [-v] [-D] [-f file] inspect or update license files. optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -f file license file to install
For example, ask for the current status:
unix$ lx license license expires: Mon Dec 30 00:00:00 2024
or install a new file lxlicense
with the desired license:
unix$ lx license lxlicense
1.21. lx logs
List and inspect logs for installed components:
usage: logs [-h] [-D] [-g rexp] [-a] [-p] [-c dst] [-s start] [-e end] [what ...] list logs positional arguments: what host|comp... options: -h, --help show this help message and exit -D enable debug diags -g rexp grep rexp -a all logs, not the last one -p print the last log (all if -a) -c dst copy the last logs to this dir (all if -a) -s start start fname time (yymmdd.hhmm or prefix) -e end end fname time (yymmdd.hhmm or prefix)
For example, list the logs for kvds at atlantis, but only those after
the time 230518.0551
(yymmdd.hhmm, or a prefix of this, can be
used):
unix$ lx logs -s 230518.0551 atlantis kvds
Print the last log for kvds101:
unix$ lx logs -p kvds101
Grep all the logs for lines with fatal
:
unix$ lx logs -g fatal
Copy all (not just the last one) kvds101 logs to /tmp:
unix$ lx -a -c /tmp kvds101
1.22. lx move
The lx move
command can be used to move tables or regions from one data
server to another while the system is down.
usage: move [-h] [-v] [-D] [-n] [-F] [what [what ...]] move tables or regions positional arguments: what tbl [rid] dstds | srcds dstds optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -F force command
For example
unix$ lx move db-APP-PERSONS kvds100
moves the files for the named table to the given server. The table name uses the data server syntax, not the SQL syntax. This command is not meant for users.
1.23. lx procs
List the processes for the DB:
usage: procs [-h] [-v] [-D] [-p] [what ...] list processes positional arguments: what host|comp... options: -h, --help show this help message and exit -v verbose -D enable debug diags -p report ports in use
For example, list the processes at host blade123
:
unix$ lx procs blade123 procs... blade123 [ lxmeta100 pid 813750 alive running kvds100 pid 813734 alive running kvms100 pid 813729 alive kvds103 pid 813746 alive running kvds101 pid 813738 alive running spread pid 813725 alive kvds102 pid 813742 alive running lxqe100 pid 813773 alive running ]
Or to see the port ustage status:
unix$ lx procs -p blade123 blade123 [ spread 14444 busy kvms100 14400 busy lxmeta100 14410 idle lxqe100 14420 busy kvds100 14500 busy kvds101 14504 busy kvds102.14508 busy kvds103 14512 busy ]
See also flag -p
of lx status
.
1.24. lx recover
usage: recover [-h] [-v] [-D] [-n] [-d dir] [-F] [what [what ...]] recover replica disks positional arguments: what repl|host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -d dir root backup dir -F force command
The command lx recover
locates disks for failed replicas or replica
components and restored them from their mirrors.
Arguments given restrict the repairs to them.
For example, after replica failures, we can run
unix$ lx recover
To try to recover their disks before restarting.
Use flag -n
for a dry run to see what would be repaired before running it,
unless you are sure to recover failed replicas.
1.25. lx report
Report system status and debug information.
usage: report [-h] report system information for debugging positional arguments: what host|comp... options: -h, --help show this help message and exit -D enable debug diags
This program collects information from the system and builds an archive to be sent to support.
unix$ lx report report: lxreport.231009... version... procs... logs... stacks... stack lxmeta100... stack kvds103... stack kvms100... stack spread... stack kvds102... stack kvds100... stack kvds101... stack lxqe100... # send this file to support. -rw-rw-r-- 1 leandata leandata 54861 Oct 9 14:58 lxreport.231009.tgz
As printed by the command output, the resulting tar file should be sent to support.
The archive includes:
-
installed version numbers
-
underlying OS names are versions
-
complete disk usage for the installed systems
-
complete process list for the installed systems
-
memory usage for the installed systems
-
lx process list
-
logs for components (last log file only, for each one)
-
stacks for each component
-
stacks for each core file found
When kvms
is still running, the archive includes also:
-
statistics for the sytem
-
long list of kv resources
-
process list for each kvds
-
file list for each kvds
1.26. lx restore
To restore a backup, use the restore
command.
usage: restore [-h] [-v] [-D] [-n] [-d dir] [-F] [what [what ...]] restore a backup positional arguments: what last|backupname [repl|host|comp...] optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -n dry run -d dir root backup dir -F force command
For example:
unix$ lx restore #disk... check... restore 24080904... restore atlantis disk/kvds100 from /usr/local/leanxcale/dump/24080904/kvds100 crypt dbf00001.kv dbf00003.kv ...
Do this while the system is stopped. By default, it selects the last backup made.
To restore a particular backup, supply its name:
unix$ lx restore 24080904
This can be done also for incremental backups. When restoring an incremental backup, the restore takes data also from previous incremental backups and the corresponding total backup.
To restore only specific hosts or components, supply their names or types as done for other commands:
unix$ lx restore lxqe
Or perhaps:
unix$ lx restore 24080904 lxqe
To restore a backup, it is usually desirable to format the disk for the
involved components before using restore
to restore their disks.
To remove a backup, it suffices to remove its directory. For example:
unix$ lx -d rm -rf dump/230720
Here we used the flag “-d” for lx
to change to $LXDIR
before
executing the remove command, which makes it easy to name the directory
used for the dump.
When restoring encrypted backups, the disk
command knows it has to decrypt,
and it uses the key at $LXDIR/lib/lxkey.pem
, the one used when making the backup.
If you did reinstall, update the key so it matches the backup key used.
For external backups, use lxrestore
instead.
1.27. lx run
Run a command on the selected installed hosts, using the lx
environment to run it:
usage: run [-h] [-v] [-D] [-d] ... run a command on the installed host(s). positional arguments: hosts host|comp... cmd... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -d cd $LXDIR before running
Arguments specify hosts where to run (as usual in the rest of commands),
perhaps none (to imply all DB hosts), and then the keyword cmd
must
be given, followed by the command and arguments to run on each host.
For example, discover where this thing is installed, by running pwd
and using
flag -d
to set the current directory to the install directory on each host.
unix$ lx run -d cmd pwd
Or, as an exceptional measure, kill all processes running:
unix$ lx run cmd killprocs
Or, kill just those at blade123
:
unix$ lx run blade123 cmd killprocs
Or kill -9
any kvds at blade123
:
unix$ lx run blade123 cmd killprocs -9 kvds
1.28. lx stack
Dump process stacks
usage: stack [-h] [-v] [-D] [-l] [what [what ...]] print stacks positional arguments: what host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only
The command understands the conventional syntax to select hosts and/or components.
For example, to dump the stack of kvms
components:
unix$ lx stack kvms localhost [ stack kvms100: [ Thread 12 (Thread 0x7f8fc57fa700 (LWP 9272)): #0 __libc_read (nbytes=40, buf=0x7f8fa8000bd8, fd=13) at linux/read.c:26 #1 __libc_read (fd=13, buf=0x7f8fa8000bd8, nbytes=40) at linux/read.c:24 ... ] ]
With flag -v
, local variables are printed too.
For java processes, both the native stack and the java stacks are printed.
1.29. lx start
Starts the DB for operation:
usage: start [-h] [-v] [-D] [-l] [-w] [-k] [-r] [what [what ...]] start the service positional arguments: what host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -w start watch -k keep core files -r restart failed QEs
For example, to start the kvds components at host atlantis and leave the rest of the installation alone:
unix$ lx start atlantis kvds
With flag -w
, start will run the watch service. This pings the DB to
make sure it can answer queries, and, when that is not the case, try to
stop and restart the system. See the Section 1.33 section
for details.
By default, start will remove files on the installed directories
with names suggesting they are core dumps.
Flag -k
prevents this from happening and can be used to keep core dump files
for debugging.
Under flag -r
, start will ask any lxmeta
process started to try to restart
a failed QE that was running, unless it was restarted less than one minute ago.
1.30. lx status
Prints the status for the system or waits for a given status:
usage: status [-h] [-v] [-D] [-w status] [-t tout] [-r] [-p] show or wait for a DB status optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -w status wait for the given status -t tout timeout for -w (secs) -r print replica status -p print process status
For example, to learn the status:
unix$ lx status status: waiting kvds100: recovering files kvds101: recovering files
Or, to wait until the status is running
:
unix$ lx status -v -w running status: waiting kvds100: recovering files kvds101: recovering files status: running unix$
The status can be any of:
-
stopped: no process is running.
-
failed: some processes did fail.
-
waiting: processes are running but there is no SQL service.
-
running: processes are running and SQL connections are available.
It can be also running with failures when using replication and suffering failures.
Flag -r
reports the replica status:
unix$ lx status -r status: stopped replica: ok unix$
The replica status can be any of:
-
no: no replication used.
-
ok: replication used and state ok.
-
single: replication used and suffering single processes (mirror failures).
For example:
unix$ lx status -r status: stopped replica: no unix$
Flag -p
reports the process status for each known component
For example:
unix$ lx status -p kvds100 alone stopped kvms100 alone stopped lxmeta100 alone stopped lxqe100 alone stopped unix$
When using replicas, the preferred mirror process is listed first.
unix$ lx status -rp status: stopped replica: ok kvds100.r2 mirror stopped snap 890191999 kvds100.r1 mirror stopped snap 352036999 kvms100.r1 mirror stopped kvms100.r2 mirror stopped lxmeta100.r1 mirror stopped snap 890191999 lxmeta100.r2 mirror stopped lxqe100.r1 mirror stopped snap 889261999 lxqe100.r2 mirror stopped snap 889260999 unix$
The mirror status for each process can be any of:
-
mirror: replicated and its mirror is ok.
-
alone: not replicated.
-
single: did survive its mirror process.
-
outdated: did stop or fail before its mirror process.
-
conflict: did survive its mirror and its mirror thinks the same.
1.31. lx stop
Halts the DB or stops individual hosts or components:
usage: stop [-h] [-v] [-D] [-l] [-w] [-k] [what [what ...]] stop the service positional arguments: what host|comp... optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags -l local run only -w do not stop watch -k kill -9
For example, halt the DB:
unix$ lx stop
Stop just the kvds servers:
unix$ lx stop kvds
If the watch service is running, make sure to stop it before stopping individual components. Otherwise it might decide to wait for lxmeta to stop and then restart the system.
Without arguments, stop will first stop the watch service and there is no extra caution needed.
Under flag -w
, stop will not stop the watch process.
Flag -k
may be used to kill components.
Do not use this unless you know what you are doing.
For example, to kill the lxqe100
unix$ lx stop -k lxqe100 stop... atlantis: [ stop: kill -9 lxqe100 pid 791944... ]
1.32. lx version
Print the installed version:
usage: version [-h] print installed version optional arguments: -h, --help show this help message and exit
The program reports the leanXcale version name, along with detailed version information for distribution packages installed:
unix$ lx version leanXcale v2.2 kv v2.2.2023-09-29.115f5fba70e3af8dc203953399088902c4534389 QE v2.2.2023-09-30.1e5933900582.26a7a5c3420cd3d5d589d1fa6cc libs v2.2.2023-09-29.67535752acf19e092a6eaf17b11ad17597897956 avatica v2.2.2023-09-27.0b0a786b36e8bc7381fb2bb01bc8b3ed56f49172 TM v2.2.2023-09-29.9a9b22cfdc9b924dbc3430e613cddab4ed667a57 lxlbins v2.2.2023-09-29.79e7e04fb16b38d08c2d5df1fe08e103d49cb22a lxinst v2.2.2023-10-02.b341e6545913aee8e0b0daf255362.273b33ea6d calcite v2.2.2023-09-27.d3dfcf24285d38add3f4e29a9c2e9eacbcd0b913 lxodata v2.2.2023-09-23.b84fa4c7d2ca3e778edd9de29389b2aa6e1a9fb8
1.33. lx watch
Watch out the system and restart it if needed:
usage: watch [-h] [-v] [-D] watch out the db and stop/start it if needed optional arguments: -h, --help show this help message and exit -v verbose -D enable debug diags
This program is started by lx start
when the flag -w
is given to
it. It is usually a bad idea to execute this command explicitly.
It waits until the DB is running, doing nothing until that point. From
that point on, if the DB ceases to be running, the program will wait for
lxmeta
to stop, and, then try to stop and start the whole system,
and finally exit.
When restarting the system, the flag -w
is used, to start a new
watcher for the new system.
The implications are that a failure to start will not restart the system more times, and that stopping individual components requires to stop this program first, or it might take actions on its own.
1.34. killprocs
This is a command for local use only:
usage: killprocs [-h] [-9] [-a] [procs] kill processes
Use lx run
to run it at any/all of the installed hosts.
Without any flag, locates the DB process pids looking at the files reporting them, and then kills them.
With flag -a
, locates any process in the system by process name (all
the ones started by the DB use bin/…
as a name), and all java
processes starting with LX
, and kills them.
A TERM signal is sent, and then a KILL signal after a few seconds.
Flag -9
can be used to send only a KILL signal.
If process/component names are given (eg, kvds
, or lxqe100
),
only those processes are killed.
For example, send a kill signal to any kvds at blade123:
unix$ lx run blade123 cmd killprocs -9 kvds
2. Configuration Reference Manual
This section provides a reference for the leanXcale configuration file.
Any property may be used as a global property (before any host declaration), as a host property (within a host declaration and before any component), or as a component property (within a component declaration).
Here, we describe properties grouped by their usual scope, for your
convenience. But, note that instead of using mem
or any other
property within a component, it can be used globally to apply to all
components, or within a host, to apply to such host.
2.1. Global properties
-
aws: Used as a global property, means that this is to install and start an AWS instance using the (only) configured host name (or
lx1
by default). -
awsdisk N: Used as a global property, fixes the disk size for the AWS AMI. 30 GiB by default.
-
awsregion name: Used as a global property, fixes the AWS used to create the installed AMIs.
-
awstype type: Used as a global property, defines the instance type, by default,
m5.large
. -
cache: When specified, the query engine is enabled to handle in-memory table replicas. By default, it is disabled. E.g.:
host blade124 lxqe cache
-
compress: Used as a global property, makes the install use compression. On non AWS installs this requires setting lxpart for each host installed.
-
crypt: Used as a global property, makes the install use disk encryption. On non AWS installs this requires setting lxpart for each host installed.
-
docker: Used as a global property, makes the install adapt to docker install. In this case, starts and stops are handled by starting/stopping docker containers.
-
fwd yes|no: Used as a global property, enables forwarding
-
lxdir path: Directory used by the install. Components and hosts may override its value. Defaults to
/usr/local/leanxcale
. E.g.:lxdir $HOME/leanxcale
-
size [small|medium|large]: When used as global property selects heuristics to perform a small, medium, or large install. A large install is the default, using the whole machine. A small install tries to use few resources, and a medium install sits in the middle. Using flags
-s
and-m
forlxinst
adds this property with the corresponding size. -
stats [only]: When used as a host property (in one host only), configures hosts to supply statistics and runs prometheus and grafana on this node to gather stats and report them. If the value is
only
, the host is used just for statistics and DB components use other configured hosts. -
tls: When used as a global property, makes the installed system use TLS for network exchanges between clients and the installed system.
-
user name: User used by the install Eg:
user nemo
-
LXLDAP uri: URI used to reach the LDAP server for authentication. System users must still be created for LDAP users to be granted access. Defining this property changes the authentication method only. User
lxadmin
is never authenticated using LDAP.LXLDAP ldap://ldapsrv:389|simple|ou=People,dc=leanxcale,dc=com
-
LXPAM yes: The PAM sudo service is used to authenticate user. Note that using
sudo
as a service requires the user to be able to read the file/etc/shadow
, which is not a good idea. -
logtrim mode: Mode can be
no
,ds
(orpersist
), orbackup
. Sets the policy for removing old DB log transaction entries to: no removes, remove after persisted in data stores, or remove after backed up. Note that log entries may surive more than asked for, this is a limit for removal, not removal request.
2.2. Host properties
-
lxdir path: Directory used by the install in the host. E.g.:
lxdir $HOME/xamplelx
-
lxpart device: Device for the partition used for the install. E.g.:
lxpart /dev/sdc1
-
addr address: Network address used (without port).
2.3. Component properties
-
addr address: Network address used. E.g.:
host atlantis kvms addr atlantis!14000
-
disk N: Size for the disk used by the component, using
M
orG
as units or assuming GiB by default. E.g.:host atlantis kvds disk 50 lxqe disk 500m
-
mem N: Size for the memory used by the component, using
M
orG
as units or assuming MiB by default. E.g.:host atlantis kvds mem 500 lxqe mem 1g
-
mirror nameid: Used for
lxqe
components. Specify the component for the mirror transaction log. E.g.:host atlantis lxqe 100 mirror lxqe101 lxqe 101 mirror lxqe100
-
LXPULL addr1;addr2;…;addrn: Used for
lxqe
components. Make this system pull transactions made on the query engines found at the given addresses. E.g.:host atlantis lxqe 100 mirror lxqe101 lxqe 101 LXPULL orion!14420;orion!14424
2.4. Other properties
Properties with upper-case names are exported as environment variables
for the process(es) executing the component involved. This includes both
global properties and host properties as well. For example, to enable
debug flags DOM
for the kv datastore:
KVDEBUG DOM
These are other properties not described before.
-
mirror name: Defines a host as a mirror of the named host. Either all hosts must have mirrors or none of them can. (For command line arguments, the syntax is
blade124+blade145
). E.g.:host blade124 host blade125 mirror blade124
-
nodb: When specified as a host property, makes this host include just the
lx
command, but no DB component. E.g.:host atlantis nodb
-
scp template: Template for the scp command used to remotely copy files to a host. Use '{user}' and '{host}' where the user and host address should be included, and
{src} ` and ` {dst}
where the source and destination file names should be placed. E.g.:scp scp {src} {user}@{host}:{dst}
-
scpi template: Template for the (internal) scp command an installed host should use to reach other installed hosts. Use the same syntax described above. E.g.:
scpi scp {src} {user}@{host}:{dst}
-
spread no|port: Port used by the spread process. Use
no
to disable it. E.g.:spread 4444
-
ssh template: Template for the ssh command used to execute remote commands on the host. Use
{user}' and `{host}
where the user and host names should be used. Eg.:ssh ssh -o StrictHostKeyChecking=no {user}@{host}
-
sshi template: Template for the (internal) ssh command an installed host should use to reach other installed hosts. Use the same syntax describe above. Eg.:
sshi ssh -o StrictHostKeyChecking=no {user}@{host}
-
web [addr]: Run a web server as an interface for running Lx commands.