Administration Reference Guide

1. Command Reference Manual

This section details all the commands available and how to use them, starting with the install command.

1.1. lxinst

This is the install program:

usage: lxinst [-h] [-v] [-D] [-s] [-m] [-n] [-f cfgfile] [-d dist] [-k key]
              [-K awspem] [-u location] [-c] [-i]
              [where ...]

leanXcale installer

positional arguments:
  where        [aws|docker|stats] dir, host, or host:dir

options:
  -h, --help   show this help message and exit
  -v           verbose
  -D           enable debug diags
  -s           small install
  -m           medium install
  -n           dry run
  -f cfgfile   use this configuration file
  -d dist      distrib file, dir, or url
  -k key       key to download the distribution.
  -K awspem    AWS pem file name w/o .pem
  -u location  update inst at location
  -c           clean. download everything
  -i           ignore system limits

Given arguments specify where to install the DB. They may be paths for directories in the local host, or host names, or host and directory names using the syntax host:dir.

Instead of arguments, a configuration file may be given using the -f flag. The configuration file has been described before in this document.

Flag -k may be used to specify the key to download the distribution.

Flag '-K' must be used when installing on AWS to supply the path (without file name extension) to the pem/pub key files.

By default, the distribution is retrieved from the leanXcale artifactory. However, it is possible to supply one or more times the -d flag giving it the name for a .tgz file, a directory, or a URL. The special argument leanxcale stands for the official repository for the distribution.

Distribution packages will be retrieved by looking at those distribution sources. By convention, the first source is usally the ./lxdist directory, used to keep the distribution when downloaded.

For example, this installs just what is downloaded at ./lxdist, without downloading anything else:

unix$ lxinst -d ./lxdist -f lxinst.conf

Once downloaded, files are not downloaded again. Flag ''-c'' cleans the ''./lxdist'' directory to force a download of everything. If a different directory is specified by the user, it is not cleaned, although packages not found will be still downloaded there. To force a download of particular packages, remove the packages desired from the ''./lxdist'' directory (or the directory specified in the arguments).

For example, this downloads a fresh copy of the distribution and installs it:

unix$ lxinst -c -f lxinst.conf

And this tries first the ../lxdist directory and then the standard leanxcale repository:

unix$ lxinst -d ./lxdist -d leanxcale -f lxinst.conf

This is actually the default when no distribution source is specified. Also, if no directory source is specified as a first option, lxdist is used as a directory to download components (other than files found on the local host).

The database has users (different from UNIX users). The user lxadmin is the administrator for the whole system. During the install you will be asked to type a password for lxadmin. To specify the password without being asked, you can set the password in the LXPASS environment variable:

unix$ export LXKEY=nemo:APA...uQy
unix$ export LXPASS=fds92f3c
unix$ lxinst /usr/local/leanxcale
...

When the users file ./lxowner exists, that file is used as the file describing the lxadmin user and its secret, instead of creating one using the user supplied password for lxadmin.

Do not use a existing file unless you know what you are doing. The reason is that this file must define lxadmin as the first user and that such user must have access to all resources, as configured.

Flag -i makes lxinst ignore system limits, to install on places with reduced disk or memory and in a hurry.

By default, the install performs a large install, trying to use all the machine resources for the service. Flag -s selects a small instead, and flag -m selects a medium install.

The install size affects components and sizes not specified by the user in command line arguments or in the configuration file. In a small install, at most 4 kdvs are added per host, and component memory is limited to 1GiB. In a medium install, at most 8 kvds components are added, and component memory is left with default values.

Flag -u is used to update a previous install. Still WIP, do not use it for now.

1.2. lx

Shell for running LeanXcale control programs. This simply fixes the environment for the installed host and runs the command given as an argument:

usage: lx [-d] cmd...

When flag -d is used, the current working directory is set to the install directory at the host before running the given command.

Most commands follow the same conventions regarding options and arguments. We describe them here for convenience:

positional arguments:
  what        host|comp...

options:
  -h, --help  show this help message and exit
  -l          local run only
  -D          enable debug diags

Flag -l is used when executing the command locally. This is used by the lx command framework, and should not be used in general by the end user.

Flag -D enables verbose diagnostics

Arguments specify what to operate (e.g., what to start, stop, etc.) may be empty to rely on the defaults (whole DB) or may specify a particular host and/or component name:

  • when only component names are given, and only those components will be involved (e.g., lxqe101).

  • when a component type name is given, components for that type are selected. (e.g., lxqe).

  • when a host name is given, any component following is narrowed to that host. If no components follow the host name, all components from the host are selected.

This may be repeated to specify different hosts and/or components.

The special host names db, repl, and repl2 may be used and stand for hosts without the nodb attribute, hosts for the first replica, and hosts for the second replica (hosts that are a mirror of other ones).

1.3. lx add

Add the given files to the named dir at the installed locations.

usage: add [-h] [-v] [-D] [-a host] ...

add files to dir on installed lx

positional arguments:
  files       files... dir

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -a host     copy to this installed host(s)

For example,

unix$ lx add ./fwd ubin

adds the forwarding command found at ./fwd to the ubin directory on the installed locations.

Directory names should be one of ubin, ulib, or tmp. Using bin or lib is dangerous, you have been warned.

1.4. lx addlib

Add the given libraries or package file(s) to the installed locations.

usage: addlib [-h] [-l] [-v] [-D] [-a host] [-x] [files [files ...]]

add libs to installed lx

positional arguments:
  files       files to add

optional arguments:
  -h, --help  show this help message and exit
  -l          local run only
  -v          verbose
  -D          enable debug diags
  -a host     copy to this installed host(s)
  -x          internal use only

If the files added are distribution packages, they are unpacked on top of each one of the installed locations.

1.5. lx backup

usage: backup [-h] [-v] [-D] [-n] [-d dir] [-F] [-i] [-e] [-r] [-o]
              [what [what ...]]

make a backup

positional arguments:
  what        host/comps

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -n          dry run
  -d dir      root backup dir
  -F          force command
  -i          incremental backups
  -e          encrypt
  -r          restore
  -o          online backup

By default, backups are kept at $LXDIR/dump at the host used to issue backup commands. The convention is to use the first host to keep backups.

That is for internal backups (backups at a installed host).

For external backups, lxbackup is used at the host keeping the backup, and the backup top-level directory is given with flag -d or defaults to ./lxdump otherwise.

Directories under dump/ are named using the date as their name. When more than one backup is made on the same date, further digits are added to number succesive backups. A sorted list for the directory reports backups from oldest to newest.

For incremental backups, a final + is added to the directory name.

To create a full, cold, backup when leanXcale is not running:

unix$ lx backup
#backup...
new 240809
make atlantis disk/kvms100 /usr/local/leanxcale/dump/240809/kvms100
make atlantis disk/kvds100 /usr/local/leanxcale/dump/240809/kvds100
make atlantis disk/kvds101 /usr/local/leanxcale/dump/240809/kvds101
make atlantis disk/lxqe100/log /usr/local/leanxcale/dump/240809/lxqe100
unix$

The printed name is the name for the directory keeping the backup, as used when restoring it. In this case, 240809.

The files kept at the backup are compressed, and must uncompressed if copied by hand. The restore command takes care of this.

Using flag -e both encrypts and compresses the backup files. Flags belong to disk, note that backup is the argument given to disk.

The key used to encrypt the backed files is kept at $LXDIR/lib/lxkey.pem on the installed sites as set up by the installer.

unix$ lx backup -e
#backup...
new 24080901
make atlantis disk/kvms100 /usr/local/leanxcale/dump/24080901/kvms100 crypt
make atlantis disk/kvds100 /usr/local/leanxcale/dump/24080901/kvds100 crypt
make atlantis disk/kvds101 /usr/local/leanxcale/dump/24080901/kvds101 crypt
make atlantis disk/lxqe100/log /usr/local/leanxcale/dump/24080901/lxqe100 crypt

To create an incremental backup use the flag -i.

unix$ lx backup -i
#backup...
new 24080902+
incr: lxqe100 += lxqe100

The output reports which components got files backed up.

The incremental backup is encrypted if the total backup it refers to is encrypted too. No flag -e should be given.

The incremental backup can be performed while the system is running.

To create a backup while the system is running (known as an online backup), use flag -o for lx backup.

unix$ lx backup -o
#backup...
new 24080902
...

This is more expensive than doing a backup with the system down. The flag is required to prevent mistakes. If the system is down when an online backup is asked for, the backup is still made as a cold backup. That is, the flag says that it is Ok to backup while the system is online.

1.6. lxbackup

This is the external version of lx backup, for use on external (non installed) hosts to keep external backups.

usage: lxbackup [-h] [-v] [-D] [-n] [-f cfgfile] [-d dir] [-i] [-e] [-r] [-o]
                [-F]
                [what [what ...]]

make a backup

positional arguments:
  what        host/comps

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -n          dry run
  -f cfgfile  config for the install
  -d dir      root backup dir
  -i          incremental backups
  -e          encrypt
  -r          restore
  -o          online backup
  -F          force command

Using lxbackup is exactly like using lx backup with a few differences:

  • Flag -f is mandatory on the first invocation and provides the installed configuration file.

  • The default backup directory is not $LXDIR/dump, but ./lxdump.

  • The command kvtar must be available at the host running lxbackup.

The external host must have ssh access to the installed hosts.

The configuration file used must be the one retrieved using lx config, and not the one written by the user to perform the install, because lx config reports addresses and details needed by the command.

Once a backup has been created, the configuration used is saved along with the backup data and there is no need to use -f to supply it.

Encryption/decryption happens at the source/destination of data when creating/extracting backups. Therefore, there is no key kept at the host running lxbackup.

To prepare a host to use lxbackup, get the lxbackup command, the configuration, and the kvtar command and copy them to the host. You might want to copy also lxrestore and lxbackups.

These commands can be found at the installed $LXDIR/bin directory on any installed host. If you are not sure regarding the $LXDIR value, use this command to find it:

unix$ lx -d pwd
/usr/local/leanxcale
unix$

Flag -d for lx makes it change to the $LXDIR directory before running the given command.

The detailed configuration file to be used is kept at $LXDIR/lib/lxinst.conf. The configuration can be retrieved using the lx config command. For example, this creates ./lxinst.conf with the detailed configuration:

unix$ lx config -o lxinst.conf
saved lxinst.conf

As an example, we can setup an external host named orion to perform backups in this way:

unix$ lx -d pwd
/usr/local/leanxcale
unix$ lx config -o /tmp/lxinst.conf
saved /tmp/lxinst.conf
unix$ cd /usr/local/leanxcale/bin
unix$ scp lxbackup lxbackups lxrestore kvtar /tmp/lxinst.conf orion:~

And then just:

orion$ lxbackup -f lxinst.conf

This is just an example. It is a good idea to copy those commands to your local bin directory on the external host.

1.7. lx backups

To list the backups known use the backups command:

usage: backups [-h] [-v] [-D] [-n] [-d dir] [what [what ...]]

list backups

positional arguments:
  what        [last|backupname] host/comps

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -n          dry run
  -d dir      root backup dir

For example:

unix$ lx backups
240810 ts 1419000
24081001+ ts 1419000
...

Those with a + in their names are incremental backups.

Verbosity can be increased adding one or more -v flags.

unix$ lx backups -v
240810 ts 1419000
240810/kvms100 ts 2551000
240810/kvds100 ts 1292000
240810/lxqe100 ts 1419000
240810/lxqe101 ts 1419000
24081001+ ts 0
24081002+ ts 1419000
24081002+/lxqe100 ts 1419000
...
unix$ lx backups -vv
240810 ts 1419000
240810/kvms100 ts 2551000
240810/kvms100/kvmeta sz 239 ts 2551000 mt 1723285636
240810/kvds100 ts 1292000
240810/kvds100/dbf00001.kv sz 393216 ts 1419000 mt 1723285636
240810/kvds100/seqs sz 4 ts 0 mt 1723285599
240810/lxqe100 ts 1419000
...
unix$ lx backups -vvv
240810 ts 1419000
	240810/kvms100 ts 2551000
		240810/kvms100/kvmeta sz 239 mt 1723285636
			localhost
			ts 1 2551000 bck 0 0 na 0
	240810/kvds100 ts 1292000
		240810/kvds100/dbf00001.kv sz 393216 mt 1723285636
			localhost
			ts 1292000 1419000 bck 0 0 na 3
			tname db-APP-PERSONS
			rmin: tpmin
			rmax: tpmax
...

It is possible to list a single backup by suppling its name and/or specific components using arguments as done with most lx commands. The name last refers to the last backup made. For example:

unix$ lx backups -v 24080901

or

unix$ lx backups -v 24080901 lxqe

1.8. lxbackups

This is the external version of lx backups. It is used exactly like lx backups.

usage: lxbackups [-h] [-v] [-D] [-n] [-f cfgfile] [-d dir] [what [what ...]]

list backups

positional arguments:
  what        [last|backupname] host/comps

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -n          dry run
  -f cfgfile  config for the install
  -d dir      root backup dir

Refer to lxbackup for installation details.

1.9. lx copy

The lx copy command can be used to copy disks from hosts, directories or components to other hosts, directories or components.

usage: copy [-h] [-v] [-D] [-n] [-F] [-l] [what [what ...]]

copy component disks

positional arguments:
  what        src... dst

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -n          dry run
  -F          force command
  -l          call from lx.cptree

		src/dst: repl|host|comp|path

The special source or target repl can be used to copy to or from the replica of a component to restore a mirror from another.

For example

unix$ lx copy kvds100 /tmp

copies the files for kvds100 to /tmp/kvds100.

The same is achieved using

unix$ lx copy kvds100 /tmp/kvds100

As another example:

unix$ lx copy repl1 repl

copies the whole set of disks at replica-1 hosts into their mirrors.

The same thing can be achieved by copying one component at a time using

unix$ lx copy lxqe100.r1 repl

1.10. lx config

Inspect or update the configuration:

usage: config [-h] [-v] [-D] [-s value] [-d] [-n] [-o fname]
              [what [what ...]]

Inspect the configuration

positional arguments:
  what        kvaddr|grafana|cons|[host] [comp|prop...]

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -s          set property values
  -d          delete
  -n          dry run
  -o fname    write the output to fname

By default, lx config prints the whole configuration, or that for elements given as arguments.

The arguments follow the conventional syntax used by most commands, but knows property names also:

  • Giving a host name narrows the rest of the arguments to that host, until another host name is given

  • Giving a component kind (e.g., lxqe) selects those components.

  • Giving a component name with included id (e.g., lxqe101) selects just that component

  • Giving a property name (not a host and not a component) selects just that property.

For example,

unix$ lx config

prints all configuration.

unix$ lx config atlantis mariner

prints the configuration for hosts atlantis and mariner (assuming those are configured host names).

unix$ lx config atlantis kvms mariner lxqe

prints the configuration for kvms components found at atlantis and lxqe components found at mariner

unix$ lxconfig kvms kvds101

prints the configuration for any kvms component and the kvds101 one.

unix$ lxconfig kvms addr

prints the addr attribute for any kvms component.

unix$ lxconfig awsdisk

prints the awsdisk property from the (global) configuration.

Use flag -o to save the configuration (or the parts selected) to the given file.

Use flag -d to remove the selected configuration entries. Do not remove hosts or components, and use this with caution.

Use flag -s to update the selected properties with new values. In this case, the argument for a property includes both its name and the new value (e.g., mem=50m)

For example:

unix$ lx config -s lxqe mem=500m kvds mem=500m

1.11. lx ctl

Issue control requests to lx components

usage: ctl [-h] [-v] [-D] comp [ctl [ctl ...]]

issue control requests

positional arguments:
  comp        component or command
  ctl         ctl arg

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags

	comps:
	ds, ms (master), qe, mm, meta (all ms), all, compname
commands:
	sync: sync the disk
	debug: add/remove/set debug flags
	stats: print statistics
useful ctls:
	all: debug +flags|-flags|flags
	qe: pull start|stop|addrs

This command issues control requests to running processes. Use with care and do not issue a control request unless you know what you are doing.

The comp argument selects the target for the control request or is a known command. It can be any of:

  • a component name, like lxqe100

  • a process name, like qe100

  • a process address, like orion!14440

  • all, to address all processes taking control requests

  • ms or kvms, to address the master kvms process

  • meta, to address all kvms processes

  • ds or kvds, to address all kvds processes

  • cli, to ask for a control request to be forwarded to all running kvms clients.

  • a specific command, like sync, in which case addressed components are implicit.

Different processes have different control requests. In general, ? can be used to learn the control requests supported.

For example,

unix$ lx ctl qe ?
help [cmd]
sync
debug flags
mdebug on/off
dump conflicts|txns|cfg|ni
print conflicts|txns|cfg|ni
txdebug on|off
up [txn|conn]
down [txn|conn]
run
halt: disable txns
abort qe
pull [stop | start | qeaddr...]
test crash|hup now|begin|commit|msgs|xfer|publish|end|rescue

This command calls kvcon to actually issue control requests, after adjusting a bit the target names as a convenience.

1.11.1. Debug Control Request

The debug control request adds, clears, or sets debug flags. This ctl is also understood as a direct command (addressing everyone).

For example, adding no extra debug flags can be used to see current debug flags set:

unix$ lx ctl debug +
local:
ms100: DMMOO
ds100: DMMOO
mm100: dDMMOO
qe100: dDMMOO

For lxmeta (mm) and lxqe (qe) processes, lowercase debug flags correspond to log4j loggers and uppercase flags correspond to standard flags used everywhere in the system.

Repeating a flag increases the debug level. Popular debug flags are:

  • D: general debug

  • O: operations

  • M: metadata

Popular debug flags for java are also:

  • j: lxjdbc debug. JDBC requests

  • m: lxmeta debug. Lxmeta processing.

  • q: lxqe debug. Query engine processing.

  • n: lxnet debug. Network operations.

  • l: lxlogger debug. DB Logger and records.

  • s: lxss debug. Snapshot server.

  • c: lxcm debug. Conflict manager.

  • t: lxtxns debug. Transactions.

  • e: lxenum debug. Enumerations.

1.11.2. Stats Control Request

To ask for statistics, use the stats command:

unix$ lx ctl lxqe100 stats
...
unix$ lx ctl stats
...
unix$ lx ctl sync
unix$ lx ctl debug +

1.11.3. Print or Dump Control Request

The dump request asks a process to dump data structures (or some of them) to the log. The print request is similar, but sends the output to the client issuing the request (i.e., the output is shown by lx ctl).

Without further arguments, the request may print the most important ones, or just remind its usage.

For query engines, these are the requests known:

unix$ lx ctl qe dump ?
dump conflicts|txns|cfg|ni

They correspond to the conclict manager structures, the transactions, the configuration, and the NetInfo information.

For lxmeta processes, we have:

unix$ lx ctl mm dump ?
dump slots|cfg|ni

Here, slots corresponds to timestamp slots.

For kvms, the request has no arguments and dumps the metadata as known by that process.

For kvds, the dump requests are:

unix$ lx ctl ds dump ?
dump [all|files|fileblks|seq|proc|stack|tids [full|blks]|mem [kind]|reqs|fid [hdr]

And the print requests are:

unix$ lx ctl ds print ?
print [files|seq|proc|stack|tids|mem [kind]|reqs

Without arguments, both requests print the table of server files used for tables.

An important request is proc, that prints the process table for the server, and another related one is stack, which prints the stacks for the processes in the server.

1.12. lx dbdata

List database data files:

usage: dbdata [-h] [-v] [-D] [-l] [-a] [-n] [-r] [-t tbl] [-i idx] [-w prop]
              [what [what ...]]

list database data files

positional arguments:
  what        path|cfg

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -l          local run only
  -a          print all content
  -n          dry run
  -r          print rows
  -t tbl      table name to list data files for
  -i idx      idx name to list data files for
  -w prop     set bck:ts

The command lists DB data files as found on disk.

For example:

unix$ lx dbdata
blade110 kvds100 dbf00001.kv tree db-APP-PAYMENTS5
blade110 kvds100 dbf00003.kv deriv db-APP-PAYMENTS5_OA
blade110 kvds100 dbf00005.kv tree db-APP-TBL2

Each line in the output lists the machine and component where the file is kept before the file information.

Flag -v makes the program more verbose:

unix$ lx dbdata
blade110 kvds100 dbf00001.kv tree db-APP-PAYMENTS5 1:2 0x8000000002
	ts 596000 2692000 snapts 0 snapno 31 bckts 0 bckdts 0 0 syncs
	rmin: tpmin
	rmax: tpmax
blade110 kvds100 dbf00003.kv deriv db-APP-PAYMENTS5_OA 3:2 0x18000000002
	ts 596000 2692000 snapts 0 snapno 6 bckts 0 bckdts 0 0 syncs
	rmin: tpmin
	rmax: tpmax
	deriv from db-APP-PAYMENTS5
blade110 kvds100 dbf00005.kv tree db-APP-TBL2 5:2 0x28000000002
	ts 2796000 2796000 snapts 0 snapno 4 bckts 0 bckdts 0 0 syncs
	rmin: tpmin
	rmax: tpmax

The information listed, and the table and index names, is taken from the on-disk files, as known by the data servers.

For example, db-APP-PAYMENTS5 refers to database db, schema APP, and table PAYMENTS5.

To ask for a particular table, use flag -t and give the table name as an argument in the format just described:

unix$ lx dbdata -t db-APP-PAYMENTS5

The region minimum and maximum values are listed also as known by the data servers (i.e., by kvds) and not by the user or the query engines.

Flag -r prints the rows in the files too. They are printed in the stored field order, not in the user SQL table field order.

unix$ lx dbdata -rt db-APP-PAYMENTS5
blade110 kvds100 dbf00001.kv tree db-APP-PAYMENTS5
	t1236000 - 10|  1
	t1244000 - 11|  1
	t1254000 - 21|  3
	...

It is possible to list only for particular hosts or components by supplying the usual names as arguments.

For example:

unix$ lx dbdata blade110

or

unix$ lx dbdata kvds100

Flag -w is meant to write properties for the data files. For now, only the backup TS as stored in the file. Use a in:

unix$ lx dbdata -w bck:777 kvds100

Where the number after bck: is the backup TS to write.

1.13. lx dblog

Print the transaction log contents:

usage: dblog [-h] [-v] [-D] [-l] [-i] [-f] [-r] [-a] [-n] [-m] [-c count]
             [-s ts] [-w prop] [-e ts]
             [what [what ...]]

print database log files

positional arguments:
  what        host|comp...

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -l          local run only
  -i          log header only
  -f          log file information only
  -r          log records only
  -a          print all record content
  -n          options -s and -e refer to record nbs.
  -m          print the mirror log
  -c count    print at most count records
  -s ts       start from this ts
  -w prop     set bck:ts or prop
  -e ts       end at this ts

The command prints the log contents for the given names. Names can be host and or component names to select the addressed components, or a log directory or a logger address.

By default the command prints the log header and the records in it.

Under flag -i, only the information in the log header is printed:

unix$ lx dblog  -i
logs...
blade105: [
	hdr vers1 sealed
		18 records 0:18
		2831 bytes
		1113002:1358002 ts
		1 meta records
		12 commit records
		12 xfer records
]

Under flag -r, only the records are printed.

For data records, only the first line of the recorded message is printed. Use flag -a to print the full record contents (eg, all tuples recorded).

Flags -s and/or -e may be used to focus the search on a timestamp interval, or, if flag -n is given, on a record number interval.

For example, print records with numbers from 10 to 14 for lxqe100:

unix$ lx dblog -r -n -s 10 -e 15 lxqe100
logs...
blade105: [
	commit[10] t1311002 txn.0x14011a r1 boff 1370
	xfer[11] t1311002 txn.0x14011a r1 boff 1759
	data[12] t1348002 db-APP-PERSONS txn.0x1491a2 r1 boff 1808
		Tadd tag 26 sz 286 flg 0x40 0x1491a2 0x0 0x1491a2 <db-APP-PERSONS> 0x0 0x1485e9
			[0]<> [0]<> tpzero [2]<0509> tpzero tpzero
	commit[13] t1348002 txn.0x1491a2 r1 boff 1857
	xfer[14] t1348002 txn.0x1491a2 r1 boff 2246
]

Flag -w should be used with care. If writes the given flag to the log file headers on disk.

To update the backup timestamp to 1234, for example, use:

unix$ lx dblog -w bck:1234

The timestamp 0 means the last one kept in the log.

Other arguments for -w are rescued, single, couple, and mirror status strings.

1.14. lx dbmeta

Print DB metadata as found on disk.

usage: dbmeta [-h] [-v] [-D] [-l] [-a] [-n] [-w prop] [-f [file]]
              [what [what ...]]

print database meta files

positional arguments:
  what        path|cfg

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -l          local run only
  -a          print all content
  -n          dry run
  -w prop     write flag
  -f [file]   meta file

This command prints the DB metadata as kept on disk.

For example, on a replicated install:

unix$ lx dbmeta
dbmeta...
blade110:
	# kvms100.r1
	meta ts 0
	kvms blade110!14400
	usr:	none
	usr:	lxadmin
blade161:
	# kvms100.r2
	meta ts 0
	kvms blade161!14400
	usr:	none
	usr:	lxadmin

The command accept a kv metadata path to list, similar to kvcon list arguments. For example, to list server metadata as kept on disk:

unix$ lx dbmeta /srv
dbmeta...
blade110:
	# kvms100.r1
	kvms blade110!14400
	kvms blade161!14400
	kvds ds100.r1 at blade110!14500
	lxmeta mm100.r2 at blade161!14410
	kvds ds100.r2 at blade161!14500
	lxmeta mm100.r1 at blade110!14410
	lxqe qe100.r1 at blade110!14420
	lxqe qe100.r2 at blade161!14420

Flag -w can be used to set or clear metadata server flags. For example:

unix$ lx dbmeta -w single /srv/ds100.r1

sets the single flag for the named server, and

unix$ lx dbmeta -w nosingle /srv/ds100.r1

clears the flag.

After restoring a replica, the fail flag for the restored component is usually cleared, and the single flag for the component used as the restore source is cleared as well.

As another example, this lists the metadata information for a table:

unix$ lx dbmeta /db/APP/tbl/INDEX2T0001

Using flag -a reports all information:

unix$ lx dbmeta -a /db/APP/tbl/INDEX2T0001
		owner 'lxadmin'
		fmt#0x0: 16 flds 1 key 16 usr 15 delta 0 sflds 0 old
		... more output removed ...
		2 regs 2 repls
		reg [ds101:69:2]
			min: tpmin
			max: tpmax
		reg [ds100:69:2]
			min: tpmin
			max: tpmax

1.15. lx disable

Disable transactions.

usage: disable [qe|qename]

For example,

	unix$ lx disable

disables transactions for just the entire system, until the they are enabled.

While transactions are disabled, lx status shows the system status as waiting (for transactions to become available).

Before using this command, stop the watcher if the system was starting with flag -w or lx watch is running.

Otherwise, because new JDBC connections are not accepted while transactions are disabled, the watcher might consider that the system is not in good health, and then restart the system.

1.16. lx enable

Enable transactions.

usage: enable [qe|qename]

For example,

	unix$ lx enable qe100

enables transactions for just the qe100 query engine.

1.17. lx fmt

Format the whole DB or the indicated hosts or components:

usage: fmt [-h] [-v] [-D] [-l] [-c] [what [what ...]]

fmt the store

positional arguments:
  what        host|comp...

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -l          local run only
  -c          clean

For example, format the kvds101 disk:

unix$ lx fmt kvds101

Flag -c removes the logs too.

1.18. lx heap

Dump java process heaps and heap histograms.

usage: heap [-h] [-v] [-D] [-l] [-d] [-s] [-c] [-r] [-o] [-n num]
            [what [what ...]]

get heap dumps and histograms

positional arguments:
  what        host|comp...

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -l          local run only
  -d          save a heap dump
  -s          save a heap histogram
  -c          cmp with baseline heap histogram
  -r          remove saved dump and histogram
  -o          include non-live objects in histogram
  -n num      number of lines

The command understands the conventional syntax to select hosts and/or components. For example, save a base histogram to compare later and check out for leaks we can:

unix$ lx heap -s lxqe
heaps...
blade105 [
	saved log/lxqe100.r1.hist
]
blade110 [
	saved log/lxqe100.r2.hist
]

Later, we can compare with another histogram using

lx heap -c lxqe
heaps...
blade105 [
	saved log/lxqe100.r1.hist2
	lxqe100.r1:
]
blade110 [
	saved log/lxqe100.r2.hist2
	lxqe100.r2:
		[B: 9919 objs 674512 bytes
		java.lang.String: 9696 objs 232704 bytes
		java.lang.Class: 1779 objs 212480 bytes
		[I: 923 objs 157680 bytes
]

To print the current histogram (top 50 lines by default), run it without arguments:

lx heap -c lxqe
heaps...
blade105 [
	lxqe100.r1:
		 num     #instances         #bytes  class name (module)
		-------------------------------------------------------
		   1:         29587        7329264  [B (java.base@11.0.22)
		   2:          2905         814536  [I (java.base@11.0.22)
		   3:          6303         756888  java.lang.Class (java.base@11.0.22)
	...

1.19. lx help

Ask for help:

usage: help [cmd]

Print usage information

Prints the list of known commands with quick usage information, or detailed usage information about the given command.

1.20. lx license

Checks the license status or installs a new license:

usage: license [-h] [-v] [-D] [-f file]

inspect or update license files.

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -f file     license file to install

For example, ask for the current status:

unix$ lx license
    license expires: Mon Dec 30 00:00:00 2024

or install a new file lxlicense with the desired license:

unix$ lx license lxlicense

1.21. lx logs

List and inspect logs for installed components:

usage: logs [-h] [-D] [-g rexp] [-a] [-p] [-c dst] [-s start]
            [-e end] [what ...]

list logs

positional arguments:
  what        host|comp...

options:
  -h, --help  show this help message and exit
  -D          enable debug diags
  -g rexp     grep rexp
  -a          all logs, not the last one
  -p          print the last log (all if -a)
  -c dst      copy the last logs to this dir (all if -a)
  -s start    start fname time (yymmdd.hhmm or prefix)
  -e end      end fname time (yymmdd.hhmm or prefix)

For example, list the logs for kvds at atlantis, but only those after the time 230518.0551 (yymmdd.hhmm, or a prefix of this, can be used):

unix$ lx logs  -s 230518.0551 atlantis kvds

Print the last log for kvds101:

unix$ lx logs  -p kvds101

Grep all the logs for lines with fatal:

unix$ lx logs  -g fatal

Copy all (not just the last one) kvds101 logs to /tmp:

unix$ lx -a -c /tmp kvds101

1.22. lx move

The lx move command can be used to move tables or regions from one data server to another while the system is down.

usage: move [-h] [-v] [-D] [-n] [-F] [what [what ...]]

move tables or regions

positional arguments:
  what        tbl [rid] dstds | srcds dstds

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -n          dry run
  -F          force command

For example

unix$ lx move db-APP-PERSONS kvds100

moves the files for the named table to the given server. The table name uses the data server syntax, not the SQL syntax. This command is not meant for users.

1.23. lx procs

List the processes for the DB:

usage: procs [-h] [-v] [-D] [-p] [what ...]

list processes

positional arguments:
  what        host|comp...

options:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -p          report ports in use

For example, list the processes at host blade123:

unix$ lx procs blade123
procs...
blade123 [
	lxmeta100	pid 813750	alive running
	kvds100	pid 813734	alive running
	kvms100	pid 813729	alive
	kvds103	pid 813746	alive running
	kvds101	pid 813738	alive running
	spread	pid 813725	alive
	kvds102	pid 813742	alive running
	lxqe100	pid 813773	alive running
]

Or to see the port ustage status:

unix$ lx procs -p blade123
blade123 [
		spread	14444	busy
		kvms100	14400	busy
		lxmeta100	14410	idle
		lxqe100	14420	busy
		kvds100	14500	busy
		kvds101	14504	busy
		kvds102.14508	busy
		kvds103	14512	busy
]

See also flag -p of lx status.

1.24. lx recover

usage: recover [-h] [-v] [-D] [-n] [-d dir] [-F] [what [what ...]]

recover replica disks

positional arguments:
  what        repl|host|comp...

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -n          dry run
  -d dir      root backup dir
  -F          force command

The command lx recover locates disks for failed replicas or replica components and restored them from their mirrors.

Arguments given restrict the repairs to them.

For example, after replica failures, we can run

unix$ lx recover

To try to recover their disks before restarting.

Use flag -n for a dry run to see what would be repaired before running it, unless you are sure to recover failed replicas.

1.25. lx report

Report system status and debug information.

usage: report [-h]

report system information for debugging

positional arguments:
  what        host|comp...

options:
  -h, --help  show this help message and exit
  -D          enable debug diags

This program collects information from the system and builds an archive to be sent to support.

unix$ lx report
report: lxreport.231009...
version...
procs...
logs...
stacks...
stack lxmeta100...
stack kvds103...
stack kvms100...
stack spread...
stack kvds102...
stack kvds100...
stack kvds101...
stack lxqe100...

# send this file to support.
-rw-rw-r-- 1 leandata leandata 54861 Oct  9 14:58 lxreport.231009.tgz

As printed by the command output, the resulting tar file should be sent to support.

The archive includes:

  • installed version numbers

  • underlying OS names are versions

  • complete disk usage for the installed systems

  • complete process list for the installed systems

  • memory usage for the installed systems

  • lx process list

  • logs for components (last log file only, for each one)

  • stacks for each component

  • stacks for each core file found

When kvms is still running, the archive includes also:

  • statistics for the sytem

  • long list of kv resources

  • process list for each kvds

  • file list for each kvds

1.26. lx restore

To restore a backup, use the restore command.

usage: restore [-h] [-v] [-D] [-n] [-d dir] [-F] [what [what ...]]

restore a backup

positional arguments:
  what        last|backupname [repl|host|comp...]

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -n          dry run
  -d dir      root backup dir
  -F          force command

For example:

unix$ lx restore
#disk...
check...
restore 24080904...
restore atlantis disk/kvds100 from /usr/local/leanxcale/dump/24080904/kvds100 crypt
dbf00001.kv
dbf00003.kv
...

Do this while the system is stopped. By default, it selects the last backup made.

To restore a particular backup, supply its name:

unix$ lx restore 24080904

This can be done also for incremental backups. When restoring an incremental backup, the restore takes data also from previous incremental backups and the corresponding total backup.

To restore only specific hosts or components, supply their names or types as done for other commands:

unix$ lx restore lxqe

Or perhaps:

unix$ lx restore 24080904 lxqe

To restore a backup, it is usually desirable to format the disk for the involved components before using restore to restore their disks.

To remove a backup, it suffices to remove its directory. For example:

unix$ lx -d rm -rf dump/230720

Here we used the flag “-d” for lx to change to $LXDIR before executing the remove command, which makes it easy to name the directory used for the dump.

When restoring encrypted backups, the disk command knows it has to decrypt, and it uses the key at $LXDIR/lib/lxkey.pem, the one used when making the backup.

If you did reinstall, update the key so it matches the backup key used.

For external backups, use lxrestore instead.

1.27. lx run

Run a command on the selected installed hosts, using the lx environment to run it:

usage: run [-h] [-v] [-D] [-d] ...

run a command on the installed host(s).

positional arguments:
  hosts       host|comp... cmd...

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -d          cd $LXDIR before running

Arguments specify hosts where to run (as usual in the rest of commands), perhaps none (to imply all DB hosts), and then the keyword cmd must be given, followed by the command and arguments to run on each host.

For example, discover where this thing is installed, by running pwd and using flag -d to set the current directory to the install directory on each host.

unix$ lx run -d cmd pwd

Or, as an exceptional measure, kill all processes running:

unix$ lx run cmd killprocs

Or, kill just those at blade123:

unix$ lx run blade123 cmd killprocs

Or kill -9 any kvds at blade123:

unix$ lx run blade123 cmd killprocs -9 kvds

1.28. lx stack

Dump process stacks

usage: stack [-h] [-v] [-D] [-l] [what [what ...]]

print stacks

positional arguments:
  what        host|comp...

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -l          local run only

The command understands the conventional syntax to select hosts and/or components. For example, to dump the stack of kvms components:

unix$ lx stack kvms
localhost [
	stack kvms100: [
		Thread 12 (Thread 0x7f8fc57fa700 (LWP 9272)):
		#0  __libc_read (nbytes=40, buf=0x7f8fa8000bd8, fd=13) at linux/read.c:26
		#1  __libc_read (fd=13, buf=0x7f8fa8000bd8, nbytes=40) at linux/read.c:24
		...
	]
]

With flag -v, local variables are printed too.

For java processes, both the native stack and the java stacks are printed.

1.29. lx start

Starts the DB for operation:

usage: start [-h] [-v] [-D] [-l] [-w] [-k] [-r] [what [what ...]]

start the service

positional arguments:
  what        host|comp...

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -l          local run only
  -w          start watch
  -k          keep core files
  -r          restart failed QEs

For example, to start the kvds components at host atlantis and leave the rest of the installation alone:

unix$ lx start atlantis kvds

With flag -w, start will run the watch service. This pings the DB to make sure it can answer queries, and, when that is not the case, try to stop and restart the system. See the Section 1.33 section for details.

By default, start will remove files on the installed directories with names suggesting they are core dumps. Flag -k prevents this from happening and can be used to keep core dump files for debugging.

Under flag -r, start will ask any lxmeta process started to try to restart a failed QE that was running, unless it was restarted less than one minute ago.

1.30. lx status

Prints the status for the system or waits for a given status:

usage: status [-h] [-v] [-D] [-w status] [-t tout] [-r] [-p]

show or wait for a DB status

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -w status   wait for the given status
  -t tout     timeout for -w (secs)
  -r          print replica status
  -p          print process status

For example, to learn the status:

unix$ lx status
status: waiting
	kvds100: recovering files
	kvds101: recovering files

Or, to wait until the status is running:

unix$ lx status -v -w running
status: waiting
	kvds100: recovering files
	kvds101: recovering files
status: running
unix$

The status can be any of:

  • stopped: no process is running.

  • failed: some processes did fail.

  • waiting: processes are running but there is no SQL service.

  • running: processes are running and SQL connections are available.

It can be also running with failures when using replication and suffering failures.

Flag -r reports the replica status:

unix$ lx status -r
status: stopped
replica: ok
unix$

The replica status can be any of:

  • no: no replication used.

  • ok: replication used and state ok.

  • single: replication used and suffering single processes (mirror failures).

For example:

unix$ lx status -r
status: stopped
replica: no
unix$

Flag -p reports the process status for each known component For example:

unix$ lx status -p
kvds100 alone stopped
kvms100 alone stopped
lxmeta100 alone stopped
lxqe100 alone stopped
unix$

When using replicas, the preferred mirror process is listed first.

unix$ lx status -rp
status: stopped
replica: ok
kvds100.r2 mirror stopped snap 890191999
kvds100.r1 mirror stopped snap 352036999
kvms100.r1 mirror stopped
kvms100.r2 mirror stopped
lxmeta100.r1 mirror stopped snap 890191999
lxmeta100.r2 mirror stopped
lxqe100.r1 mirror stopped snap 889261999
lxqe100.r2 mirror stopped snap 889260999
unix$

The mirror status for each process can be any of:

  • mirror: replicated and its mirror is ok.

  • alone: not replicated.

  • single: did survive its mirror process.

  • outdated: did stop or fail before its mirror process.

  • conflict: did survive its mirror and its mirror thinks the same.

1.31. lx stop

Halts the DB or stops individual hosts or components:

usage: stop [-h] [-v] [-D] [-l] [-w] [-k] [what [what ...]]

stop the service

positional arguments:
  what        host|comp...

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags
  -l          local run only
  -w          do not stop watch
  -k          kill -9

For example, halt the DB:

unix$ lx stop

Stop just the kvds servers:

unix$ lx stop kvds

If the watch service is running, make sure to stop it before stopping individual components. Otherwise it might decide to wait for lxmeta to stop and then restart the system.

Without arguments, stop will first stop the watch service and there is no extra caution needed.

Under flag -w, stop will not stop the watch process.

Flag -k may be used to kill components. Do not use this unless you know what you are doing. For example, to kill the lxqe100

unix$ lx stop -k lxqe100
stop...
atlantis: [
	stop: kill -9 lxqe100 pid 791944...
]

1.32. lx version

Print the installed version:

usage: version [-h]

print installed version

optional arguments:
  -h, --help  show this help message and exit

The program reports the leanXcale version name, along with detailed version information for distribution packages installed:

unix$ lx version
leanXcale v2.2
    kv         v2.2.2023-09-29.115f5fba70e3af8dc203953399088902c4534389
    QE         v2.2.2023-09-30.1e5933900582.26a7a5c3420cd3d5d589d1fa6cc
    libs       v2.2.2023-09-29.67535752acf19e092a6eaf17b11ad17597897956
    avatica    v2.2.2023-09-27.0b0a786b36e8bc7381fb2bb01bc8b3ed56f49172
    TM         v2.2.2023-09-29.9a9b22cfdc9b924dbc3430e613cddab4ed667a57
    lxlbins    v2.2.2023-09-29.79e7e04fb16b38d08c2d5df1fe08e103d49cb22a
    lxinst     v2.2.2023-10-02.b341e6545913aee8e0b0daf255362.273b33ea6d
    calcite    v2.2.2023-09-27.d3dfcf24285d38add3f4e29a9c2e9eacbcd0b913
    lxodata    v2.2.2023-09-23.b84fa4c7d2ca3e778edd9de29389b2aa6e1a9fb8

1.33. lx watch

Watch out the system and restart it if needed:

usage: watch [-h] [-v] [-D]

watch out the db and stop/start it if needed

optional arguments:
  -h, --help  show this help message and exit
  -v          verbose
  -D          enable debug diags

This program is started by lx start when the flag -w is given to it. It is usually a bad idea to execute this command explicitly.

It waits until the DB is running, doing nothing until that point. From that point on, if the DB ceases to be running, the program will wait for lxmeta to stop, and, then try to stop and start the whole system, and finally exit.

When restarting the system, the flag -w is used, to start a new watcher for the new system.

The implications are that a failure to start will not restart the system more times, and that stopping individual components requires to stop this program first, or it might take actions on its own.

1.34. killprocs

This is a command for local use only:

usage: killprocs [-h] [-9] [-a] [procs]

kill processes

Use lx run to run it at any/all of the installed hosts.

Without any flag, locates the DB process pids looking at the files reporting them, and then kills them.

With flag -a, locates any process in the system by process name (all the ones started by the DB use bin/…​ as a name), and all java processes starting with LX, and kills them.

A TERM signal is sent, and then a KILL signal after a few seconds.

Flag -9 can be used to send only a KILL signal.

If process/component names are given (eg, kvds, or lxqe100), only those processes are killed.

For example, send a kill signal to any kvds at blade123:

unix$ lx run blade123 cmd killprocs -9 kvds

2. Configuration Reference Manual

This section provides a reference for the leanXcale configuration file.

Any property may be used as a global property (before any host declaration), as a host property (within a host declaration and before any component), or as a component property (within a component declaration).

Here, we describe properties grouped by their usual scope, for your convenience. But, note that instead of using mem or any other property within a component, it can be used globally to apply to all components, or within a host, to apply to such host.

2.1. Global properties

  • aws: Used as a global property, means that this is to install and start an AWS instance using the (only) configured host name (or lx1 by default).

  • awsdisk N: Used as a global property, fixes the disk size for the AWS AMI. 30 GiB by default.

  • awsregion name: Used as a global property, fixes the AWS used to create the installed AMIs.

  • awstype type: Used as a global property, defines the instance type, by default, m5.large.

  • cache: When specified, the query engine is enabled to handle in-memory table replicas. By default, it is disabled. E.g.:

    host blade124
        lxqe
            cache
  • compress: Used as a global property, makes the install use compression. On non AWS installs this requires setting lxpart for each host installed.

  • crypt: Used as a global property, makes the install use disk encryption. On non AWS installs this requires setting lxpart for each host installed.

  • docker: Used as a global property, makes the install adapt to docker install. In this case, starts and stops are handled by starting/stopping docker containers.

  • fwd yes|no: Used as a global property, enables forwarding

  • lxdir path: Directory used by the install. Components and hosts may override its value. Defaults to /usr/local/leanxcale. E.g.:

    lxdir $HOME/leanxcale
  • size [small|medium|large]: When used as global property selects heuristics to perform a small, medium, or large install. A large install is the default, using the whole machine. A small install tries to use few resources, and a medium install sits in the middle. Using flags -s and -m for lxinst adds this property with the corresponding size.

  • stats [only]: When used as a host property (in one host only), configures hosts to supply statistics and runs prometheus and grafana on this node to gather stats and report them. If the value is only, the host is used just for statistics and DB components use other configured hosts.

  • tls: When used as a global property, makes the installed system use TLS for network exchanges between clients and the installed system.

  • user name: User used by the install Eg:

    user nemo
  • LXLDAP uri: URI used to reach the LDAP server for authentication. System users must still be created for LDAP users to be granted access. Defining this property changes the authentication method only. User lxadmin is never authenticated using LDAP.

    LXLDAP	ldap://ldapsrv:389|simple|ou=People,dc=leanxcale,dc=com
  • LXPAM yes: The PAM sudo service is used to authenticate user. Note that using sudo as a service requires the user to be able to read the file /etc/shadow, which is not a good idea.

  • logtrim mode: Mode can be no, ds (or persist), or backup. Sets the policy for removing old DB log transaction entries to: no removes, remove after persisted in data stores, or remove after backed up. Note that log entries may surive more than asked for, this is a limit for removal, not removal request.

2.2. Host properties

  • lxdir path: Directory used by the install in the host. E.g.:

    lxdir $HOME/xamplelx
  • lxpart device: Device for the partition used for the install. E.g.:

    lxpart /dev/sdc1
  • addr address: Network address used (without port).

2.3. Component properties

  • addr address: Network address used. E.g.:

    host atlantis
        kvms
            addr atlantis!14000
  • disk N: Size for the disk used by the component, using M or G as units or assuming GiB by default. E.g.:

    host atlantis
        kvds
            disk 50
        lxqe
            disk 500m
  • mem N: Size for the memory used by the component, using M or G as units or assuming MiB by default. E.g.:

    host atlantis
        kvds
            mem 500
        lxqe
            mem 1g
  • mirror nameid: Used for lxqe components. Specify the component for the mirror transaction log. E.g.:

    host atlantis
        lxqe 100
            mirror lxqe101
        lxqe 101
            mirror lxqe100
  • LXPULL addr1;addr2;…​;addrn: Used for lxqe components. Make this system pull transactions made on the query engines found at the given addresses. E.g.:

    host atlantis
        lxqe 100
            mirror lxqe101
        lxqe 101
            LXPULL orion!14420;orion!14424

2.4. Other properties

Properties with upper-case names are exported as environment variables for the process(es) executing the component involved. This includes both global properties and host properties as well. For example, to enable debug flags DOM for the kv datastore:

KVDEBUG DOM

These are other properties not described before.

  • mirror name: Defines a host as a mirror of the named host. Either all hosts must have mirrors or none of them can. (For command line arguments, the syntax is blade124+blade145). E.g.:

    host blade124
    host blade125
        mirror blade124
  • nodb: When specified as a host property, makes this host include just the lx command, but no DB component. E.g.:

    host atlantis
        nodb
  • scp template: Template for the scp command used to remotely copy files to a host. Use '{user}' and '{host}' where the user and host address should be included, and {src} ` and ` {dst} where the source and destination file names should be placed. E.g.:

    scp scp {src} {user}@{host}:{dst}
  • scpi template: Template for the (internal) scp command an installed host should use to reach other installed hosts. Use the same syntax described above. E.g.:

    scpi scp {src} {user}@{host}:{dst}
  • spread no|port: Port used by the spread process. Use no to disable it. E.g.:

    spread 4444
  • ssh template: Template for the ssh command used to execute remote commands on the host. Use {user}' and `{host} where the user and host names should be used. Eg.:

    ssh ssh -o StrictHostKeyChecking=no {user}@{host}
  • sshi template: Template for the (internal) ssh command an installed host should use to reach other installed hosts. Use the same syntax describe above. Eg.:

    sshi ssh -o StrictHostKeyChecking=no {user}@{host}
  • web [addr]: Run a web server as an interface for running Lx commands.