Installation Guide

1. Introduction

LeanXcale is an ultrascalable distributed database. It is built of several components that can scale out to cope with your workload. To get into the details and try to take the most of the deployment, you should read the Architecture and Concepts documents.

You can install LeanXcale in Ubuntu, CentOS, or Red Hat (RHEL). We’ve tested this Install Guide with Ubuntu 20.04 LTS, Ubuntu 18.04 LTS and Red Hat Enterprise Linux 8, but it should work in any other modern Linux distribution too.

2. Installing LeanXcale

2.1. Prerequisites

In order for the installation to be able to execute, all the machines in the cluster must meet some prerequisites. These are summarized as follows:

You need the following (very common) components to be installed:
- Python >= 2.7
- Ansible >= 2.6
- JAVA release 8
- bash
- netcat
- libpthread

In multi-node setups, the Master node of LeanXcale needs to be able to communicate with the rest of nodes by SSH. There are two ways to do it, which are described in the Appendix A. If you’re using a single-node setup, you can skip this step.

You’ll also need to create a directory in a place where the user that’s going to start the database has rwx permissions. The directory needs to be created in all the machines that will conform the cluster:

user@host:~$ mkdir lxs

2.1.1. Ubuntu 20.04

// You have to run this commands in all the nodes of the cluster

sudo apt update
sudo apt install ansible python3-psutil python3-pip openjdk-8-jre-headless

2.1.2. Ubuntu 18.04

// You have to run this commands in all the nodes of the cluster

sudo apt update
sudo apt install ansible python-psutil python3-pip openjdk-8-jre-headless
pip3 install --upgrade pip

2.1.3. Red Hat Enterprise Linux 8

// You have to run this commands in all the nodes of the cluster

sudo yum -y install python3 python3-devel gcc nmap-ncat java-1.8.0-openjdk-headless
pip3 install ansible --user

2.2. Unpacking

The first step for installation is choosing a machine from the cluster to be the Master (the orchestrator of the cluster). On the chosen machine, change to the directory you just created and decompress the LeanXcale binaries there:

user@host:~$ cd lxs
user@host:~/lxs$ tar xvf ../LeanXcale_1.6.6_latest.tgz

Before running any admin command, we advice to execute the env.sh script that you’ll find on the root of the installation folder, because there we set some useful variables and aliases that you’ll find very useful when you manage your LeanXcale instance:

# Change to the directory where you installed LeanXcale first
user@host:~ cd lsx
user@host:~/lxs$ source ./env.sh

We recommend that you put this line in your .bashrc so it gets executed each time you login or boot your machine:

source ~/lxs/env.sh ~/lxs

Now the master server of the cluster is ready to be configured so you can later deploy the installation to the rest of the servers.

2.3. Simple Configuration: the inventory file

For configuring the cluster, the basic information you need to know is just the hostname of the machines you want to deploy to.

With that information, you can just configure the servers of the cluster in the inventory file:

user@host:~/lxs$ vim conf/inventory

Following you can see an example of a simple configuration file:

[all:vars]
BASEDIR="{{ lookup('env','BASEDIR') }}"
INSTALLATION_BASEDIR="{{ lookup('env','BASEDIR') }}"
INSTANCENAME="myinstance"
MASTERALERTHTTP="http://localhost:5001/api/v1/alerts"

BACKUPDIR=/path/to/leanxcale/backup/directory/

#Login account for the cluster
USER="{{ lookup('env','USER') }}"

# If {{USER}} has SUDO permissions, add SUDO=sudo. Otherwise you may need to
# create some folders and grant access to the USER for some actions to work
SUDO=no

# If you want to use NUMA, set it to yes
NUMA=no

HA=no
HAPROXY=yes

[defaults]
# The following line tells the configuration SCRIPT to get the values from the
# machine. All resources will be allocated to LeanXcale
# sizing mem=10 sockets=0-0 ncores=4 nthreadscore=2 fsdata=. fslog=.

# Metadata servers
[meta]
172.31.70.204 ansible_connection=local

#Datastore Servers. You can have multiple data store instances
[datastores]
172.31.70.204 ansible_connection=local
172.31.65.72 ansible_connection=ssh
172.31.35.18 ansible_connection=ssh

2.4. Completing the Installation

Once you have finished changing the configuration file, you can complete the installation. This last step will build your detailed configuration file including the resource allocation, and deploy the installation to the rest of the nodes of the cluster.

user@host:~/lxs$ admin/deploycluster.sh

After a while, in the logs of the process printed on screen you’ll see the result. You should get failed=0 as the result for every host:

[...]
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
172.31.35.18   : ok=15   changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
172.31.65.72   : ok=15   changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
172.31.70.204  : ok=18   changed=7    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

From this point the cluster is installed and set up to be started.

2.5. Default Resource Allocation

The line starting with sizing defines the default resource allocation. The line assumes that all machines in the cluster have the same HW configuration. However, the parameters in the line could be defined for each machine if you need to override the default sizing. The line has the following parameters:

mem: Memory available in the machine to run the components (in GB). If not set it defaults to take all the memory in each machine and distribute it for LeanXcale.
sockets: Lists of sockets in the machine to be used to run the components. Values like 0-2 or 0,1,2 or 0,4 are allowed. In DataStore machines, there should be two socket lists definition: The socket list for KVDS and the socket list for QE. Those list must be separated by colon ':'

Again, if not set all sockets in the machine will be used.
ncores: This is the number of physical cores to be used in each machine to run LeanXcale components. Take care that this is physical cores so hyperthreads should not be counted on.
nthreadscore: Number of hyperthreads per core
fslog: Folder where the transaction logging files are located. These are not the components logs, but the transaction logging that guarantees database durability. If SUDO is not granted, the folder with RW permissions for `USER`should have been created by the administrator
fsdata: Parent Folder where database is storing data. If section [DSFs] is available, then those folders will be used.

It is highly recommended separating transaction redo logging into different disks from database data.

An example of this configuration is the following:

...
sizing mem=128G sockets=0-1 ncores=12 nthreadscore=0 fslog=/txnlogs fsdata=/lxdata
...

This means: The machines have 128GB available and 2 sockets (0 to 1), there are 12 physical cpus in total and there is no hyperthreading.

Then, redo logging will be written to the filesystem mounted at /txnlogs and data will go to /lxdata.

2.6. Configuration

The default configuration is based on some workload assumptions that may not be met by your application, so resources may not be making the most of your resources with this configuration.

If you know about your workload and want to have a better deployment to better use of the resources you may want to read the rest of the document and refine your inventory file before going to step 5.

There are also some basic options for using different filesystems covered in next section.

2.6.1. Some more Points about Installation

So far you have done a very simple installation. However, there are at least a couple of points you must consider to do a good use of your resources.

In the inventory, you may have noticed a line having:

sizing mem=- sockets=- ncores=- nthreadscore=- fsdata=. fslog=.

This tells the installation process to use all memory, all sockets, all cores and use the same filesystems as used for BASEDIR, but you can change those easily to fit your needs (mainly of you don’t want to allocate all resources to LeanXcale).

The most important part is the filesystem definition. For high performance It is highly recommended to isolate the IO for the transaction logging from the IO of the datastores, therefore you should have different filesystems in different disks.

mem: Memory available in the machine to run LeanXcale components (in GB)
sockets: Lists of sockets in the machine to be used to run the components. Values like "0-2" or "0,1,2" or "0,4" are allowed.
ncores: This is the number of physical cores to be used in each machine to run LeanXcale components. Take care that this is physical cores so hyperthreads should not be counted on.
fslog: Folder where the transaction logging files are located. These are not the components logs, but the transaction logging that guarantees database durability. If SUDO is not granted, the folder with RW permissions for USER should have been created by the administrator
fsdata: Parent Folder where database is storing data.

This might be a resulting configuration:

sizing mem=64G sockets=0,1 ncores=- nthreadscore=- fsdata=/disk1/lxdata fslog=/disk2/lxtxlog

There are few more points you can fine tune to improve the allocation of resources and the performance of your distributed database. For full information on the installation options, please refer to the Installation and Configuration Guide.

2.7. Default Component Deployment

Components are typically classified into 2 classes:

Metadata components:
- Zookeeper
- Configuration Manager
- Snapshot Server
- Commit Sequencer
- KiVi Metadata Server
Datastore components:
- Query Engine
- Transactional Loggers
- Kivi Datastore Server
Conflict Managers: This are not in either of the two classes, but for the sake of simplicity are - by default - deployed with metadata components unless High Availability is set up.

Giving this classification, the metadata components are deployed in the metadata server while datastore components will be distributed considering the number of datastore servers and their HW capacity.

2.8. Advanced Configuration

There are quite a few detail parameters that can be set to fine tune your installation. Those parameters are further discussed in [Annex I. Advanced Configuration].

Resource allocation
Filesystems
Detailed component deployment

3. High Availability

LeanXcale can be set-up as a high availability cluster. High availability means that in case of a single component failure or single machine failure the system will keep working non-stop.

You can further define the configuration parameters so the system can keep working in case of two-machine failure or even stronger high availability situations.

3.1. Configuring High Availability

Configuring High Availability is pretty simple. It just requires that you set the following configuration parameter in the inventory file:

HA=yes

3.2. Implications

The implications of configuring this parameter are:

The install SCRIPT will check that there are at least three machines configured for the cluster.

There has to be at least 2 machines defined as metadata servers and 2 machines defined as datastore servers. One machine can act both as metadata and datastore and that’s the reason there has to be at least 3 machines in the cluster.

The following configuration could be a valid HA configuration:

[all:vars]
BASEDIR="{{ lookup('env','BASEDIR') }}"
BACKUPDIR=/path/to/leanxcale/backup/directory/
USER="{{ lookup('env','USER') }}"

HA=yes
netiface=eth0

[defaults]
sizing mem=- sockets=- ncores=- nthreadscore=- fsdata=. fslog=.

[meta]
metaserver1
dataandmetaserver1

[datastores]
dataandmetaserver1
dataserver2

Components will be installed as follows:
- Zookeeper: Zookeeper is the master for keeping the global configuration, health information and arbitration. The install SCRIPT will configure a Zookeeper cluster with three Zookeeper members replicated in different machines.
- Transaction Loggers: The loggers are components that log all transaction updates (called writesets) to persistent storage to guarantee durability of transactions. The loggers will be configured and replicated in groups so there is a different logger component on a different machine. Therefore the logging information will be replicated in order to recover information in case of a crash.
- Conflict Manager: Conflict Managers are in charge of detecting write-write conflicts among concurrent transactions. They are high available "per se", because If one Conflict Manager fails the conflict key buckets are transferred to another Conflict Manager. No special configuration is needed for High Availability except guaranteeing that there are at least 2 Conflict Managers configured in 2 different machines.
- Commit Sequencer: The commit sequencer is in charge of distributing commit timestamps to the Local Transactional Managers. Two Commit Sequencers will be configured, but only one will be acting as master and the other as follower. The follower will take over in case the master fails.
- Snapshot Server: The Snapshot Sever provides the most fresh coherent snapshot on which new transactions can be started. As in the case of the Commit Sequencer, 2 snapshot servers will be started one will be acting as master and the other as follower. The follower will take over in case the master fails.
- Configuration Manager: The configuration manager handles system configuration and deployment information. It also monitors the other components. Two configuration managers will be started and one will be the master configuration manager.
- Query Engine: The Query Engine parses SQL queries and transforms them into a query plan which will derive in a set of actions to the datastores. The query engines are usually configured in a cluster in which any of them can take part of the load so High availability has no special requirement on them.
- Datastores: There will usually be several datastores in different machines. The important point in datastores is replication.

3.3. Replication

So far, High availability configuration ensures the components are configured in a way that no component failure or machine crash will cause the system to fail.

While components can be available data replication is about data availability so, If a machine or a disk crashes there is another copy of the data so you can keep working on that copy of the data. Data replication can be enabled regardless whether you want also to have high availability or not.

LeanXcale provides a full synchronous replication solution and you can configure as many replicas as you want. Basically, you can define a mirroring datastore configuration in the inventory file:

...
[datastores]
mirror-1:
  server1
  server2
mirror-2:
  server3
  server4

This doesn’t mean every server needs to be mirrored. You may also have unmirrored servers to keep data that doesn’t need to be replicated.

Besides, there are 2 kinds of replicas:

Replicas to minimize the risk of losing data and being able to recover service as soon as possible.
Replicas to improve performance. This are usually small tables that are used really frequently (usually read) so the application can benefit from having several copies that can be read in parallel.

4. Authentication, SSL & Permissions

There are two options to configure LeanXcale authentication

LDAP based authentication. You can set up an LDAP server just for LeanXcale, but this is most interesting for integrating LeanXcale in your organization’s LDAP (or Active Directory) and provide some kind of Single Sign On or at least the chance to use a common password within the organization.

Open shared access. This is not to be used in production except for shared data. Access level can be set in the firewall based on IP rules, but all users accessing will be granted access and they will be able to use a user schema. This is very easy to set-up for development and testing environments.

Permissions are granted through roles.

Communications can be configured so SSL is used for connections between components though It is recommended to run all components behind a firewall and use JDBC over SSL and the clients' connections over SSL.

4.1. TLS for HAProxy

Securing communication between clients and LeanXcale server is enabled activating TLS options during deployment of cluster server.

If server deployment includes HAProxy then clients-HAProxy links may be secured, but intra-cluster HAProxy-QE links will not. In contrast, direct clients-QE links may be secured in absence of HAProxy. That is, securing QE and securing HAP are mutually exclusive.

To init the HAProxy with TLS activated (HTTPS) you need to:

Add this line in the [all:vars] section of the inventory.
```
SECURE_COMUNICATION_HAP = {"certificate": "/path/to/lxs-basedir/conf/server.pem"}
```
Certificate file server.pem to be always in $BASEDIR/conf.
This options is not compatible with SECURE_COMUNICATION_QE.
Defined that value, the Ansible-Installer adds security option to haproxy.cfg and HAProxy will start with TLS activated. HAPROXY=yes is not needed in inventory file.

To create server.pem you may need to run these steps:

# Generate new private/public key pair if a trusted pair from a trusted CA is not available
# Option -nodes enable HAProxy run up without prompting for a password
openssl req -x509 -newkey rsa:2048 -sha256 -days 3650 -nodes \
  -keyout private.key -out client.crt \
  -subj "/CN=leanxcale.com/OU=IT Dep./O=LeanXcale/L=Madrid/ST=Madrid/C=ES" \
  -addext "subjectAltName=DNS:leanxcale.com,DNS:localhost,IP:127.0.0.1"
# Bundle Private_Key/Public_certificate into server.pem
cat private.key client.crt > server.pem

The subjectAltName option is a comma-list of server names, IPs or domains as accessed by clients. A typical production sample would be -addext "subjectAltName=DNS:leanxcale.com,IP:10.0.0.10".

To be sure that TLS has been configured and HTTPS is activated for the HAP communication:

curl -X OPTIONS --insecure https://localhost:1522

Note port 1522 for connecting to database.

4.2. TLS for Query Engine

To init the QE with TLS activated (HTTPS) you need to:

Add this line in the [all:vars] section of the inventory. The password can be obfuscate to avoid having clear passwords in the inventory.
```
SECURE_COMUNICATION_QE = {"keystore": "/path/to/lxs-basedir/conf/keystore.jks", "truststore": "/path/to/lxs-basedir/truststore.ts", "keystore_password": "OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v", "truststore_password": "OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v"}
```
Store files Keystore.jks/truststore.ts to be always in $BASEDIR/conf.
This options is not compatible with SECURE_COMUNICATION_HAP.
If all these values are defined, the Ansible-Installer adds the QE properties to the leanxcale-site.xml and starts the HTTPServer with TLS activated.

To create the keystore and the truststore you need to run the steps of HAProxy section to generate server.pem. Due to Jetty limitations, -addext option must not be run and hostname is specified in -subj "/CN=<hostname>/…"

Additionally:

# Export private/public pair server.pem into PKCS12 store with PASSWORD
openssl pkcs12 -export -name lxqeserver -in server.pem -out keystore.p12 -password pass:PASSWORD
# Import PKCS12 into java keystore
keytool -importkeystore -srcalias lxqeserver \
  -srckeystore keystore.p12 -srcstorepass PASSWORD \
  -destkeystore keystore.jks -deststorepass PASSWORD
# Import the client certificate in the truststore
keytool -importcert -v -noprompt -trustcacerts -alias lxqeserver \
  -file client.crt -keystore truststore.ts -deststorepass PASSWORD
# with this command you can get the obfuscated password
java -cp $BASEDIR/LX-BIN/lib/jetty-util-9.4.28.v20200408.jar org.eclipse.jetty.util.security.Password PASSWORD

To be sure that TLS has been configured and HTTPS is activated for the QE communication:

curl -X OPTIONS --insecure https://localhost:1529

Note port 1529 for connecting to database.

4.3. TLS for clients

The client side (lxClient, Squirrel, Python…) will need the deployment of the TLS certificate if it is auto-generated, not from trusted Certificate Authority. Usually, this is done adding it to the OS root CA certificates or to a Java truststore.

Add certificate to root CA certificates in Ubuntu OS:

sudo mkdir /usr/local/share/ca-certificates/extra
sudo cp client.crt /usr/local/share/ca-certificates/extra/client.crt
sudo update-ca-certificates

Add certificate to root CA certificates in Windows OS:

Manage the certificate with app Microsoft Management Console. For an easy installation, double clicki in the .crt file and then select the Trusted Root Certification Authorities Store as the store for installing the certificate. It will show a warning about trusting the CA.
Add certificate to root CA certificates in Mac OS:

Manage the certificate with app Keychain Access.
Alternatively, Java clients may import client.crt to desired truststore in client machine
```
# import the certificate in truststore
keytool -importcert -v -trustcacerts -alias lxqeserver -file client.crt -keystore truststore.ts
```
To access to a truststore other than JVM default, exec Java with option -Djavax.net.ssl.trustStore=truststore.ts Option -Dcom.sun.net.ssl.checkRevocation=false enable java clients to work without certificate deployment but no server ID verification is done.

5. Appendix A: setting up connections between nodes of a cluster

The Master node of LeanXcale needs to be able to communicate with the rest of nodes. This can be done in two ways: using SSK keys at the operative system level or using them within the LeanXcale inventory.

5.1. Using SSH

While using this method, all the machines should have SSH connectivity between, and the right SSH keys should be set for the user starting the cluster, so it can get access to any machine in the cluster without any authentication mechanism. This way, our administration scripts can deploy, configure, start and stop remotely without having to ask for passwords for each action.

5.2. Using PEM files and the LeanXcale inventory

If you prefer this method, you should generate a PEM file for all the machines and put that PEM file in the inventory.

6. Appendix B. Troubleshooting

This section covers some common errors that may happen while trying to install LeanXcale.

6.1. admin/deploycluster.sh fails

Check connectivity as explained in the previous section and solve connectivity issues if there were any.
If there were no connectivity issues, deploycluster.sh also installs the python dependencies that the operation scripts need. It does so by using pip3 and, sometimes, pip3 is installed in a way that fail when run by ansible.

Connect to the hosts failing and do:
```
user@host:~/lxs$ sudo su -
root@host:/ pip3 install psutil ansible docker netiface pyjolokia
root@host:/ pip3 install prometheus_client sortedcontainers bloom_filter
```

After taking these actions, run again admin/deploycluster.sh