Installation Guide

1. Introduction

LeanXcale is an ultrascalable distributed database. It is built of several components that can scale out to cope with your workload. To get into the details and try to take the most of the deployment, you should read the Architecture and Concepts documents.

You can install LeanXcale in Ubuntu, CentOS, or Red Hat (RHEL). We’ve tested this Install Guide with Ubuntu 20.04 LTS, Ubuntu 18.04 LTS and Red Hat Enterprise Linux 8, but it should work in any other modern Linux distribution too.

2. Installing LeanXcale

2.1. Prerequisites

In order for the installation to be able to execute, all the machines in the cluster must meet some prerequisites. These are summarized as follows:

  • You need the following (very common) components to be installed:

    • Python 3 (>= 3.6) including pip3. pip3 will later be used by the deployment scripts to install the python dependencies for some tools.

    • Ansible >= 2.9

    • JAVA release 8

    • bash

    • netcat

    • libpthread

In multi-node setups, the Master node of LeanXcale needs to be able to communicate with the rest of nodes by SSH. There are two ways to do it, which are described in the Appendix A. If you’re using a single-node setup, you can skip this step.

You’ll also need to create a directory in a place where the user that’s going to start the database has rwx permissions. The directory needs to be created in all the machines that will conform the cluster:

user@host:~$ mkdir lxs

2.1.1. Ubuntu 20.04

// You have to run this commands in all the nodes of the cluster

sudo apt update
sudo apt install ansible python3-psutil python3-pip openjdk-8-jre-headless

2.1.2. Ubuntu 18.04

// You have to run this commands in all the nodes of the cluster

sudo apt update
sudo apt install ansible python-psutil python3-pip openjdk-8-jre-headless
pip3 install --upgrade pip

2.1.3. Red Hat Enterprise Linux 8

// You have to run this commands in all the nodes of the cluster

sudo yum -y install python3 python3-devel gcc nmap-ncat java-1.8.0-openjdk-headless
pip3 install ansible --user

2.2. Unpacking

The first step for installation is choosing a machine from the cluster to be the Master (the orchestrator of the cluster). On the chosen machine, change to the directory you just created and decompress the LeanXcale binaries there:

user@host:~$ cd lxs
user@host:~/lxs$ tar xvf ../LeanXcale_1.5.4_latest.tgz

Before running any admin command, we advice to execute the env.sh script that you’ll find on the root of the installation folder, because there we set some useful variables and aliases that you’ll find very useful when you manage your LeanXcale instance:

# Change to the directory where you installed LeanXcale first
user@host:~ cd lsx
user@host:~/lxs$ source ./env.sh

We recommend that you put this line in your .bashrc so it gets executed each time you login or boot your machine:

source ~/lxs/env.sh ~/lxs

Now the master server of the cluster is ready to be configured so you can later deploy the installation to the rest of the servers.

2.3. Simple Configuration: the inventory file

For configuring the cluster, the basic information you need to know is just the hostname of the machines you want to deploy to.

With that information, you can just configure the servers of the cluster in the inventory file:

user@host:~/lxs$ vim conf/inventory

Following you can see an example of a simple configuration file:

[all:vars]
BASEDIR="{{ lookup('env','BASEDIR') }}"
INSTALLATION_BASEDIR="{{ lookup('env','BASEDIR') }}"
INSTANCENAME="myinstance"
MASTERALERTHTTP="http://localhost:5001/api/v1/alerts"

BACKUPDIR=/path/to/leanxcale/backup/directory/

#Login account for the cluster
USER="{{ lookup('env','USER') }}"

# If {{USER}} has SUDO permissions, add SUDO=sudo. Otherwise you may need to
# create some folders and grant access to the USER for some actions to work
SUDO=no

# If you want to use NUMA, set it to yes
NUMA=no

HA=no
HAPROXY=yes

[defaults]
# The following line tells the configuration SCRIPT to get the values from the
# machine. All resources will be allocated to LeanXcale
# sizing mem=10 sockets=0-0 ncores=4 nthreadscore=2 fsdata=. fslog=.

# Metadata servers
[meta]
172.31.70.204 ansible_connection=local

#Datastore Servers. You can have multiple data store instances
[datastores]
172.31.70.204 ansible_connection=local
172.31.65.72 ansible_connection=ssh
172.31.35.18 ansible_connection=ssh

2.4. Completing the Installation

Once you have finished changing the configuration file, you can complete the installation. This last step will build your detailed configuration file including the resource allocation, and deploy the installation to the rest of the nodes of the cluster.

If you have problems in this step, please give a look to the Troubleshooting section.

user@host:~/lxs$ admin/deploycluster.sh

After a while, in the logs of the process printed on screen you’ll see the result. You should get failed=0 as the result for every host:

[...]
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
172.31.35.18   : ok=15   changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
172.31.65.72   : ok=15   changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
172.31.70.204  : ok=18   changed=7    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

From this point the cluster is installed and set up to be started. You can now go to the Operations manual and read how to run the database cluster and check everything is running OK.

2.5. Default Resource Allocation

The line starting with sizing defines the default resource allocation. The line assumes that all machines in the cluster have the same HW configuration. However, the parameters in the line could be defined for each machine if you need to override the default sizing. The line has the following parameters:

  • mem: Memory available in the machine to run the components (in GB). If not set it defaults to take all the memory in each machine and distribute it for LeanXcale.

  • sockets: Lists of sockets in the machine to be used to run the components. Values like 0-2 or 0,1,2 or 0,4 are allowed. In DataStore machines, there should be two socket lists definition: The socket list for KVDS and the socket list for QE. Those list must be separated by colon ':'

    Again, if not set all sockets in the machine will be used.

  • ncores: This is the number of physical cores to be used in each machine to run LeanXcale components. Take care that this is physical cores so hyperthreads should not be counted on.

  • nthreadscore: Number of hyperthreads per core

  • fslog: Folder where the transaction logging files are located. These are not the components logs, but the transaction logging that guarantees database durability. If SUDO is not granted, the folder with RW permissions for `USER`should have been created by the administrator

  • fsdata: Parent Folder where database is storing data. If section [DSFs] is available, then those folders will be used.

It is highly recommended separating transaction redo logging into different disks from database data.

An example of this configuration is the following:

...
sizing mem=128G sockets=0-1 ncores=12 nthreadscore=0 fslog=/txnlogs fsdata=/lxdata
...

This means: The machines have 128GB available and 2 sockets (0 to 1), there are 12 physical cpus in total and there is no hyperthreading.

Then, redo logging will be written to the filesystem mounted at /txnlogs and data will go to /lxdata.

2.6. Configuration

The default configuration is based on some workload assumptions that may not be met by your application, so resources may not be making the most of your resources with this configuration.

If you know about your workload and want to have a better deployment to better use of the resources you may want to read the rest of the document and refine your inventory file before going to step 5.

There are also some basic options for using different filesystems covered in next section.

2.6.1. Some more Points about Installation

So far you have done a very simple installation. However, there are at least a couple of points you must consider to do a good use of your resources.

In the inventory, you may have noticed a line having:

sizing mem=- sockets=- ncores=- nthreadscore=- fsdata=. fslog=.

This tells the installation process to use all memory, all sockets, all cores and use the same filesystems as used for BASEDIR, but you can change those easily to fit your needs (mainly of you don’t want to allocate all resources to LeanXcale).

The most important part is the filesystem definition. For high performance It is highly recommended to isolate the IO for the transaction logging from the IO of the datastores, therefore you should have different filesystems in different disks.

  • mem: Memory available in the machine to run LeanXcale components (in GB)

  • sockets: Lists of sockets in the machine to be used to run the components. Values like "0-2" or "0,1,2" or "0,4" are allowed.

  • ncores: This is the number of physical cores to be used in each machine to run LeanXcale components. Take care that this is physical cores so hyperthreads should not be counted on.

  • fslog: Folder where the transaction logging files are located. These are not the components logs, but the transaction logging that guarantees database durability. If SUDO is not granted, the folder with RW permissions for USER should have been created by the administrator

  • fsdata: Parent Folder where database is storing data.

This might be a resulting configuration:

sizing mem=64G sockets=0,1 ncores=- nthreadscore=- fsdata=/disk1/lxdata
fslog=/disk2/lxtxlog

There are few more points you can fine tune to improve the allocation of resources and the performance of your distributed database. For full information on the installation options, please refer to the Installation and Configuration Guide.

2.7. Default Component Deployment

Components are typically classified into 2 classes:

  • Metadata components:

    • Zookeeper

    • Configuration Manager

    • Snapshot Server

    • Commit Sequencer

    • KiVi Metadata Server

  • Datastore components:

    • Query Engine

    • Transactional Loggers

    • Kivi Datastore Server

  • Conflict Managers: This are not in either of the two classes, but for the sake of simplicity are - by default - deployed with metadata components unless High Availability is set up.

Giving this classification, the metadata components are deployed in the metadata server while datastore components will be distributed considering the number of datastore servers and their HW capacity.

2.8. Advanced Configuration

There are quite a few detail parameters that can be set to fine tune your installation. Those parameters are further discussed in [Annex I. Advanced Configuration].

  • Resource allocation

  • Filesystems

  • Detailed component deployment

3. High Availability

LeanXcale can be set-up as a high availability cluster. High availability means that in case of a single component failure or single machine failure the system will keep working non-stop.

You can further define the configuration parameters so the system can keep working in case of two-machine failure or even stronger high availability situations.

3.1. Configuring High Availability

Configuring High Availability is pretty simple. It just requires that you set the following configuration parameter in the inventory file:

HA=yes

3.2. Implications

The implications of configuring this parameter are:

  • The install SCRIPT will check that there are at least three machines configured for the cluster.

There has to be at least 2 machines defined as metadata servers and 2 machines defined as datastore servers. One machine can act both as metadata and datastore and that’s the reason there has to be at least 3 machines in the cluster.

The following configuration could be a valid HA configuration:

[all:vars]
BASEDIR="{{ lookup('env','BASEDIR') }}"
BACKUPDIR=/path/to/leanxcale/backup/directory/
USER="{{ lookup('env','USER') }}"

HA=yes
netiface=eth0

[defaults]
sizing mem=- sockets=- ncores=- nthreadscore=- fsdata=. fslog=.

[meta]
metaserver1
dataandmetaserver1

[datastores]
dataandmetaserver1
dataserver2

  • Components will be installed as follows:

    • Zookeeper: Zookeeper is the master for keeping the global configuration, health information and arbitration. The install SCRIPT will configure a Zookeeper cluster with three Zookeeper members replicated in different machines.

    • Transaction Loggers: The loggers are components that log all transaction updates (called writesets) to persistent storage to guarantee durability of transactions. The loggers will be configured and replicated in groups so there is a different logger component on a different machine. Therefore the logging information will be replicated in order to recover information in case of a crash.

    • Conflict Manager: Conflict Managers are in charge of detecting write-write conflicts among concurrent transactions. They are high available "per se", because If one Conflict Manager fails the conflict key buckets are transferred to another Conflict Manager. No special configuration is needed for High Availability except guaranteeing that there are at least 2 Conflict Managers configured in 2 different machines.

    • Commit Sequencer: The commit sequencer is in charge of distributing commit timestamps to the Local Transactional Managers. Two Commit Sequencers will be configured, but only one will be acting as master and the other as follower. The follower will take over in case the master fails.

    • Snapshot Server: The Snapshot Sever provides the most fresh coherent snapshot on which new transactions can be started. As in the case of the Commit Sequencer, 2 snapshot servers will be started one will be acting as master and the other as follower. The follower will take over in case the master fails.

    • Configuration Manager: The configuration manager handles system configuration and deployment information. It also monitors the other components. Two configuration managers will be started and one will be the master configuration manager.

    • Query Engine: The Query Engine parses SQL queries and transforms them into a query plan which will derive in a set of actions to the datastores. The query engines are usually configured in a cluster in which any of them can take part of the load so High availability has no special requirement on them.

    • Datastores: There will usually be several datastores in different machines. The important point in datastores is replication.

3.3. Replication

So far, High availability configuration ensures the components are configured in a way that no component failure or machine crash will cause the system to fail.

While components can be available data replication is about data availability so, If a machine or a disk crashes there is another copy of the data so you can keep working on that copy of the data. Data replication can be enabled regardless whether you want also to have high availability or not.

LeanXcale provides a full synchronous replication solution and you can configure as many replicas as you want. Basically, you can define a mirroring datastore configuration in the inventory file:

...
[datastores]
mirror-1:
  server1
  server2
mirror-2:
  server3
  server4

This doesn’t mean every server needs to be mirrored. You may also have unmirrored servers to keep data that doesn’t need to be replicated.

Besides, there are 2 kinds of replicas:

  • Replicas to minimize the risk of losing data and being able to recover service as soon as possible.

  • Replicas to improve performance. This are usually small tables that are used really frequently (usually read) so the application can benefit from having several copies that can be read in parallel.

4. Authentication, SSL & Permissions

There are two options to configure LeanXcale authentication

  • LDAP based authentication. You can set up an LDAP server just for LeanXcale, but this is most interesting for integrating LeanXcale in your organization’s LDAP (or Active Directory) and provide some kind of Single Sign On or at least the chance to use a common password within the organization.

  • Open shared access. This is not to be used in production except for shared data. Access level can be set in the firewall based on IP rules, but all users accessing will be granted access and they will be able to use a user schema. This is very easy to set-up for development and testing environments.

Permissions are granted through roles.

Communications can be configured so SSL is used for connections between components though It is recommended to run all components behind a firewall and use JDBC over SSL and the clients' connections over SSL.

5. Appendix A: setting up connections between nodes of a cluster

The Master node of LeanXcale needs to be able to communicate with the rest of nodes. This can be done in two ways: using SSH keys at the operating system level or using them within the LeanXcale inventory.

5.1. Using SSH Key Based Access at the OS

This is a standard OS mechanism in which you configure SSH keys among the hosts in the cluster so once the user has access to host-k he is trusted to access any other host in the cluster with the same user.

The right SSH keys should be set for the user starting the cluster, so it can get access to any machine in the cluster without any authentication mechanism. This way, LeanXcale’s administration scripts can deploy, configure, start and stop being run from one host and take action on all the hosts without having to ask for passwords for each action.

5.2. Using key files in the LeanXcale inventory

This method is similar, but the key can be set up to be used only by ansible.

The steps for setting up this are:

  • Generate a key pair

    user@host:~/lxs$ ssh-keygen -t rsa -b 4096 -f lx.pem

    You have to leave the passphrase empty. This will generate 2 files (lx.pem and lx.pem.pub).

  • Set all the hosts to autorize access with the key. For every host in the cluster do:

    user@host:~/lxs$ ssh-copy-id -i lx.pem.pub user@anotherhost

  • Then configure the inventory file to use this key file. For that, go to BASEDIR/conf/inventory and add the following line:

    ...
ansible_ssh_private_key_file=/lxs/lx.pem

[defaults]
...

5.3. Checking SSH Connectivity

You can check whether ansible is able to connect to the nodes you configured in the inventory by running:

user@host:~/lxs$ ansible -i $BASEDIR/conf/inventory -m ping datastores

If the connectivity is OK, you should see a message like this from all the nodes:

54.196.158.227 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

6. Appendix B. Troubleshooting

This section covers some common errors that may happen while trying to install LeanXcale.

6.1. admin/deploycluster.sh fails

  • Check connectivity as explained in the previous section and solve connectivity issues if there were any.

  • If there were no connectivity issues, deploycluster.sh also installs the python dependencies that the operation scripts need. It does so by using pip3 and, sometimes, pip3 is installed in a way that fail when run by ansible.

    Connect to the hosts failing and do:

    user@host:~/lxs$ sudo su -
root@host:/ pip3 install psutil ansible docker netiface pyjolokia
root@host:/ pip3 install prometheus_client sortedcontainers bloom_filter

After taking these actions, run again admin/deploycluster.sh