Deploy LeanXcale using Docker

1. Introduction

LeanXcale is an ultrascalable distributed database. It is built of several components that can scale out to cope with your workload.

Although optimal performance can be achieved using a bare metal installation, using Docker provides significant maintenance advantages and flexibility.

Since LeanXcale is a distributed solution, it can be deployed in many different ways, and you which setup is most efficient depends on the workload.

LeanXcale provides some interesting elasticity capabilities, but this is not currently automated. The LeanXcale team is working on an Elastic Manager that can automatically adapt your deployment to the workload.

Currently, deployment is based on a single Docker image in which you can start just one component or a set of components.

1.1. Prerequisites

This document assumes you have already downloaded the LeanXcale docker image and you are ready to run it in your machines.

1.2. Deployment Details

To get into the details and try to take the most of the deployment you should read the Architecture and Concepts documents and then go to the Docker Deployment Details section.

2. A Simple Deployment

In the following example, we will make a few tests over a small set of data with a simple deployment.

In order to keep things simple we will use four different configurations in one docker configuration, instead of use one docker for each component.

We should start the dockers in order. <lx_image_name> is the identifier of LeanXcale container.

  1. Metadata components docker.

First we run the docker image:

docker run -d <lx_image_name>

Then we start the components inside the docker:

docker exec -ti <container_name> bash -c "LX-BIN/scripts/startAll.sh 'ZK LgCmS MtM KVMS'"
The single and double quotes are important.

It is very important to get the IP assigned to this docker because it has to be informed to the rest of the dockers deployed

In the following, we will use {ZKMETA_IP} as the IP for this docker.

  1. We will run one docker with the Conflict Manager

docker run -d -e "ZK_FORCE={ZKMETA_IP}" -e "KVMS_FORCE={ZKMETA_IP}" <lx_image_name>

Let’s not forget to start the conflict manager process:

docker exec -ti <container_name> bash -c "LX-BIN/scripts/startAll.sh CflM-1"
  1. We will run 4 dockers with 4 datastore components (1 KVDS each). This is the core of the data distribution.

docker run -d -e "ZK_FORCE={ZKMETA_IP}" -e "KVMS_FORCE={ZKMETA_IP}" <lx_image_name>
docker exec -ti <container_name> bash -c "LX-BIN/scripts/startAll.sh KVDS-1"

docker run -d -e "ZK_FORCE={ZKMETA_IP}" -e "KVMS_FORCE={ZKMETA_IP}" <lx_image_name>
docker exec -ti <container_name> bash -c "LX-BIN/scripts/startAll.sh KVDS-1"

docker run -d -e "ZK_FORCE={ZKMETA_IP}" -e "KVMS_FORCE={ZKMETA_IP}" <lx_image_name>
docker exec -ti <container_name> bash -c "LX-BIN/scripts/startAll.sh KVDS-1"

docker run -d -e "ZK_FORCE={ZKMETA_IP}" -e "KVMS_FORCE={ZKMETA_IP}" <lx_image_name>
docker exec -ti <container_name> bash -c "LX-BIN/scripts/startAll.sh KVDS-1"
  1. We will run one docker with one Query Engine, and Transaction Logger:

docker run -d -e "ZK_FORCE={ZKMETA_IP}" -e "KVMS_FORCE={ZKMETA_IP}" <lx_image_name>

we start the services:

docker exec -ti <container_name> bash -c "LX-BIN/scripts/startAll.sh 'LgLTM-1 sleep-1 QE-1"

Now we have a a simple cluster up and ready.

The cluster set up in this section is quite small. With default parameters, this means around 68GB of memory and 8 machine cores.

3. Communication Ports

In order to connect with the database and allow that each component are visible to each other, you will need to open the following ports, over the following containers:

Container Type

Ports to open

Metadata components

2181, 13400, 13200, 13300, 44000

Conflict Manager

11100

KVDS

9992

Query Engine

13422, 1529

4. Planning your Deployment

For a very simple deployment a cluster is unnecessary, but typically, your implementations will require more advanced configurations. The right configuration will depend on your workload or size of your dataset.

For datasets, the important part is the size of your "active" dataset - the part of the data that is frequently accessed and needs good response times.

For example: Let’s assume you have an application that stores the trends of product costs over the last year. But really 99% of the requests are for the last month or less. So, If someone requests more than a month you can afford to answer with a longer time. Then, we could say that your active dataset is 1/12 of the dataset size.

4.1. Sample Layouts

In the following table, you can see some layout examples with numbers for typical resource allocation. We suggest you start with one that is close to your scenario:

Container Config Type

Memory

Tiny Layout

Small Layout

Medium Size Layout [cols=">,>,>,>,>",]

Query Engine

12GB

1

4

16

KVDS

12GB

4

16

64

Conflict Manager

8GB

1

2

8

Metadata components

1

1

1

Resources

Estimated CPU Cores

8

27

105

Active Dataset Size (GB)

48

192

768

Active % :: 50%:

Dataset Size (GB)

96

384

1536

Active % :: 10%:

Dataset Size (GB)

480

1920

7680

These typical layouts are suggestions providing as a good balance between memory and CPU consumption, not limits. For example, a KVDS can effectively manage much more memory than 128GB. Feel free to adjust the defaults when startin the dockers for any of the components.

4.2. Memory

In order to setup the memory of each component you can set it by sending to the container the following variables:

KVDSMEM_FORCE=12G
CFLMMEM_FORCE=2G
QEMEM_FORCE=4G

The calls for each container would look like this:

docker run -d -e "ZK_FORCE={ZKMETA_IP}" -e "KVMS_FORCE={ZKMETA_IP}" -e "KVDSMEM_FORCE=12G" <lx_image_name>
docker exec -ti <container_name> bash -c "LX-BIN/scripts/startAll.sh KVDS-1"
docker run -d -e "ZK_FORCE={ZKMETA_IP}" -e "KVMS_FORCE={ZKMETA_IP}" -e "KVDSMEM_FORCE=2G" <lx_image_name>
docker exec -ti <container_name> bash -c "LX-BIN/scripts/startAll.sh CflM-1"
docker run -d -e "ZK_FORCE={ZKMETA_IP}" -e "KVMS_FORCE={ZKMETA_IP}" -e "QEMEM_FORCE=4G" <lx_image_name>
docker exec -ti <container_name> bash -c "LX-BIN/scripts/startAll.sh 'LgLTM-1 sleep-1 QE-1"

5. Performance Recommendations

For best results, attach local volumes specifically to the containers for the datastores and the Query Engine, including the Transaction Logger.

The datastores are also built for taking advantage of NUMA capabilities. In order to take advantage of this, you should specially configure each datastore docker to run in a single physical core separate from any other docker.

Datastore cores have to be physical cores - hyperthreading cores are not real cores and will not work.

Other dockers should be configured not to run in the cores devoted for the datastore dockers.

6. Create Schema

Once you have your database up and running, it is time to create your schema. The best way is to use SQL DDL instructions over a SQL client. We have instructions to install the SQuirreL graphical client or you can use a command line based one like sqline.

Be careful to provide the following JDBC URL connection: jdbc:leanxcale://{ZKMETA_IP}:1522/testdb

appuser@docker:/lx/LX-BIN/bin$ lxClient
LeanXcale Client version 1.4
lx> connect 'jdbc:leanxcale://localhost:1522/testdb';
lx> CREATE TABLE PCAPS_RECORDS (
    --id_partition SMALLINT NOT NULL,
source_ip VARCHAR(32) NOT NULL,
timestamp TIMESTAMP NOT NULL,
destination_ip VARCHAR(255) NOT NULL,
source_port INTEGER,
destination_port INTEGER,
protocol SMALLINT,
packet VARCHAR(2000),
  --CONSTRAINT pk_pcap PRIMARY KEY(id_partition, source_ip, timestamp)
  CONSTRAINT pk_pcap PRIMARY KEY(source_ip, timestamp)
);> > > > > > > > > > >
0 rows inserted/updated/deleted
lx> show tables;
TABLE_SCHEM         |TABLE_NAME                    |REMARKS
------------------------------------------------------------------------
APP                 |PCAPS_RECORDS                 |
SYS                 |SYSALIASES                    |
SYS                 |SYSCHECKS                     |
SYS                 |SYSCOLPERMS                   |
SYS                 |SYSCOLUMNS                    |
SYS                 |SYSCONGLOMERATES              |
SYS                 |SYSCONSTRAINTS                |
SYS                 |SYSDEPENDS                    |
SYS                 |SYSFILES                      |
SYS                 |SYSFOREIGNKEYS                |
SYS                 |SYSKEYS                       |
SYS                 |SYSPERMS                      |
SYS                 |SYSROLES                      |
SYS                 |SYSROUTINEPERMS               |
SYS                 |SYSSCHEMAS                    |
SYS                 |SYSSEQUENCES                  |
SYS                 |SYSSTATEMENTS                 |
SYS                 |SYSSTATISTICS                 |
SYS                 |SYSTABLEPERMS                 |
SYS                 |SYSTABLES                     |
SYS                 |SYSTRIGGERS                   |
SYS                 |SYSUSERS                      |
SYS                 |SYSVIEWS                      |
SYSIBM              |SYSDUMMY1                     |

24 rows selected
lx> describe PCAPS_RECORDS;
COLUMN_NAME         |TYPE_NAME|DEC&|NUM&|COLUM&|COLUMN_DEF|CHAR_OCTE&|IS_NULL&
------------------------------------------------------------------------------
SOURCE_IP           |VARCHAR  |NULL|NULL|32    |NULL      |64        |NO
TIMESTAMP           |TIMESTAMP|9   |10  |29    |NULL      |NULL      |NO
DESTINATION_IP      |VARCHAR  |NULL|NULL|255   |NULL      |510       |NO
SOURCE_PORT         |INTEGER  |0   |10  |10    |NULL      |NULL      |YES
DESTINATION_PORT    |INTEGER  |0   |10  |10    |NULL      |NULL      |YES
PROTOCOL            |SMALLINT |0   |10  |5     |NULL      |NULL      |YES
PACKET              |VARCHAR  |NULL|NULL|2000  |NULL      |4000      |YES

7 rows selected
lx>

LeanXcale Query Engine was built from a fork of Apache Calcite, so you can check the Apache Calcite SQL Grammar if you have any doubt on how to do any SQL operation.

7. Partitioning for Data Distribution

Partitioning is a very important point. If you don’t distribute your data among the different datastores you won’t benefit from any distribution, you won’t be able to scale and won’t be benefiting from any parallelization and have any speed improvement.

7.1. The Importance of Keys

LeanXcale is based on distributing data ranges and has primary key clustered indexes. This means it is important that you select a good primary key because all queries for the primary keys will be using the clustered index. You can also have local secondary indexes that also have interesting properties.

8. Partitioning Strategies

For partitioning there are two strategies;

  • Uniform distribution

  • Calculated distribution

8.1. Split Tool

The python tool splitTableRange.py is used to create partitioning splits.

The following information must be provided:

  • Fully qualified table name; <DB_name-user_name-table_name>

  • Minimum value

  • Maximum value

  • Number of split points.

A split points value of 0 will automatically split evenly based on the number of KVDS process launched.

8.1.1. Syntax:

python splitTableRange.py <DB_name-user_name-table_name> <min> <max> <points>

8.2. Uniform Distribution

If you don’t know the statistical distribution of your data and don’t have a data set, the first approach is to do a uniform distribution over a range.

Example:

Let’s say that we had a table T_MSISDN_IP whose primary key is the MSISDN (the phone number) and we know we are going to feed information related with the MSISDN but the range of phone numbers is known.

python splitTableRange.py T_MSISDN_IP 600000000 699999999 0

8.3. Calculated Distribution

To create a calculated distribution, you can use a dataset and calculate the split points considering the data distribution of the data set in a CSV file.

For this purpose we provide the lxSplitFromCSV utility.

The tool analyzes data inside the CSV file. This analysis is based on the column or columns containing the primary key of the table.

Next, the tool creates the partitions needed to achieve the best performance possible based on the number of KVDS processes (containers) launched on your cluster.

8.4. Parameters

The lxSplitFromCSV utility needs the following parameters:

$> ./lxSplitFromCSV

usage: LX table split
 -c,--delimiter <arg>   CSV char delimiter
 -d,--database <arg>    database name
 -f,--file <arg>        path to CSV file
 -h,--header            flag specifying whether CSV has header row
 -j,--jdbcurl <arg>     database JDBC URL
 -k,--kiviurl <arg>     Kivi meta server URL. The proper format is kvmsHost!kvmsPort
 -r                     Use reservoir sampling (slower, but more reliable
                        with multi-line columns)
 -s,--schema <arg>      schema name
 -t,--table <arg>       table name

9. Populating Data

For populating data you can use the lxCSVLoad utility. This utility can be called from within an image. To access the CSV files easily, attach the volumes containing them.

/lx/LX-BIN/bin/lxCSVLoad

This utility will load data directly to the KiVi datastore in order to improve the loading phase performance.

Here are the input parameters needed by the lxCSVLoader:

$> ./lxCSVLoad
usage: lxCSVLoad
 -c,--connection_string <arg>      lxis connection string.The proper
                                   format is host:port.
 -t,--table <arg>                  Name of the table
 -f,--csv_file <arg>               Path to the csv file
 -y,--column_separator <arg>       [Column separator used on the csv file.
                                   Default '|']
  -ts,--timestamp <arg>             [Value of timestamp in case there isn''t
                                   a Zookeeper connection. Default value
                                   is 1]
 -v,--verbose                      [Verbose output]
 -th,--threads <arg>               [Number of writers threads. Default
                                   value is 5]
 -params,--params <arg>            [path of the file with the parameters
                                   for the loader]

Here is an example of invocation:

./lxCSVLoad -c 125.11.22.30:9876 -t tpcc-APP-ORDER_LINE -f /lx/LX-DATA/CSVs/1-20/order_line-1_20.csv &

Is a common situation have some tables bigger than others. To avoid this situation is a common practice to split down the CSV files and launch an instance of lxCSVLoader per CSV file, loading all of them to the same table.

10. Update Stats

In order to obtain the best query plan after you are done with the database data population you should update the statistics of the database.

This can be done by executing the following script in the Zookeeper container:

/lx/scripts/collect_usertable_stats.sh

You just need to run it and it will retrieve from Zookeeper all the information needed and launch sequentially the update statistics process.

11. Stopping

If you want your data to persist you should attach the dockers to external volumes so data is persisted and can be read again upon a new start of the docker.

Otherwise, you will have to start from scratch every time you stop the dockers.

If your data is being persisted, it is recommended that you stop all dockers properly.

To do so, run this command in the Metadata container:

docker exec -ti <Metadata_container_name> bash -c "admin/stopcluster.sh"

Then you can stop all the containers.

Don’t forget to reattach the volumes in the same way when you restart.

12. Docker Deployment Details

As you will see in the documentation our database have the following components. Here the component name followed by the component short name.

The component short name is used in all commands
  1. Zookeeper: ZK

  2. Logger CmS: LgCmS

  3. MasterMind: MtM

  4. Conflict Manager: CflM

  5. Datastore KiVi Meta server: KVMS

  6. Datastore KiVi Data server: KVDS-1

  7. Transactional Logger: LgLTM

  8. Query Engine: QE

This is the recommended component start order

By default all components start with the default configuration of memory to be used. But they can be changed in order to improve the performance.

The docker image will be the same for each component, so you will use just the one provided before.

In order to launch the database you will need to launch at least one instance of each component.

Zookeeper is used to coordinate the components available on the cluster, so the first step will be lunch this component.

You have to know the IP of the container where zookeeper and Datastore KiVi Meta server will be running because the rest of the containers will use it. Or you just can use the container name as zk_node and KVMS_node and they will be auto-configured.

12.1. Command syntax

This is the generic command to launch a container:

docker run -d -e "ZK_FORCE=<zookeeper_container_IP>" -e "KVMS_FORCE=<kvms_container_IP>" <lx_image_name>

The generic command to launch components:

docker exec -ti <container_name> bash -c "LX-BIN/scripts/startAll.sh <component_short_name>"

12.2. Multiple instances

To increase performance, you can launch as many instances as needed of the following components:

  1. Conflict Manager: CflM

  2. Datastore KiVi Data server: KVDS-1

  3. Transactional Logger: LgLTM

  4. Query Engine: QE

12.3. Important performance concerns

  • When using KVDS containers co-located on the same machine, attach persistent volumes - this will reduce the I/O latency.

  • Processor and and memory resources for Conflict Managers should be based on the expected amount of simultaneous insert/updates operations.

13. Kubernetes

For further information on running LeanXcale docker containers over Kubernetes, see the Kubernetes Configuration.