Deploy LeanXcale using Docker

LeanXcale is an ultrascalable distributed database. It is built of several components that can scale out to cope with your workload.

Although optimal performance can be achieved using a bare metal installation, using Docker provides significant maintenance advantages and flexibility.

Since LeanXcale is a distributed solution, it can be deployed in many different ways, and you which setup is most efficient depends on the workload.

LeanXcale provides some interesting elasticity capabilities, but this is not currently automated. The LeanXcale team is working on an Elastic Manager that can automatically adapt your deployment to the workload.

Currently, deployment is based on a single Docker image in which you can start just one component or a set of components.

In the following example, we will make a few tests over a small set of data with a simple deployment.

In order to keep things simple we will use four different configurations in one docker configuration, instead of use one docker for each component.

We should start the dockers in order. <lx_image_name> is the identifier of LeanXcale container.

First we run the docker image:

Then we start the components inside the docker:

It is very important to get the IP assigned to this docker because it has to be informed to the rest of the dockers deployed

In the following, we will use {ZKMETA_IP} as the IP for this docker.

Let’s not forget to start the conflict manager process:

we start the services:

Now we have a a simple cluster up and ready.

The cluster set up in this section is quite small. With default parameters, this means around 68GB of memory and 8 machine cores.

In order to connect with the database and allow that each component are visible to each other, you will need to open the following ports, over the following containers:

For a very simple deployment a cluster is unnecessary, but typically, your implementations will require more advanced configurations. The right configuration will depend on your workload or size of your dataset.

For datasets, the important part is the size of your "active" dataset - the part of the data that is frequently accessed and needs good response times.

For example: Let’s assume you have an application that stores the trends of product costs over the last year. But really 99% of the requests are for the last month or less. So, If someone requests more than a month you can afford to answer with a longer time. Then, we could say that your active dataset is 1/12 of the dataset size.

For best results, attach local volumes specifically to the containers for the datastores and the Query Engine, including the Transaction Logger.

The datastores are also built for taking advantage of NUMA capabilities. In order to take advantage of this, you should specially configure each datastore docker to run in a single physical core separate from any other docker.

Other dockers should be configured not to run in the cores devoted for the datastore dockers.

Once you have your database up and running, it is time to create your schema. The best way is to use SQL DDL instructions over a SQL client. We have instructions to install the SQuirreL graphical client or you can use a command line based one like sqline.

Be careful to provide the following JDBC URL connection: jdbc:leanxcale://{ZKMETA_IP}:1522/testdb

LeanXcale Query Engine was built from a fork of Apache Calcite, so you can check the Apache Calcite SQL Grammar if you have any doubt on how to do any SQL operation.

Partitioning is a very important point. If you don’t distribute your data among the different datastores you won’t benefit from any distribution, you won’t be able to scale and won’t be benefiting from any parallelization and have any speed improvement.

For partitioning there are two strategies;

For populating data you can use the lxCSVLoad utility. This utility can be called from within an image. To access the CSV files easily, attach the volumes containing them.

This utility will load data directly to the KiVi datastore in order to improve the loading phase performance.

Here are the input parameters needed by the lxCSVLoader:

Here is an example of invocation:

Is a common situation have some tables bigger than others. To avoid this situation is a common practice to split down the CSV files and launch an instance of lxCSVLoader per CSV file, loading all of them to the same table.

In order to obtain the best query plan after you are done with the database data population you should update the statistics of the database.

This can be done by executing the following script in the Zookeeper container:

You just need to run it and it will retrieve from Zookeeper all the information needed and launch sequentially the update statistics process.

If you want your data to persist you should attach the dockers to external volumes so data is persisted and can be read again upon a new start of the docker.

Otherwise, you will have to start from scratch every time you stop the dockers.

If your data is being persisted, it is recommended that you stop all dockers properly.

To do so, run this command in the Metadata container:

Then you can stop all the containers.

Don’t forget to reattach the volumes in the same way when you restart.

As you will see in the documentation our database have the following components. Here the component name followed by the component short name.

By default all components start with the default configuration of memory to be used. But they can be changed in order to improve the performance.

The docker image will be the same for each component, so you will use just the one provided before.

In order to launch the database you will need to launch at least one instance of each component.

Zookeeper is used to coordinate the components available on the cluster, so the first step will be lunch this component.

You have to know the IP of the container where zookeeper and Datastore KiVi Meta server will be running because the rest of the containers will use it. Or you just can use the container name as zk_node and KVMS_node and they will be auto-configured.

For further information on running LeanXcale docker containers over Kubernetes, see the Kubernetes Configuration.