LeanXcale Developer’s Guide

These are some changes worth noting:

The distribution source is hosted on different projects at the gitlab:

The last three ones might not be installed if not asked for. The last two ones are for stats. They are big packages, beware.

Other packages include drivers for using the system:

And, another package includes system tests, and therefore depends on everything.

The documentation is kept in this package:

There are three distributions, each one contains the lxinst program used to install that distribution, along with the rest of the distribution:

In the development and stable distributions, here is a directory per version, for example v2.0. In the public distribution there is a zip file per distribution.

In the directory for a version, there are compressed tar files (always using a .tgz extension, and never .tar.gz).

Each package may supply a single tar file for each one of these:

For example, these are some file names:

In the public distribution, there is a zip file per distribution. For example, this may be a file name:

The third number after the version is the date for the release, to identify releases published for bug fixes and the like.

When there is any problem with the system, please, follow these steps:

First, try to gather as much info as possible about your problem:

Once you gathered all the possible info, save at least the logs for further inspection, and try to reproduce the issue with the smallest possible test that still reproduces the issue.

For example, reduce the number of table columns, the number of tables, the number of insertions, and so on, to the bare minimum and see if the issue still happens.

In many cases, it is possible to reproduce the issues with just a couple of requests, even if you found them after a 3 hour test run.

With the information at hand about the smallest test that reproduced the problem:

To work on a particular package, it is important to know and follow the conventions used to build the whole sytem.

Not all these .tgz files must be built by LXBUILD. Only those needed.

Never create more than a single tar file for each architecture (including here port and src as architecture names).

Java packages CANNOT build using central repositories. That is, all depedencies must be previously downloaded at the ./lib directory.

This is important to preserve independence while working on a package, and also to keep things under control when multiple packages are being updated.

If work on multiple packages at once, you can just use the LXBUILD script from one of them, after making changes, and then extract the tar files built at the root of the other package you are working on if it relies on those files. For example:

In general, if you want to use the whole system to test your package, you can just install the distribution:

Here, all the packages have been downloaded at ./lxdist, and the distribution has been installed at blade124:~/xamplelx.

We can start it as usual, for example:

And the same goes to use any other DB command.

To update your package with a new version, with the DB stopped, you can just go to the installed directory and unpack the tar files for your package:

It is probably easier to use the addlib command to copy and unpack the new package on the installed targets:

Yet another use is to use addlib to add actual library files to the installed targets. For example, this achieves a similar effect to the previous command:

See the install guide for more.

The LXBUILD script is given the LXVERS environment variable with the system version (currently v2.0). For example, it is called like in:

It is imperative that this script either works, or exits with a non-zero status.

To write your own LXBUILD, just copy one and adapt it to your needs.

This is an example LXBUILD file, used by the TM package as of today:

Here, you might replace this with something else, to build using your preferred build tools:

For example, it could be:

Also, you must adapt the final part too:

To use the gradle installed on the gitlab runner, instead of downloading it all times, you can put this early in the gradlew (gradle wrapper) of your project:

This uses such gradle when it is found, and continues with the rest of gradlew otherwise.

To create updated packages for your package at the repository, a .gitlab-ci.yml file should be added. This file must include just this:

In this example, the DEPS variable includes the packages needed to build the one at hand.

The gitlab will download the tar files for them, and unpack them, before running your LXBUILD script and uploading the resulting tars to the distribution.

Package tests should not depend on anything outside the package.

When using gradle, use gradle test tasks for your tests.

Tests should not be built by default.

Most system tests are kept at the lx/lxtests.git gitlab project. This project includes tests written in Java and tools to run system tests (from any other package or language) and to update releases.

We describe system tests here and how to write system tests in Java in a following section.

System tests are:

A comment at the start of line in the Main.java or LXTEST file, with the format:

or

destribes the test flags. This comment is not needed if there are no flags to describe.

The known test flags are:

When platforms (unix, mirror, docker) are not given, unix is taken by default. Otherwise, only the found platforms are used. A test is meant to run at the underlying host (considered unix) or at docker, or on a mirror setup with an external host for the mirror. By default, tests run only on the unix platform.

For example, this says that the test can run at unix, docker, and that it is a long test:

For unix tests, the DB is installed at ./testdb on the project root. For docker tests, the containers have the usual lx1 host names.

For mirror tests, the mirror host must have exactly the same directory used for the local test execution.

To skip a test, we might use

It is also possible to create a SKIP file listing tests to skip, one per line, at the root of the lxtest directory.

System tests are executed usually using bin/runtests as shown later.

For system tests written in Java, use the lx/lxtests.git gitlab project. Here we describe the conventions for adding tests, see above to learn how to actually execute tests.

Create a package in that project at src/main/java/com/leanxcale/tests for your tests. For example, com.leanxcale.tests.rec1 has a recovery test.

At that package, create a Main class to run the test with the usual main method. No class may be named Main other than the test in this package. The program must take flag -k to keep the processes started and the installed DB for inspection after the test. Take a look to com.leanxcale.tests.rec1.Main as an example.

The test MUST do an exit(0) if the test passes and exit(1) if the test fails (if the program terminates with an exception upon errors, that is fine). The tools LxTest.passed() and LxTest.failed() will take care of this if you use them.

A main program might be as follows:

Here, you can see how the log levels for a few components have been adjusted before installing (and starting) the DB and running the test.

The install, kill, format, and related methods from LxTest will do nothing if the environment variable LXTESTRUNNING is set. This can be used to run a test on an already running ./testdb database.

If you want to write several related tests, for example recovery tests, name their packages starting with a common prefix (eg, rec1, rec2, etc.).

If your tests want to share common code, you are free to create a shared package and put that code there (use the prefix for your test packages as the package name) or just put the shared code on the first test and use it at will.

The kv library includes tools that work with a kvseq server to permit sequencing for events for tests that need to coordinate. Events are single-line strings that are kept forever and are sequenced by the server receiving them.

The kvseq server may run at any host and will accept connections from other processes and convey events to them:

To start it, run it with the default tcp!localhost!6666 address or supply your own listen address with flag -a:

Once started, programs that want to sequence events (or wait for them) may use these functions from the kv library (or their kv.Conn equivalents):

dials the server at addr and identifes as name (use something like qe100).

notifies that the event s did happen.

blocks the caller until the event s did happen at least nb times.

returns -1 (or false in java) if the event s did not happen at least nb times.

Events arrive.name and leave.name are automatically generated for clients as they dial the server and they disconnect from it.

By convention, if $KVSEQ is set, NetInfo will initialize the client for java processes.

For example, for a particular test, a program might call

and another call

to wait for the former to create a tree. Or perhaps,

to cause an abort if the former created its tree three times already.

The kvseq command may be used as a client by giving the -n option with the name to use for the client. For example:

will produve the events:

Debug flag -O might be used to make the client report operations (events sent and received), but the main use of the command is to generate events from the shell.

Releases are made of packages and should be published only when tests pass.

The lxtest project described before is the basis for running system tests and includes the mkrlse command to download, test, and publish releases.

As a convenience, blades 107, 107, and 110 have a testing environment setup as descibed in the previous section about running system tests.

The lxtest project is cloned at

Plus, there is a crontab entry to run the mkrlse program every night:

The crontab is a single line, folded here for readability.

The mkrlse program pulls changes, downloads the distribution, rebuilds the tests, runs them (excluding very long ones), updates the status at the gitlab, and when tests pass, publishes the downloaded distribution as the stable one.

The ~/lxtest* directories are used as a base to run mkrlse and/or bin/runtests for the releases of interest.

Usually, they include a clone of the qe_calcite project and of the kv project too, within lxtest. You can mostly forget about this, but it might be useful to know it.

The mkrlse command usage is:

It may be called with a series of steps, like in

or without arguments (which would do exactly as shown in the above command).

As a convenience, public is understood to mean

which downloads the stable distribution, builds drivers and test code, and uploads the public distribution and its drivers.

When tests are asked for, the command uses bin/runtests -a to run the tests. It is often handly to call bin/runtests by hand to run tests or to run only a subset of existing tests.

In what follows, we use ~/lxtest as the example directory, but note it might be ~/lxtest21 or some other one depending on the release.

The packages downloaded from the distribution are kept at the ~/lxtest/lxdist directory. Thus, to try out a different version of a particular package, we might

to see how it behaves.

Note that calling mkrlse will download packages and overwrite those you have copied by hand at lxdist.

When tests pass, mkrlse pushes a tests passed commit to the gitlab, including the runtests.out file listing the tests executed.

When tests fail, a tests failed commit is pushed, and the output of the failed test is included and pushed in a FAILS.out file. This also triggers the failure of a pipeline stage on that project, so the event is notified.

Check any of these files or look at the blade for further inspection on the testing status for the system.

The lxtest project includes a command bin/qelogtxns useful to debug transactions and conflicts.

To use it, enable debug diagnostics for both the query engine and kv. For example, in a standard test, you can do this using

In any case, you can adjust the log4j.properties file and the KVDEBUG property in the installed configuration file with similar values.

It is advisable to run using a single query engine so the whole set of transactions is shown in a single log file.

After running the desired load, use qelogtxns to perform basic checks and report the transactions discovered in the log file. For example:

Here, each committed transaction is reported, along with the STS and CTS values. The final status for the transaction is reported after its name (e.g., rdonly or committed in this example).

Transactions that did not write and were aborted to avoid the (empty) commit overhead are reported as rdonly, and do not have a known CTS.

The line

reports the start and the point in the log file where this did happen.

The line

reports the commit and the point in the log file where this did happen.

Lines like this (or the ones with Rscan)

report the reads made and the values retrieved, and where in the log did they happen.

Lines like

report the writes made, and the CTS used.

Conflicts checked out are shown as in

It is also possible to run the tool to show all transactions, including those with (conflicts or) errors. For example:

In this case, transactions are shown as in

Note the errors status in the transaction header line, and also the conflict reported for a conflict check.

To build kv and spread for other systems, you might adapt and/or use AWSBUILD. This script (found on the lxinst repository) knows how to create an AWS instance, download and copying kv and spread for it, and get back the package tar files.

So far, this has been used only for RHEL, beware. You need the AWS keys, plus the lxinst.pem file using for access to lxinst installed hosts.

Beware that the script creates and removes directories for kv and spread. It is better to run it on a directory created just for this.

For example:

The domain aws.leanxcale.com is delegated from the leanxcale DNS to the Amazon Route 53 zone of the same name.

All installs using lxinst for AWS using a tag, set the tag to tag.aws.leanxcale.com where tag is the tag given as the value for the awstag config property.

The gitlab project lx/awsdns.git contains a tool, mklambda that creates event watchers and a lambda function to update A records in the DNS for instances that become running.

To create the lambda at a region:

To remove the lambda at a region:

To create the lambda at all regions:

To remove the lambda from all regions:

If the lambda was already created at a region, it is not updated, and nothing is done.

For the gitlab, we use the drunner6 runner.

The gitlab source for the runner is at

Just pull and add/remove things to the docker file.

Then, to update the runner:

Refer to the

project. The README file there describes what you need to install to update the docs and how to generate them from the source.

The docs are generated whenever you push a new version to the v2.0 branch or any other release branch.

To make changes on the docs,

If there are conflicts, resolve them by editing by hand the conflicting files and then using git add on them to resolve the conflicts, and git commit to finish the merge.

Remember that, by convention, the last release number where we are working on is the current development release. This is so even if we decided to publish a public distribution for it.

To start a new release (e.g., v2.1), start with the current one (e.g., v2.0). Make sure nobody is changing it, and it passed the tests, and you either pulled the last changes for the packages involved or cloned at the last version.

First, clone the lxinst package from the git and create the new branch.

Then, edit the installer so it will fetch packages from the new version. Just change src/lxdefs and update the LXVERS and LXREPO definitions early in the file. For example,

Run LXBUILD to be sure it is ok and then push the new version.

The scripts LXBUILD at this and every other package will use the git branch as the version name, but, for safety, it may be a good idea to edit the default version number they use as well.

Now, go through the release packages and, for each one, create a branch with the new version, for example:

It is a good idea to do this in order, using the package list shown early in this document.

You should also go to blade105 or whereever you are running daily tests, and update the setup to test daily the new development release.

One way is to update the existing setup to run the new release. You might also copy it to a different directory to keep the testing setup for the old one.

For rexample, rename the directory for the new release, and fetch the new branches

Then edit the crontab to run the new release tests

The unstable distribution will be published directly by the pushes. To generate the stable distribution, go to the blade running the lxtest program mkrlse and run it. Do this in the branch for the new version, of course.

That will run the tests, and, when they pass, publish the new stable distribution for the new version.

Once the new version is ok for a new public distribution, use

as usual.

To hunt for leaks, use a running system.

For Java processes, locate the process pid, eg.

Then, extract a heap histogram:

Here, 50 lines of output suffice unless you want to check out everything. After a while, giving time to the GC to enter and reclaim unused space, extract a new one:

Then, you can use the jleak tool as found on the bin directory of the lxtest project, to inspect both ones and report the types for which total allocation did grow sorted by increase of size in bytes.

It is also useful to produce a full heap dump

And to inspect the resulting file with something like the Eclipse Memory Analyzer, which includes an automated report for suggested leaks.

Here, we got 564 extra ArrayLists with their total size increased by 13536 bytes. You’ll have to search for them in a heap dump if that is a suspected leak.

Now, for C processes, and for the C part of the QE process, you can use memleax.

It can be installed as in

And, once available, you can run it for a running pid. It will intercept calls to memory allocation/deallocation and, those that survive the given interval in seconds will be reported.

For example