Reader

Reader is class whose purpose will load a data source. There are different types of "Readers" depending on the data source.

  • JdbcReader

  • GenericCSVReader

  • RandomReader

  • JsonReader

"Readers" have a common parameters and other specific properties depending on the data source:

Properties

The following table describes the common properties of a "Reader" class:

Field Type Optional Description

type

Following values are allowed :

jdbc ,

csv,

random,

json ,

No

This field represents the data source that the loader will use.

scheduler_rate

Number

Yes

Scheduler indicates how often the reader process is executed. Default value = -1 , scheduler will not be considered.

execution_time

Number

Yes

Maximum time to execute reader process, when the reading process is greater than this time, the process ends. Default value = -1 and execution time will not be considered.

state_dir

String

Yes

Directory execution state persistence.This parameter represents the directory where the information of the last execution is stored.If this parameter exists, the program checks if there is a file in this directory with the value "PENDING", which will indicate that the last load was wrong and stop the execution.

filter

List

Yes

More detailed information in the "Filter" table.

JdbcReader

This reader will execute one or several queries and obtains the data that will be passed to the writer. Reader described below corresponds with "type=jdbc".

Field Type Optional Description

url

String

No

Defined by url where the db is running. Example : url: jdbc:leanxcale://localhost:1522/db;txn_mode=NO_CONFLICTS;mode=RUN

user

String

No

Database user.

password

String

No

User’s password.

driver

String

No

Driver to connect to DB. Example : driver: com.leanxcale.client.Driver

params_dir

String

Yes

Files stored in this directory contains dinamyc value to execute de queries.This option can also be set without files, for this the query must contain the character '?'. <br/>Example : query: select * from APP.PTTB_PARTY where PTLASTUPDATEDDTTM > ?#PTLASTUPDATEDDTTM

max_retries

Number

Yes

Number of retries to connect de DB . By default, is 4.

tables

List

No

In "second level tags" , you can find more information about the required fields in JdbcReaderTables.

Example:

reader:
  type: jdbc
  url: jdbc:leanxcale://localhost:1522/db
  user: APP
  password: APP
  driver: com.leanxcale.client.Driver
  batch: 100
  addNullWhenEmpty: false
  tables:
    - name: pto_80_test
      filter:
          - position : 4
            operator : =
            value : "#name"
            type : string

GenericCSVReader

This Reader type is charged of iterate on csv file that represents a table , and then the date will be loaded to destination (specified in "writer Yaml"). The reader described below corresponds with "type=csv".

Properties :

Field Type Optional Description

tables

List

No

In "second level tags" , you can find more information about the required fields in table CSVReaderTables.

properties

List

Yes

In "second level tags" , you can find more information about the required fields properties in CSVReaderProperties.

Example:

reader:
type: csv
tables:
- name: servicios
files: src/test/resources/csv/servicios.csv
format: RFC4180
properties:
header: True
separator: ','

RandomReader

In this case, the data is not read from a source, but the data is randomly generated.To use this "reader" by setting "type : random" in the configuration file.

Field Type Optional Description

tables

List

No

Second level tags contains information about "CSVReaderProperties".

Example:

reader:
type: random
tables:
- name: LOADERRANDOMTEST
  ntuples: 10
  columns:
- name: IDTEST
  type: INTEGER
  subtype: BIGINT
  min: 1
  distribution: INCREMENTAL

- name: NAME
  type: VARCHAR
  distribution: RANDOM
  min: 6
  max: 6

- name: NUMBER
  type: INTEGER
  distribution: RANDOM
  min: 10000
  max: 99999

- name: FLAG
  type: SMALLINT
  distribution: RANDOM
  min: 0
  max: 1

- name: TIMETEST
  type: TIMESTAMP
  distribution: RANDOM
  min: 2021-01-01 00:00:00.000
  max: 2022-09-31 00:00:00.000

- name: SMALLNUMBER
  type: SMALLINT
  distribution: RANDOM
  min: 1
  max: 9999

JsonReader

This reader load data from one or more json files. This reader corresponds with type="json" as indicated at the beginning of this section.

Field Type Optional Description

name

String

No

Table name.

files

List

No

List of files with Json format are processed to load the previous table. Example : files: [src/test/resources/json/alltypes_sample.json, src/test/resources/json/alltypes_sample_2.json]

Example:

reader:
  type: json
  tables:
    - name: test
      files: [src/test/resources/json/alltypes_sample.json, src/test/resources/json/alltypes_sample_2.json]