Skip to content

Commit

Permalink
update and clean up readme
Browse files Browse the repository at this point in the history
  • Loading branch information
@elakito
elakito committed Mar 18, 2022
1 parent 827abac commit 80d2310
Showing 1 changed file with 32 additions and 70 deletions.
102 changes: 32 additions & 70 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,53 +40,7 @@ which should produce the Kafka Connector jar file `kafka-connector-hana_m-n.jar`

## QuickStart

This instruction assumes Kafka installation is locally available so that you can start Kafka Connect as a standalone instance using its `bin/connect-standalone` command.

For getting started with this connector, the following steps need to be completed.

- Assume there is a table in Hana suitable for this sample. In the following, it is assumed that there is a table named `PERSONS1` with the following SQL schema `(PersonID int primary key, LastName varchar(255), FirstName varchar(255))`.

- Create the config file for source named [`connect-hana-source-1.properties`](config/connect-hana-source-1.properties) and placed it in folder `config`.


```
name=test-topic-1-source
connector.class=com.sap.kafka.connect.source.hana.HANASourceConnector
tasks.max=1
topics=test_topic_1
connection.url=jdbc:sap://<url>/
connection.user=<username>
connection.password=<password>
test_topic_1.table.name=<schemaname>."PERSONS1"
```

The above configuration says this source connector should read records from Hana table `PERSONS1` and send them to Kafka topic `test_topic_1`.

- Create the config file for sink named [`connect-hana-sink-1.properties`](config/connect-hana-sink-1.properties) and place it in folder `config`.


```
name=test_topic_1_sink
connector.class=com.sap.kafka.connect.sink.hana.HANASinkConnector
tasks.max=1
topics=test_topic_1
connection.url=jdbc:sap://<url>/
connection.user=<username>
connection.password=<password>
auto.create=true
test_topic_1.table.name=<schemaname>."PERSONS1_RES"
```

The above configuration says this sink connector should read messages from Kafka topic `test_topic_1` and insert to Hana table `PERSONS1_RES`.

- Start the above kafka-connect source and sink connectors using the standalone connector properties [`connect-standalone.properties`](config/connect-standalone.properties) with the following command.


```
./bin/connect-standalone config/connect-standalone.properties config/connect-hana-test-source-1.properties config/connect-hana-test-sink-1.properties
```

The above scenario is the simplest scenario of transferring records between Hana and Kafka. For the detail of this scenario and other more complex scenarios, refer to [Examples](#examples).
There are some examples that can be executed by following the instructions. For the detail of these examples, refer to [Examples](#examples).


#### Running Kafka Connect
Expand All @@ -102,54 +56,62 @@ The full list of configuration options for `kafka connector for SAP Systems` is

* Sink

* `topics` - This setting can be used to specify `a comma-separated list of topics`. Must not have spaces.
* `topics` - This setting can be used to specify `a comma-separated list of topics`. Must not have spaces.

* `auto.create` - This setting allows the creation of a new table in SAP DBs if the table specified in `{topic}.table.name` does not exist. Should be a `Boolean`. Default is `false`.
* `auto.create` - This setting allows the creation of a new table in SAP DBs if the table specified in `{topic}.table.name` does not exist. Should be a `Boolean`. Default is `false`.

* `auto.evolve` - This setting allows the evolution of the table schema with some restriction, namely when the record contains additional nullable fields that are not present previously, the corresponding columns will be added. In contrast, when the record contains less fields, the table schema will not be changed. Should be a `Boolean`. Default is `false`.
* `auto.evolve` - This setting allows the evolution of the table schema with some restriction, namely when the record contains additional nullable fields that are not present previously, the corresponding columns will be added. In contrast, when the record contains less fields, the table schema will not be changed. Should be a `Boolean`. Default is `false`.

* `batch.size` - This setting can be used to specify the number of records that can be pushed into SAP DB table in a single flush. Should be an `Integer`. Default is `3000`.
* `batch.size` - This setting can be used to specify the number of records that can be pushed into SAP DB table in a single flush. Should be an `Integer`. Default is `3000`.

* `max.retries` - This setting can be used to specify the maximum no. of retries that can be made to re-establish the connection to SAP DB in case the connection is lost. Should be an `Integer`. Default is `10`.
* `max.retries` - This setting can be used to specify the maximum no. of retries that can be made to re-establish the connection to SAP DB in case the connection is lost. Should be an `Integer`. Default is `10`.

* `{topic}.table.name` - This setting allows specifying the SAP DBs table name where the data needs to be written to. Should be a `String`. Must be compatible to SAP DB Table name like `"SCHEMA"."TABLE"`.
* `{topic}.table.name` - This setting allows specifying the SAP DBs table name where the data needs to be written to. Should be a `String`. Must be compatible to SAP DB Table name like `"SCHEMA"."TABLE"`.

* `{topic}.table.type` - This is a DB specific configuration setting which allows creation of Row & Column tables if `auto.create` is set to true. Default value is `column`. And supported values are `column, row`.
* `{topic}.table.type` - This is a DB specific configuration setting which allows creation of Row & Column tables if `auto.create` is set to true. Default value is `column`. And supported values are `column, row`.

* `{topic}.insert.mode` - This setting can be used to specify one of the available insertion modes `insert` and `upsert`. Default is `insert`.
* `{topic}.insert.mode` - This setting can be used to specify one of the available insertion modes `insert` and `upsert`. Default is `insert`.

* `{topic}.delete.enabled` - This setting can be used to allow the deletion of the record when its corresponding tombstone record is received by the connector. Default is `false`.
* `{topic}.delete.enabled` - This setting can be used to allow the deletion of the record when its corresponding tombstone record is received by the connector. Default is `false`.

* `{topic}.pk.mode` - This setting can be used to specify the primary key mode required when `auto.create` is set to `true` & the table name specified in `{topic}.table.name` does not exist in SAP DB. Default is `none`. And supported values are `record_key, record_value`.
* `{topic}.pk.mode` - This setting can be used to specify the primary key mode required when `auto.create` is set to `true` & the table name specified in `{topic}.table.name` does not exist in SAP DB. Default is `none`. And supported values are `record_key, record_value`.

* `{topic}.pk.fields` - This setting can be used to specify `a comma-separated list of primary key fields` when `{topic}.pk.mode` is set to `record_key` or `record_value`. Must not have spaces.
* `{topic}.pk.fields` - This setting can be used to specify `a comma-separated list of primary key fields` when `{topic}.pk.mode` is set to `record_key` or `record_value`. Must not have spaces.

* `{topic}.table.partition.mode` - This is a SapDB Sink specific configuration setting which determines the table partitioning in SAP DB. Default value is `none`. And supported values are `none, hash, round_robin`.
* `{topic}.table.partition.mode` - This is a SapDB Sink specific configuration setting which determines the table partitioning in SAP DB. Default value is `none`. And supported values are `none, hash, round_robin`.

* `{topic}.table.partition.count` - This is a SapDB Sink specific configuration setting which determines the number of partitions the table should have. Required when `auto.create` is set to `true` and table specified in `{topic}.table.name` does not exist in SAP DBs. Should be an `Integer`. Default value is `0`.
* `{topic}.table.partition.count` - This is a SapDB Sink specific configuration setting which determines the number of partitions the table should have. Required when `auto.create` is set to `true` and table specified in `{topic}.table.name` does not exist in SAP DBs. Should be an `Integer`. Default value is `0`.

* Source

* `topics` - This setting can be used to specify `a comma-separated list of topics`. Must not have spaces.
* `topics` - This setting can be used to specify `a comma-separated list of topics`. Must not have spaces.

* `mode` - This setting can be used to specify the mode in which data should be fetched from SAP DB table. Default is `bulk`. And supported values are `bulk, incrementing`.
* `mode` - This setting can be used to specify the mode in which data should be fetched from SAP DB table. Default is `bulk`. And supported values are `bulk, incrementing`.

* `queryMode` - This setting can be used to specify the query mode in which data should be fetched from SAP DB table. Default is `table`. And supported values are `table, query ( to support sql queries )`. When
* `queryMode` - This setting can be used to specify the query mode in which data should be fetched from SAP DB table. Default is `table`. And supported values are `table, query ( to support sql queries )`. When
using `queryMode: query` it is also required to have `query` parameter defined. This query parameter needs to be prepended by TopicName. If the `incrementing.column.name` property is used together to constrain the result, then it can be omitted from its where clause.

* `{topic}.table.name` - This setting allows specifying the SAP DB table name where the data needs to be read from. Should be a `String`. Must be compatible to SAP DB Table name like `"SCHEMA"."TABLE"`.
* `{topic}.table.name` - This setting allows specifying the SAP DB table name where the data needs to be read from. Should be a `String`. Must be compatible to SAP DB Table name like `"SCHEMA"."TABLE"`.

* `{topic}.query` - This setting allows specifying the query statement when `queryMode` is set to `query`. Should be a `String`.
* `{topic}.query` - This setting allows specifying the query statement when `queryMode` is set to `query`. Should be a `String`.

* `{topic}.poll.interval.ms` - This setting allows specifying the poll interval at which the data should be fetched from SAP DB table. Should be an `Integer`. Default value is `60000`.
* `{topic}.poll.interval.ms` - This setting allows specifying the poll interval at which the data should be fetched from SAP DB table. Should be an `Integer`. Default value is `60000`.

* `{topic}.incrementing.column.name` - In order to fetch data from a SAP DB table when `mode` is set to `incrementing`, an incremental ( or auto-incremental ) column needs to be provided. The type
* `{topic}.incrementing.column.name` - In order to fetch data from a SAP DB table when `mode` is set to `incrementing`, an incremental ( or auto-incremental ) column needs to be provided. The type
of the column can be numeric types such as `INTEGER`, `FLOAT`, `DECIMAL`, datetime types such as `DATE`, `TIME`, `TIMESTAMP`, and character types `VARCHAR`, `NVARCHAR` containing alpha-numeric characters. This considers SAP DB Timeseries tables also. Should be a valid column name ( respresented as a `String`) present in the table. See [data types in SAP HANA](https://help.sap.com/viewer/4fe29514fd584807ac9f2a04f6754767/LATEST/en-US/20a1569875191014b507cf392724b7eb.html)

* `{topic}.partition.count` - This setting can be used to specify the no. of topic partitions that the Source connector can use to publish the data. Should be an `Integer`. Default value is `1`.
* `{topic}.partition.count` - This setting can be used to specify the no. of topic partitions that the Source connector can use to publish the data. Should be an `Integer`. Default value is `1`.

* `numeric.mapping` - This setting can be used to control whether the DECIMAL column types are mapped to the default decimal type or one of the primitive types. The supported values are `none`, `best_fit`, and `best_fit_eager_double`. The default value is `none`.


* `numeric.mapping` - This setting can be used to control whether the DECIMAL column types are mapped to the default decimal type or one of the primitive types. The supported values are `none`, `best_fit`, and `best_fit_eager_double`. The default value is `none`.
* Transformations

* `EscapeFieldNameCharacters` - This SMT translates the field names by escaping certain characters of the names to make them valid in the naming scheme of the target. Each escaped character is represented in a sequence of UTF-8 bytes, each in form `<esc><xx>`, where `<esc>` is the specified escape character and `<xx>` is the hexiadecimal value of the byte.
* `type` - `com.sap.kafka.connect.transforms.EscapeFieldNameCharacters$Key` and `com.sap.kafka.connect.transforms.EscapeFieldNameCharacters$Value`
* `escape.char` - The escape character to be used.
* `valid.chars.default` - This value specifies the valid character set used in escaping those characters outside of the specified set. When this value is not set, the names are unescaped.
* `valid.chars.first` - This value optinally specifies the valid character set for the first character if this difers from the rest.

## Examples

Expand Down Expand Up @@ -177,4 +139,4 @@ Currently only SAP Hana is supported.

## License

Copyright (c) 2015-2021 SAP SE or an SAP affiliate company and kafka-connect-sap contributors. Please see our [LICENSE](LICENSE) for copyright and license information. Detailed information including third-party components and their licensing/copyright information is available [via the REUSE tool](https://api.reuse.software/info/github.com/SAP/kafka-connect-sap).
Copyright (c) 2015-2022 SAP SE or an SAP affiliate company and kafka-connect-sap contributors. Please see our [LICENSE](LICENSE) for copyright and license information. Detailed information including third-party components and their licensing/copyright information is available [via the REUSE tool](https://api.reuse.software/info/github.com/SAP/kafka-connect-sap).

0 comments on commit 80d2310

Please sign in to comment.