[WIP] prelim set-up of resources to test kafka-connect source/sink demo

[Analect]

@solsson
In an attempt to take your addon-connect branch a step further, I branched to connect-test-debezium (this PR) in order to test the set-up in this blog, which seeks to use a debezium source connector for mySQL and a confluent JDBC sink connector.

I created a basic helm chart outside this repo to initially test, which was based on this docker-compose file and was essentially using the various debezium images, which themselves are built on top of the confluent ones, as far as I understand. This appeared to work well in terms of getting changes in mySQL flowing to postgres.

I then tried to get things working with the same resources in here (although not structured as a helm chart) using your pre-existing set-up for kafka, zookeeper, rest and schema-registry. However, the connect image is based on the Dockerfile in the PR, which itself is a replica of this one.

Initially, I was getting a problem with Exception in thread "main" org.apache.kafka.common.config.ConfigException: Invalid value tcp://10.7.245.223:80 for configuration rest.port: Not a number of type INT (see below), where somehow that tcp address above was getting set as an ENV VARIABLE KAFKA_PORT. So I added REST_PORT key/value in the connect-deployment.yaml which seems to have overcome that problem.

2017-10-02 22:42:58,913 INFO   ||  Added alias 'TimestampRouter' to plugin 'org.apache.kafka.connect.transforms.TimestampRouter'   [org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader]
2017-10-02 22:42:58,913 INFO   ||  Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey'   [org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader]
Exception in thread "main" org.apache.kafka.common.config.ConfigException: Invalid value tcp://10.7.245.223:80 for configuration rest.port: Not a number of type INT
	at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:713)
	at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:460)
	at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:453)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:62)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:75)
	at org.apache.kafka.connect.runtime.WorkerConfig.<init>(WorkerConfig.java:197)
	at org.apache.kafka.connect.runtime.distributed.DistributedConfig.<init>(DistributedConfig.java:289)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:65)

However, as per this gist, there is still something causing the connector to crash and I was wondering if you might have any insights. It could be down to something with the connector image (built on a debezium image with a confluent plugin/connector on-board) that is somehow not sitting well with your solsson images. Would you have any thoughts on what might be going on here? Thanks.

[solsson]

Haven't got time to test on my own now, so I jump to the conclusion that this is a case in point (of the argument I made here) that using .properties files is more transparent than the use of env to override property values :)

Doesn't the log show effective configurations, prior to the stuff in the gist? Both Kafka and Confluent's components tend to do so, and it is highly useful. For example from the KSQL experiment I see:

[2017-10-03 07:58:55,059] INFO KsqlRestConfig values: 
	metric.reporters = []
	ksql.command.topic.suffix = commands
	ssl.client.auth = false
...

[solsson]

Please reopen if this is still relevant. I'm going through PRs for the v5.0 i.e. Kafka 2.1 on JDK 11.