Unexpected response code: 500

[sivatacc]

version:
Consul v1.4.0+ent
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

3 Consul server running in Dev - DC1
3 Consul server running in Prod - DC2

./consul members -wan listing both DC1 and DC2 consul servers fine.

Consul configured for TLS encryption using Self Signed Key and Cert. Refer: https://learn.hashicorp.com/consul/advanced/day-1-operations/certificates

From Devv - DC1 I am running below command:
./consul-replicate -log-level trace -config="replicate_consul.hcl"

in PROD - DC2 one entry with secure key.

Config file;
{
"Consul": {
"Address": "127.0.0.1:8501",
"Auth": {
"Enabled": true,
"Username": "test",
"Password": "test"
},
"Retry": {
"Attempts": 12,
"Backoff": 250000000,
"MaxBackoff": 60000000000,
"Enabled": true
},
"SSL": {
"CaCert": "/consul-agent-ca.pem",
"CaPath": "/home/ec2-user/",
"Cert": "/dc1-server-consul-0.pem",
"Enabled": true,
"Key": "/dc1-server-consul-0-key.pem",
"ServerName": "secure.prod.com",
"Verify": false
},
"Token": "abcd1234",
"Transport": {
"DialKeepAlive": 30000000000,
"DialTimeout": 30000000000,
"DisableKeepAlives": false,
"IdleConnTimeout": 90000000000,
"MaxIdleConns": 100,
"MaxIdleConnsPerHost": 17,
"TLSHandshakeTimeout": 10000000000
}
},
"Excludes": [
{
"Source": "my-key"
}
],
"KillSignal": 2,
"LogLevel": "trace",
"MaxStale": 2000000000,
"PidFile": "/home/ec2-user/replica-pid",
"Prefixes": [
{
"Datacenter": "dc2",
"Dependency": {},
"Destination": "secure_new",
"Source": "secure"
}
],
"ReloadSignal": 1,
"StatusDir": "service/consul-replicate/statuses",
"Syslog": {
"Enabled": true,
"Facility": "LOCAL5"
},
"Wait": {
"Enabled": true,
"Min": 5000000000,
"Max": 10000000000
}
}

Log:
2019/04/17 13:49:25.363364 [WARN] (clients) disabling consul SSL verification
2019/04/17 13:49:25.363419 [INFO] (runner) creating watcher
2019/04/17 13:49:25.363548 [INFO] (runner) starting
2019/04/17 13:49:25.363632 [INFO] creating pid file at "/home/ec2-user/replica-pid"
2019/04/17 13:49:25.363791 [DEBUG] (watcher) adding kv.list(secure@dc2)
2019/04/17 13:49:25.363823 [TRACE] (watcher) kv.list(secure@dc2) starting
2019/04/17 13:49:25.363849 [DEBUG] (watcher) adding kv.list(secure@dc2)
2019/04/17 13:49:25.363860 [TRACE] (watcher) kv.list(secure@dc2) already exists, skipping
2019/04/17 13:49:25.363898 [TRACE] (view) kv.list(secure@dc2) starting fetch
2019/04/17 13:49:25.363957 [TRACE] kv.list(secure@dc2): GET /v1/kv/secure?dc=dc2&stale=true&wait=1m0s
2019/04/17 13:49:25.366430 [WARN] (view) kv.list(secure@dc2): Unexpected response code: 500 (retry attempt 1 after "250ms")
2019/04/17 13:49:25.616604 [TRACE] (view) kv.list(secure@dc2) starting fetch
2019/04/17 13:49:25.616665 [TRACE] kv.list(secure@dc2): GET /v1/kv/secure?dc=dc2&stale=true&wait=1m0s
2019/04/17 13:49:25.618849 [WARN] (view) kv.list(secure@dc2): Unexpected response code: 500 (retry attempt 2 after "500ms")
2019/04/17 13:49:26.119001 [TRACE] (view) kv.list(secure@dc2) starting fetch
2019/04/17 13:49:26.119067 [TRACE] kv.list(secure@dc2): GET /v1/kv/secure?dc=dc2&stale=true&wait=1m0s
2019/04/17 13:49:26.121201 [WARN] (view) kv.list(secure@dc2): Unexpected response code: 500 (retry attempt 3 after "1s")
2019/04/17 13:49:27.121363 [TRACE] (view) kv.list(secure@dc2) starting fetch
2019/04/17 13:49:27.121416 [TRACE] kv.list(secure@dc2): GET /v1/kv/secure?dc=dc2&stale=true&wait=1m0s
2019/04/17 13:49:27.123424 [WARN] (view) kv.list(secure@dc2): Unexpected response code: 500 (retry attempt 4 after "2s")

[sivatacc]

any one can help me out on this issue ?

[valarauca]

I have recreated this issue with consul connect. The root cause appears to be iptables are rejecting my connections.