No jobs after starting chainlink nodes

[JoshOrndorff]

When I run the substrate-chainlink/setup script, the chainlink nodes start, but they do not have any jobs included.

Looking back at the script output, I see three instances of:

Adding Jobspec #2 to Chainlink node...
./internal-scripts/add-jobspec.sh: line 50: jq: command not found
(23) Failed writing body
Jobspec has been added to Chainlink node
Done adding jobspec #2

The script failed because I did not have the jq command installed locally. I got past this error by running sudo apt install jq.

[JoshOrndorff]

Well actually install jq made that error message go away, but the chainlink nodes still don't have any jobs registered.

After installing jq I ran

docker-compose down
./setup

[JoshOrndorff]

This is the contents of my jobids.txt

joshy@pop-os:~/ProgrammingProjects/chainlink-polkadot/substrate-chainlink$ cat jobids.txt 

null
null
null

[JoshOrndorff]

The final hint is this message that appears three times in the output of the setup script: Unknown types found, no types for Callback

[boxhock]

Hey, since jq wasn't installed when you ran the setup the first time, your environment can be in a weird state.

After getting jq installed, try clearing down all containers and volumes and try running the setup script again.

[tk-o]

Hey folks, I've been also able to observe the problem that @JoshOrndorff has flagged in this issue. Here's the stdout I get after running ./setup script:

Endpoint: ws://host.docker.internal:9944/
Starting Chainlink Docker containers
substrate-chainlink_substrate-adapter3_1 is up-to-date
substrate-chainlink_postgres-node3_1 is up-to-date
substrate-chainlink_postgres-node1_1 is up-to-date
substrate-chainlink_postgres-node2_1 is up-to-date
substrate-chainlink_substrate-adapter2_1 is up-to-date
substrate-chainlink_substrate-adapter1_1 is up-to-date
substrate-chainlink_chainlink-node3_1 is up-to-date
substrate-chainlink_chainlink-node1_1 is up-to-date
substrate-chainlink_chainlink-node2_1 is up-to-date
Waiting for http://localhost:6691/.
Service on http://localhost:6691/ is ready.
Chainlink node http://localhost:6691/ is running.
Waiting for http://localhost:6692/.
Service on http://localhost:6692/ is ready.
Chainlink node http://localhost:6692/ is running.
Waiting for http://localhost:6693/.
Service on http://localhost:6693/ is ready.
Chainlink node http://localhost:6693/ is running.
Done starting Chainlink Docker containers
Adding External Initiator #1 to Chainlink node...
Running External Initiator #1...
substrate-chainlink_postgres-node1_1 is up-to-date
substrate-chainlink_external-initiator-node1_1 is up-to-date
EI has been added to Chainlink node
Done adding EI #1
Adding External Initiator #2 to Chainlink node...
Running External Initiator #2...
substrate-chainlink_postgres-node2_1 is up-to-date
substrate-chainlink_external-initiator-node2_1 is up-to-date
EI has been added to Chainlink node
Done adding EI #2
Adding External Initiator #3 to Chainlink node...
Running External Initiator #3...
substrate-chainlink_postgres-node3_1 is up-to-date
substrate-chainlink_external-initiator-node3_1 is up-to-date
EI has been added to Chainlink node
Done adding EI #3
Adding External Adapter #1 to Chainlink node...
EA has been added to Chainlink node
Done adding EA #1
Adding External Adapter #2 to Chainlink node...
EA has been added to Chainlink node
Done adding EA #2
Adding External Adapter #3 to Chainlink node...
EA has been added to Chainlink node
Done adding EA #3
Waiting a bit for the external initiators to be ready
Adding Jobspec #1 to Chainlink node...
Jobspec has been added to Chainlink node
Done adding jobspec #1
Adding Jobspec #2 to Chainlink node...
Jobspec has been added to Chainlink node
Done adding jobspec #2
Adding Jobspec #3 to Chainlink node...
Jobspec has been added to Chainlink node
Done adding jobspec #3
Done setting up Chainlink nodes and services
Funding accounts and registering operators...
2021-03-19 21:26:32        METADATA: Unknown types found, no types for Callback
Imported operator with address 5EsiCstpHTxarfafS3tvG7WDwbrp9Bv6BbyRvpwt3fY8PCtN
Free balance is: 0
Operator funded
Registering operator 5EsiCstpHTxarfafS3tvG7WDwbrp9Bv6BbyRvpwt3fY8PCtN
Operator registered
2021-03-19 21:27:08        METADATA: Unknown types found, no types for Callback
Imported operator with address 5CDogos4Dy2tSCvShBHkeFeMscwx9Wi2vFRijjTRRFau3vkJ
Free balance is: 0
Operator funded
Registering operator 5CDogos4Dy2tSCvShBHkeFeMscwx9Wi2vFRijjTRRFau3vkJ
Operator registered
2021-03-19 21:27:44        METADATA: Unknown types found, no types for Callback
Imported operator with address 5HjLHE3A9L6zxUEn6uy8mcx3tqyVxZvVXu7okovfWzemadzs
Free balance is: 0
Operator funded
Registering operator 5HjLHE3A9L6zxUEn6uy8mcx3tqyVxZvVXu7okovfWzemadzs
Operator registered
All operators are now ready
Jobids can be found in jobids.txt or their respective Chainlink GUIs

I noticed there are no jobs added to Chainlink nodes (hence null values in the jobidx.txt file, I guess) — here's a screenshot I took on Jobs dashboard of my first Chainlink node:

---

More context

One thing I could see is that all the env files for EIs have their every env var set to NULL:

Could this prevent a job spec from being registered on a Chainlink node?

[tk-o]

I did add -v flag to the curl calls inside add-ei.sh and add-jobspec.sh. These curl requests are not completed successfully.

add-ei.sh

The curl call gets 400 in response while trying to register an external initiator:

Adding External Initiator #1 to Chainlink node...
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 6691 (#0)
> POST /v2/external_initiators HTTP/1.1
> Host: localhost:6691
> User-Agent: curl/7.64.1
> Accept: */*
> Cookie: clsession=MTYxNjE1MDkzN3xEdi1CQkFFQ180SUFBUkFCRUFBQVJ2LUNBQUVHYzNSeWFXNW5EQTRBREdOc2MyVnpjMmx2Ymw5cFpBWnpkSEpwYm1jTUlnQWdOMlprT0RkbU5EY3pZamcwTkRjMFlqa3pNREUyWkRobE5qUTJNVEl6WVdZPXyegdQypU-TGuiP1UU_ROT27I5K5LCIE4TcobKobY3_WA==; explorer=%7B%22status%22%3A%22disconnected%22%2C%22url%22%3A%22%22%7D
> Content-Type: application/json
> Content-Length: 99
> 
} [99 bytes data]
* upload completely sent off: 99 out of 99 bytes
< HTTP/1.1 400 Bad Request
< Content-Type: application/json; charset=utf-8
* Replaced cookie explorer="%7B%22status%22%3A%22disconnected%22%2C%22url%22%3A%22%22%7D" for domain localhost, path /, expire 0
< Set-Cookie: explorer=%7B%22status%22%3A%22disconnected%22%2C%22url%22%3A%22%22%7D; Path=/; SameSite=Strict
< Strict-Transport-Security: max-age=5184000; includeSubDomains
< X-Content-Type-Options: nosniff
< X-Dns-Prefetch-Control: off
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Ratelimit-Limit: 1000
< X-Ratelimit-Remaining: 992
< X-Ratelimit-Reset: 1616150947
< X-Xss-Protection: 1; mode=block
< Date: Fri, 19 Mar 2021 10:48:57 GMT
< Content-Length: 53
< 
{ [53 bytes data]
* Connection #0 to host localhost left intact
* Closing connection 0
Running External Initiator #1...

add-jobspec.sh

It looks like there's a 500 error returned for a request which attempts to register a job spec on the Chainlink node:

Adding Jobspec #1 to Chainlink node...
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 6691 (#0)
> POST /v2/specs HTTP/1.1
> Host: localhost:6691
> User-Agent: curl/7.64.1
> Accept: */*
> Cookie: clsession=MTYxNjE1MDYzOHxEdi1CQkFFQ180SUFBUkFCRUFBQVJ2LUNBQUVHYzNSeWFXNW5EQTRBREdOc2MyVnpjMmx2Ymw5cFpBWnpkSEpwYm1jTUlnQWdZMk5qT1Rrek9ETXlNVGxpTkROa09XRm1ZMkUwTURRNFlUVXhZbVEyTnpjPXxxe893TSb_f9WU39DY2JCYoeDhVC505Vid6i4uP54SpQ==; explorer=%7B%22status%22%3A%22disconnected%22%2C%22url%22%3A%22%22%7D
> Content-Type: application/json
> Content-Length: 501
> 
} [501 bytes data]
* upload completely sent off: 501 out of 501 bytes
< HTTP/1.1 500 Internal Server Error
< Content-Type: application/json; charset=utf-8
* Replaced cookie explorer="%7B%22status%22%3A%22disconnected%22%2C%22url%22%3A%22%22%7D" for domain localhost, path /, expire 0
< Set-Cookie: explorer=%7B%22status%22%3A%22disconnected%22%2C%22url%22%3A%22%22%7D; Path=/; SameSite=Strict
< Strict-Transport-Security: max-age=5184000; includeSubDomains
< X-Content-Type-Options: nosniff
< X-Dns-Prefetch-Control: off
< X-Download-Options: noopen
< X-Frame-Options: DENY
< X-Ratelimit-Limit: 1000
< X-Ratelimit-Remaining: 996
< X-Ratelimit-Reset: 1616150698
< X-Xss-Protection: 1; mode=block
< Date: Fri, 19 Mar 2021 10:43:58 GMT
< Content-Length: 222
< 
{ [222 bytes data]
* Connection #0 to host localhost left intact
* Closing connection 0
Jobspec has been added to Chainlink node
Done adding jobspec #1

[tk-o]

@JoshOrndorff: were you able to find a way around these issues? I'm trying to sort them out so I could focus on my hackathon project (Chainlink Hackathon Spring 2021), any hints would be much appreciated 🙃

@boxhock: is the additional context, that I attached above, making you think of some possible solutions here? I'm new to Chainlink node and would be grateful if we could sort this issue out 🤝

[JoshOrndorff]

No, I never got this pallet working.

[tk-o]

I've been trying to add job spec manually via node operator UI, but I get a message on not being authorised:

Here's the request sent from the UI:

curl 'http://localhost:6691/v2/specs' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:86.0) Gecko/20100101 Firefox/86.0' -H 'Accept: application/json' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: http://localhost:6691/jobs/new?format=json' -H 'Content-Type: application/json' -H 'Origin: http://localhost:6691' -H 'Connection: keep-alive' -H 'Cookie: explorer=%7B%22status%22%3A%22disconnected%22%2C%22url%22%3A%22%22%7D; clsession=MTYxNjIwMzUzOXxEdi1CQkFFQ180SUFBUkFCRUFBQVJ2LUNBQUVHYzNSeWFXNW5EQTRBREdOc2MyVnpjMmx2Ymw5cFpBWnpkSEpwYm1jTUlnQWdNemc1TURJME0yVXpZV00yTkdNeVpqaGhZVFUyTnpoaFltSTRNalE0TVRVPXxugOK73fCHPcq04vMjKZQUUmhsriQ4XlW88dK7wNMeXA==' --data-raw '{"initiators":[{"type":"external","params":{"name":"test-ei","body":{"endpoint":"substrate-node","accountIds":["0x7c522c8273973e7bcf4a5dbfcc745dba4a3ab08c1e410167d7b1bdf9cb924f6c"]}}}],"tasks":[{"type":"httpget"},{"type":"jsonparse"},{"type":"multiply"},{"type":"substrate","params":{"type":"int128"}}]}'

Here's the reponse body:

{"errors":[{"detail":" notify 'test-ei' (http://substrate-chainlink_external-initiator-node1_1:8080/jobs) received bad response '401 Unauthorized'"}]}