Stop downloading ancient version of litecoin

[c0deright]

downloading old versions automatically

Currently when installing litecore-node it's preinstall script download installs an ancient version v0.13.2.1-litecore-rc2 of a special version of litecoin.

That ancient version of litecoin is indeed downloaded twice! Once when npm install runs and a second time when litecore-node install runs. This is awful on so many levels!

Instead litecore-node should NOT download any binary at all but let the user decide which version to run.

skip that useless download

When setting

SKIP_BITCOIN_DOWNLOAD=1
VERIFY_BITCOIN_DOWNLOAD=0

it skips the download but later bails out because something tries to chmod the nonexisting binary litecoind:

npm WARN deprecated json3@3.3.2: Please use the native JSON object instead of JSON 3

> litecore-node@3.1.11 preinstall /usr/local/lib/node_modules/litecore-node
> ./scripts/download

/usr/local/bin/litecore-node -> /usr/local/lib/node_modules/litecore-node/bin/litecore-node
npm ERR! path /usr/local/lib/node_modules/litecore-node/bin/litecoind
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall chmod
npm ERR! enoent ENOENT: no such file or directory, chmod '/usr/local/lib/node_modules/litecore-node/bin/litecoind'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-06-14T14_11_25_422Z-debug.log

There are so many workarounds needed to get this installed and running in a meaningfull manner...

For a sane way to use litecoin-node see below: Remove all the auto-installed litecoin versions and install the latest version on your own.

Example Dockerfile with newest litecoin binary included

#
# build with: docker build . -t mytag -f <Dockerfile>
#
FROM node:8-slim

# litecore-node can't be installed without it downloading an ancient
# version of litecoin. We do remove the ancient version below in
# `remove not needed files`

# ENV SKIP_BITCOIN_DOWNLOAD=1
ENV VERIFY_BITCOIN_DOWNLOAD=0

# install dependencies etc in one big RUN statement
RUN apt-get update \
	&& extraPackages=" \
		python \
		libzmq5 \
	" \
	&& buildDepends=" \
		build-essential \
		jq \
		libzmq3-dev \
	" \
	&& apt-get install -y --no-install-recommends \
		$extraPackages \
		$buildDepends \
	\
	&& npm install -g litecore-node --unsafe-perm=true \
	\
	&& cd /root \
	&& litecore-node create litecore-node \
	\
	&& cd /root/litecore-node \
	&& litecore-node install insight-lite-api \
	\
# apply changes to litecore-node.json via 'jq` tool
	&& jq '.servicesConfig.bitcoind.spawn.exec = "/usr/local/bin/litecoind"' < litecore-node.json \
		| jq '.servicesConfig += {"insight-lite-api":{"disableRateLimiter":true}}' > /tmp/modified.json \
	&& mv /tmp/modified.json litecore-node.json \
	\
	&& ln -snf /usr/share/zoneinfo/Europe/Berlin /etc/localtime \
	&& echo 'Europe/Berlin' > /etc/timezone \
	\
# move everything to /litecore-node
	&& mv /root/litecore-node /litecore-node \
	&& chown -R node:node /litecore-node \
	\
# remove not needed files
	&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false $buildDepends \
	&& rm -rf \
		/var/lib/apt/lists/* \
		/root/.npm \
		/root/.node-gyp \
		\
		/usr/local/lib/node_modules/litecore-node/bin/litecoin* \
		/usr/local/bin/litecoind \
		/litecore-node/node_modules/litecore-node/bin/litecoin* \
		/litecore-node/node_modules/.bin/litecoind \
		\
		/litecore-node/node_modules/*/test \
		/litecore-node/node_modules/zeromq/windows \
		/litecore-node/node_modules/litecore-node/*/test \
		/litecore-node/node_modules/litecore-node/*/benchmarks \
		\
		/usr/local/lib/node_modules/litecore-node/test \
		/usr/local/lib/node_modules/litecore-node/benchmarks \
		/usr/local/lib/node_modules/litecore-node/node_modules/*/test \
		/usr/local/lib/node_modules/litecore-node/node_modules/zeromq/windows

ENV LITECOIN_OS x86_64-linux-gnu
ENV LITECOIN_VERSION 0.17.1
ENV LITECOIN_FILENAME litecoin-${LITECOIN_VERSION}-${LITECOIN_OS}.tar.gz
ENV LITECOIN_SHA256 9cab11ba75ea4fb64474d4fea5c5b6851f9a25fe9b1d4f7fc9c12b9f190fed07

RUN curl -fSL "https://download.litecoin.org/litecoin-${LITECOIN_VERSION}/linux/${LITECOIN_FILENAME}" -o /tmp/litecoin.tar.gz \
        && echo "$LITECOIN_SHA256 /tmp/litecoin.tar.gz" | sha256sum -c - \
	&& tar -C /usr/local --strip-components=1 -xzf /tmp/litecoin.tar.gz \
	&& ldconfig \
	&& rm /tmp/litecoin.tar.gz /usr/local/bin/litecoin-qt /usr/local/bin/test_litecoin

USER node
WORKDIR /litecore-node
EXPOSE 3001

ENV NODE_ENV production

CMD litecore-node start

bitcoin.conf

server=1
whitelist=127.0.0.1
txindex=1
addressindex=1
timestampindex=1
spentindex=1
zmqpubrawtx=tcp://127.0.0.1:29332
zmqpubhashblock=tcp://127.0.0.1:29332
rpcallowip=127.0.0.1
rpcuser=bitcoin
rpcpassword=local321
disablewallet=1

[losh11]

litecore-node auto installs the binary as the Litecoin Core backend it connects to has custom endpoints that it needs to connect to. So if anyone is willing to update litecoin-litecore (the backend) to 0.17.1, we'd be able to bump that.

[c0deright]

What special endpoints are these? Why not integrate these into the main litecoin project?

Isn't the endpoint just supposed to provide ZMQ and (tx|address|timestamp|spent)index like mentioned above in bitcoin.conf?

At least that's how it's working with dash.

[losh11]

Changes can be seen in commit history. It includes changes like block data to JSON, which will never be accepted into Core.

[c0deright]

Now we have the problem I tried to warn you about: ancient version of litecore-litecoin isn't accepting block 1780451 because of tx 39f5a0b1ecd570d6b6761ce22ef9b364cc63c5215b88278ec880e7df6208813f, please see litecore-archive/litecore-litecoin#10

ERROR: ConnectBlock(): CheckInputs on 39f5a0b1ecd570d6b6761ce22ef9b364cc63c5215b88278ec880e7df6208813f failed with mandatory-script-verify-flag-failed (Script failed an OP_EQUALVERIFY operation) (code 16)
InvalidChainFound: invalid block=b51ffd12be2ed31b2accecfa0199d00252954fc529e18f7eaf0d4f243408fc52  height=1780451  log2_work=73.859306  date=2020-01-30 21:31:46
InvalidChainFound:  current best=ec5b5831ca4c92ad7c52f80a8f4cd6a74c4550b6a1b1667f2c85ee32bebf5224  height=1780450  log2_work=73.859305  date=2020-01-30 21:15:39

How are we supposed to fix that?

[c0deright]

Using litecoind from litecoin-0.17.1 on the same datadir (!) did result in a similar error at this block, even when using reconsiderblock:

2020-01-31T10:55:36Z UpdateTip: new best=5d53720aa7f701f664c8752faf63b02c6d96d7131627c72eb383d8f300535bc8 height=1780445 version=0x20000000 log2_work=73.859295 tx=40541094 date='2020-01-30T21:03:32Z' progress=0.999589 cache=1.1MiB(8262txo)
2020-01-31T10:55:36Z UpdateTip: new best=dd5a082edbb67d5d640ab27763a7f8b7cb0df6936d9591743a9985a98e05bcba height=1780446 version=0x20000000 log2_work=73.859297 tx=40541163 date='2020-01-30T21:06:25Z' progress=0.999590 cache=1.2MiB(8610txo)
2020-01-31T10:55:36Z UpdateTip: new best=72e22fa7bd790f0dbc4706bebbc647ac2b981758751d0d481cc9be8d94daf68e height=1780447 version=0x20000000 log2_work=73.859299 tx=40541189 date='2020-01-30T21:07:23Z' progress=0.999591 cache=1.2MiB(8647txo)
2020-01-31T10:55:36Z UpdateTip: new best=79928e1a0c63709633e07f576dd85d25142c65cfc351e55f571cbb63f6ff1fe9 height=1780448 version=0x20000000 log2_work=73.859301 tx=40541239 date='2020-01-30T21:09:43Z' progress=0.999592 cache=1.2MiB(9137txo)
2020-01-31T10:55:36Z UpdateTip: new best=56f0e25c16f22e953f9bd41e2ec936f2e1659a83ccd7f3d922f5554fde5dfa7f height=1780449 version=0x20000000 log2_work=73.859303 tx=40541354 date='2020-01-30T21:14:51Z' progress=0.999594 cache=1.2MiB(9284txo)
2020-01-31T10:55:36Z UpdateTip: new best=ec5b5831ca4c92ad7c52f80a8f4cd6a74c4550b6a1b1667f2c85ee32bebf5224 height=1780450 version=0x20000000 log2_work=73.859305 tx=40541375 date='2020-01-30T21:15:39Z' progress=0.999595 cache=1.2MiB(9275txo)
2020-01-31T10:55:37Z ERROR: ConnectBlock: CheckQueue failed
2020-01-31T10:55:37Z InvalidChainFound: invalid block=b51ffd12be2ed31b2accecfa0199d00252954fc529e18f7eaf0d4f243408fc52  height=1780451  log2_work=73.859306  date=2020-01-30T21:31:46Z
2020-01-31T10:55:37Z InvalidChainFound:  current best=ec5b5831ca4c92ad7c52f80a8f4cd6a74c4550b6a1b1667f2c85ee32bebf5224  height=1780450  log2_work=73.859305  date=2020-01-30T21:15:39Z
2020-01-31T10:55:37Z ERROR: ConnectTip(): ConnectBlock b51ffd12be2ed31b2accecfa0199d00252954fc529e18f7eaf0d4f243408fc52 failed
2020-01-31T10:55:37Z InvalidChainFound: invalid block=e57d23d7678de2a4bb2536874510f37f782ac7d4bd49e59e65760da2ff9dc92c  height=1780467  log2_work=73.859337  date=2020-01-30T22:19:19Z
2020-01-31T10:55:37Z InvalidChainFound:  current best=ec5b5831ca4c92ad7c52f80a8f4cd6a74c4550b6a1b1667f2c85ee32bebf5224  height=1780450  log2_work=73.859305  date=2020-01-30T21:15:39Z
2020-01-31T10:55:37Z CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain.
Chain state database corruption likely.
2020-01-31T10:55:37Z CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain.
Chain state database corruption likely.

For now i killed the local blockchain and am letting litecore-litecoin redownload everything. Let's see if the error occurs again.

[losh11]

Please let us know if that fixes it. I find it kind of hard to believe that we’ve somehow managed to accept a block causing a chainfork on older clients, since we haven’t hardforked like that...

…

On Fri, 31 Jan 2020 at 11:16, @c0deright ***@***.***> wrote: Using litecoind from litecoin-0.17.1 did result in a similar error at this block, even when using reconsiderblock: 2020-01-31T10:55:36Z UpdateTip: new best=5d53720aa7f701f664c8752faf63b02c6d96d7131627c72eb383d8f300535bc8 height=1780445 version=0x20000000 log2_work=73.859295 tx=40541094 date='2020-01-30T21:03:32Z' progress=0.999589 cache=1.1MiB(8262txo) 2020-01-31T10:55:36Z UpdateTip: new best=dd5a082edbb67d5d640ab27763a7f8b7cb0df6936d9591743a9985a98e05bcba height=1780446 version=0x20000000 log2_work=73.859297 tx=40541163 date='2020-01-30T21:06:25Z' progress=0.999590 cache=1.2MiB(8610txo) 2020-01-31T10:55:36Z UpdateTip: new best=72e22fa7bd790f0dbc4706bebbc647ac2b981758751d0d481cc9be8d94daf68e height=1780447 version=0x20000000 log2_work=73.859299 tx=40541189 date='2020-01-30T21:07:23Z' progress=0.999591 cache=1.2MiB(8647txo) 2020-01-31T10:55:36Z UpdateTip: new best=79928e1a0c63709633e07f576dd85d25142c65cfc351e55f571cbb63f6ff1fe9 height=1780448 version=0x20000000 log2_work=73.859301 tx=40541239 date='2020-01-30T21:09:43Z' progress=0.999592 cache=1.2MiB(9137txo) 2020-01-31T10:55:36Z UpdateTip: new best=56f0e25c16f22e953f9bd41e2ec936f2e1659a83ccd7f3d922f5554fde5dfa7f height=1780449 version=0x20000000 log2_work=73.859303 tx=40541354 date='2020-01-30T21:14:51Z' progress=0.999594 cache=1.2MiB(9284txo) 2020-01-31T10:55:36Z UpdateTip: new best=ec5b5831ca4c92ad7c52f80a8f4cd6a74c4550b6a1b1667f2c85ee32bebf5224 height=1780450 version=0x20000000 log2_work=73.859305 tx=40541375 date='2020-01-30T21:15:39Z' progress=0.999595 cache=1.2MiB(9275txo) 2020-01-31T10:55:37Z ERROR: ConnectBlock: CheckQueue failed 2020-01-31T10:55:37Z InvalidChainFound: invalid block=b51ffd12be2ed31b2accecfa0199d00252954fc529e18f7eaf0d4f243408fc52 height=1780451 log2_work=73.859306 date=2020-01-30T21:31:46Z 2020-01-31T10:55:37Z InvalidChainFound: current best=ec5b5831ca4c92ad7c52f80a8f4cd6a74c4550b6a1b1667f2c85ee32bebf5224 height=1780450 log2_work=73.859305 date=2020-01-30T21:15:39Z 2020-01-31T10:55:37Z ERROR: ConnectTip(): ConnectBlock b51ffd12be2ed31b2accecfa0199d00252954fc529e18f7eaf0d4f243408fc52 failed 2020-01-31T10:55:37Z InvalidChainFound: invalid block=e57d23d7678de2a4bb2536874510f37f782ac7d4bd49e59e65760da2ff9dc92c height=1780467 log2_work=73.859337 date=2020-01-30T22:19:19Z 2020-01-31T10:55:37Z InvalidChainFound: current best=ec5b5831ca4c92ad7c52f80a8f4cd6a74c4550b6a1b1667f2c85ee32bebf5224 height=1780450 log2_work=73.859305 date=2020-01-30T21:15:39Z 2020-01-31T10:55:37Z CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. 2020-01-31T10:55:37Z CheckForkWarningConditions: Warning: Found invalid chain at least ~6 blocks longer than our best chain. Chain state database corruption likely. For now i killed the local blockchain and am letting litecore-litecoin redownload everything. Let's see if the error occurs again. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#19>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAT43OUKAXN72T6P65LYMJ3RAQCBZANCNFSM4HYII6QQ> .

[c0deright]

Purging the local blockchain and redownloading everything fixed the issue. So it's obviously not something hardfork or consensus related.
I still find the error message very strange.