Push sporadically fails with huge layer sizes

[dfunckt]

A user reported that building an image with a layer sized about 2GB failed to be pushed to the registry by the builder.

https://app.frontapp.com/open/cnv_3oo50b

Front conversations

[dfunckt]

Another user reported on gitter failure to upload an image with a huge layer (around 5GB). Logentries reported:

22 Mar 2017 20:58:13.608 9913f31b235e docker-registry[78]: time="2017-03-22T18:58:13.574736698Z" level=error msg="error resolving upload: s3aws: Path not found: /docker/registry/v2/repositories/sensei2/76b52eb4d9f642056be26484c66605ec36eb3272/_uploads/648eaf0a-c3d8-4451-80e4-fec70b224a75/data" go.version=go1.6.2 http.request.host="registry2.resin.io:445" http.request.id=452c5cf9-3804-462b-991a-e508a99eb71f http.request.method=PUT http.request.remoteaddr=52.205.226.13 http.request.uri="/v2/sensei2/76b52eb4d9f642056be26484c66605ec36eb3272/blobs/uploads/648eaf0a-c3d8-4451-80e4-fec70b224a75?_state=WWEJ0K5Uq1yEuAMp62GT-sk5_gozRZzQXLL98aUKnYp7Ik5hbWUiOiJzZW5zZWkyLzc2YjUyZWI0ZDlmNjQyMDU2YmUyNjQ4NGM2NjYwNWVjMzZlYjMyNzIiLCJVVUlEIjoiNjQ4ZWFmMGEtYzNkOC00NDUxLTgwZTQtZmVjNzBiMjI0YTc1IiwiT2Zmc2V0IjoxNjE2ODk0OTUyLCJTdGFydGVkQXQiOiIyMDE3LTAzLTIyVDE4OjUyOjQzWiJ9&digest=sha256%3Ad03d70e350429563e1c5117030956ef8da341878ce9ed516e83a0e6fc5069508" http.request.useragent="docker/1.12.3 go/go1.7 git-commit/9b80787 kernel/4.7.3-coreos-r3 os/linux arch/amd64" instance.id=7d72c203-c2e9-4d6a-bbb5-e4adc954cce9 vars.name="sensei2/76b52eb4d9f642056be26484c66605ec36eb3272" vars.uuid=648eaf0a-c3d8-4451-80e4-fec70b224a75 version=v2.4.1
22 Mar 2017 20:58:13.674 9913f31b235e docker-registry[78]: time="2017-03-22T18:58:13.574845198Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: Path not found: /docker/registry/v2/repositories/sensei2/76b52eb4d9f642056be26484c66605ec36eb3272/_uploads/648eaf0a-c3d8-4451-80e4-fec70b224a75/data" err.message="unknown error" go.version=go1.6.2 http.request.host="registry2.resin.io:445" http.request.id=452c5cf9-3804-462b-991a-e508a99eb71f http.request.method=PUT http.request.remoteaddr=52.205.226.13 http.request.uri="/v2/sensei2/76b52eb4d9f642056be26484c66605ec36eb3272/blobs/uploads/648eaf0a-c3d8-4451-80e4-fec70b224a75?_state=WWEJ0K5Uq1yEuAMp62GT-sk5_gozRZzQXLL98aUKnYp7Ik5hbWUiOiJzZW5zZWkyLzc2YjUyZWI0ZDlmNjQyMDU2YmUyNjQ4NGM2NjYwNWVjMzZlYjMyNzIiLCJVVUlEIjoiNjQ4ZWFmMGEtYzNkOC00NDUxLTgwZTQtZmVjNzBiMjI0YTc1IiwiT2Zmc2V0IjoxNjE2ODk0OTUyLCJTdGFydGVkQXQiOiIyMDE3LTAzLTIyVDE4OjUyOjQzWiJ9&digest=sha256%3Ad03d70e350429563e1c5117030956ef8da341878ce9ed516e83a0e6fc5069508" http.request.useragent="docker/1.12.3 go/go1.7 git-commit/9b80787 kernel/4.7.3-coreos-r3 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=38.332036ms http.response.status=500 http.response.written=232 instance.id=7d72c203-c2e9-4d6a-bbb5-e4adc954cce9 vars.name="sensei2/76b52eb4d9f642056be26484c66605ec36eb3272" vars.uuid=648eaf0a-c3d8-4451-80e4-fec70b224a75 version=v2.4.1
22 Mar 2017 20:58:13.674 9913f31b235e docker-registry[78]: ::1 - - [22/Mar/2017:18:58:13 +0000] "PUT /v2/sensei2/76b52eb4d9f642056be26484c66605ec36eb3272/blobs/uploads/648eaf0a-c3d8-4451-80e4-fec70b224a75?_state=WWEJ0K5Uq1yEuAMp62GT-sk5_gozRZzQXLL98aUKnYp7Ik5hbWUiOiJzZW5zZWkyLzc2YjUyZWI0ZDlmNjQyMDU2YmUyNjQ4NGM2NjYwNWVjMzZlYjMyNzIiLCJVVUlEIjoiNjQ4ZWFmMGEtYzNkOC00NDUxLTgwZTQtZmVjNzBiMjI0YTc1IiwiT2Zmc2V0IjoxNjE2ODk0OTUyLCJTdGFydGVkQXQiOiIyMDE3LTAzLTIyVDE4OjUyOjQzWiJ9&digest=sha256%3Ad03d70e350429563e1c5117030956ef8da341878ce9ed516e83a0e6fc5069508 HTTP/1.0" 500 232 "" "docker/1.12.3 go/go1.7 git-commit/9b80787 kernel/4.7.3-coreos-r3 os/linux arch/amd64"

[imrehg]

It fails for me all the time. Test case is for example:

FROM resin/%%RESIN_MACHINE_NAME%%-alpine

RUN dd if=/dev/urandom of=/goliath.data bs=1M count=2000

CMD ls -lah /goliath.data; while : ; do sleep 200; done