lambda really slow to respond

[kc-dot-io]

I've set up LambCI to build and test via docker on ECS and my build is passing which is great!

However, it seems like there is now 2 test runners attached to my PR that are posting in slack.

This is fundamentally ok, but I'm a bit confused about what is going on.

It seems the Docker build is completing and showing a green status, if you click through the details it links to cloudwatch logs, which is less ideal because I have to setup all my users, but ok. The issue is that for some reason the other build which appears to be the lambda one stays pending yellow and the build results sent to s3 just say:

LambCI v0.9.13 triggered on stack "x-ci"

Looking up keys in x x

$ rm -rf /tmp/lambci/build
$ git clone --depth 5 https://XXXX@github.com/x/xgit -b ci/npm-token /tmp/lambci/build/x/x
Cloning into '/tmp/lambci/build/x/x'...
$ cd /tmp/lambci/build/x/x && git checkout -qf x

Build #143 started...

Build log: https://x-buildresults-5retbcjur19l.s3.amazonaws.com/gh/x/x/builds/143/ef8c1d7a5e18ff7438849c7ee54202b3.html

Running task x-ecs-BuildTask-x on ECS cluster x-ecs-Cluster-x

I assume, if I leave it, it may end up finishing on its own eventually? Any idea why the build results aren't and status aren't getting updated ?

[kc-dot-io]

Also it seems like if you try to rebuild any of the previous builds, it never finishes, rather when the build finishes, it just kicks off another. Perhaps Lambda is stuck in some sort of infinite loop?

[mhart]

See if there are any logs in the /lambci/ecs log group – that might help you debug

[kc-dot-io]

@mhart everything is running consistently now and I think the issue is here.

The code running on lambda is never completed because this line early returns. Can this be removed so that the lambda environment also completes when the ecs task finishes?

[kc-dot-io]

Here is what I found in the /lambci/ecs logs:

2017-08-18T06:12:53.918Z 9262e2f7-83db-11e7-a5ab-a99ad49cbcba Task timed out after 300.00 seconds

[kc-dot-io]

I'm currently debugging this through npm run lambda:update.

[mhart]

@slajax when the ECS build is launched, the Lambda shuts down – that's intentional. There's no reason for it to be running anymore. Everything else is dealt with from the ECS side of things.

[kc-dot-io]

@mhart Yeah I understand that part now.

I figured out how to debug the code pretty deep last night. In my case it appears that the lambda task is not shutting down cleanly, nor does it update the build status when it does.

Even after the ECS task finishes it still shows as a pending task in all of my PRs and you can see in the logs that the Lambda task eventually hits / bills it's max execution time and even then it doesn't update the build status in my PR for Lambda task.

I'm testing out a little hack that will make a filter request to the CloudWatch ECS log and source the status of the docker build to report back on the build log page in s3 - this would address #3. This would in theory also allow us to make sure the lambda task gets resolved when the ECS one is, but I get how semantically there is no real reason to leave it running after ECS runs. I would be fine with this if it was resolving the build status properly rather then leaving it in a pending state.

Can you confirm if you are ever seeing any issues with the lambda task getting stuck in the pending state even when the ECS task passes?

I'll keep debugging this and see where I end up with it. If I can address #3 without adding too much overhead to the lambda execution, then I'll propose a PR for you to review.

[kc-dot-io]

@mhart after much debugging, I tracked my issue down to this line being hardcoded.

I am using a custom stack name but it doesn't look like that STACK is passed forward into the lambci/ecs docker image. This results in my continuous-integration/my-custom-name not getting the finished status because it is sent to continuos-integration/lambci.

If I create a new CF stack and name it the default lambci, everything is fine. I'll see if I can figure out how to update the docker image so that the stack can be inferred from AWS_LAMBDA_FUNCTION_NAME as it is here.

Any suggestions on the change set you'd prefer to see would be welcome.

[kc-dot-io]

@mhart Proposed fixes are above. Thanks for the great project.

[kc-dot-io]

this was solved by #21