Issues with parallelizing thousands of tasks

[drunksaint]

Hi! I am trying to run a script that can be parallelized trivially - basically each line of the input is independent from one another. I have a 15k line input file that takes around 10 minutes to process on a single core. A single line takes ~1.4s to run. I split the input file into 15k files containing single lines, generated the makefile and passed it to lambda through gg with -j=2000. I encountered a few issues:

• the time the full job took to run was almost 3 minutes. It should have taken a few seconds plus download of around 200 MB from s3 to lambda which shouldn't be much. Is there a way to debug which step is taking time on lambda and why? It is probably the file download, but could be something else.
• the job never runs to completion the first time. there are always ~1% jobs stuck. Killing the process and restarting it once or twice always completes the execution though. Is there a way to specify retries in case of error? Or something else that I can do to fix this?
• the creation of thunks takes around 45 seconds using gg infer make -j$(nproc) on a 6 core machine. And the wrapper uses gg create-thunk because I used pyinstaller to build the binary. The wrapper does just two things - it collects the single line input file and creates a thunk for the command. The binary and common input file are collected just once manually outside the wrapper function. I was considering using gg to convert the whole process to something that takes around 5-10sec max to run so that it can be run as an api. Is there a faster way to create these thunks?

The command I use is of the format

command single_line_input_file.txt big_constant_input_file.txt single_line_output_file.txt

The single_line_input_file.txt and single_line_output_file.txt change for each execution. The command and big_constant_input_file.txt are the same for every execution. the command is around 70MB and the big_constant_input_file.txt is around 140MB. So the same file is being downloaded to 2k lambdas parallely. I remember @sadjad, you mentioned in a talk that gg does each file transfer in parallel chunks. Perhaps this combined with the 2k parallel lambdas trying to download the same 2 files is hitting the s3 concurrency limit?

[sadjad]

Hi @drunksaint,

the time the full job took to run was almost 3 minutes. It should have taken a few seconds plus download of around 200 MB from s3 to lambda which shouldn't be much. Is there a way to debug which step is taking time on lambda and why? It is probably the file download, but could be something else.

When you run gg force it tells you how much time it spent on uploading and downloading. And I agree with you, downloading 200 MBs from S3 should be fast. I wonder if the issue is the stragglers; some workers that just take too long to finish their task and just drag out the runtime. Could you please run it again and tell me the total time vs upload/download time?

the job never runs to completion the first time. there are always ~1% jobs stuck. Killing the process and restarting it once or twice always completes the execution though. Is there a way to specify retries in case of error? Or something else that I can do to fix this?

Good question! This is quite common with Lambdas, and gg force has built in support to fight stragglers. Start gg force with --timeout [SECONDS] option; if a Lambda task takes more than SECONDS to finish, gg will spawn a new Lambda and redo the task, while leaving the first one continue running. This should effectively mitigate this issue.

the creation of thunks takes around 45 seconds using gg infer make -j$(nproc) on a 6 core machine. And the wrapper uses gg create-thunk because I used pyinstaller to build the binary. The wrapper does just two things - it collects the single line input file and creates a thunk for the command. The binary and common input file are collected just once manually outside the wrapper function. I was considering using gg to convert the whole process to something that takes around 5-10sec max to run so that it can be run as an api. Is there a faster way to create these thunks?

Although gg create-thunk is convenient, it's really slow, and is not suggested when you're dealing with a large number of thunks. Similarly, gg infer's main goal is to bridge the gap between legacy code and Lambda, and is not recommended if you're building an application from scratch. There are a couple alternatives:

• (my personal favorite way of generating thunks) Using gg's C++ API. Take a look at the https://github.com/StanfordSNR/gg/blob/master/examples folder, probably https://github.com/StanfordSNR/gg/blob/master/examples/gtest/create-thunks.cc#L226 is the best place to start. You need to write a single C++ program that generates all your thunks.

• We have a new tool, gg repl that can help you generate thunks faster. This tool receives commands via stdin and basically gives you a stateful command interface to generate thunks. [https://github.com/r2t2-project/pbrt-v3/blob/master/scripts/dump_partials_cloud.sh](Here's one example).

The idea is that this utility keeps five stacks: (1) arguments, (2) data (thunks and values), (3) executables, (4) outputs and (5) symlinks. Use you commands like push arg test, push output out, pop arg and pop output to manipulate these stacks, and when you issue the create thunk command, it will use the current values for those 5 stacks to generate the thunk. The stacks won't reset after generating a thunk, allowing you to reuse the values already in there, eliminating the need to rehash files.

But of course, if you're going to go down any of these two routes, that means you need to forgo gg infer.

So the same file is being downloaded to 2k lambdas parallely. I remember @sadjad, you mentioned in a talk that gg does each file transfer in parallel chunks. Perhaps this combined with the 2k parallel lambdas trying to download the same 2 files is hitting the s3 concurrency limit?

gg downloads files in parallel, but never breaks down a single file to chunks. I really doubt that's the issue (for example, in the talk, my demo downloads 2000 instances of gcc at the same time from S3 without breaking a sweat).

---

I think setting the --timeout option will fix the issues you're facing, but if not, I'm more than happy to work with you to get to the bottom of this.

[sadjad]

Another point I missed: make sure your AWS account actually has the capacity to run 2,000 parallel Lambdas.

The red number in the status bar shows the number of running Lambdas:

[drunksaint]

Hey @sadjad, thanks for the super detailed response!

For the thunk creation step, I'll try out one of the steps you mentioned - will probably go with the c++ api since you recommended it. Is either step faster than the other?

For the execution step, earlier when I mentioned the 3 minute time, I had timed it manually from the time the upload from my local machine completed, to the time the download to my local machine started. So I measured just the total lambda execution time, since I'm running these tests from outside ec2 and upload/download times to my machine are higher than they should be (should probably start doing this from ec2). The timeout parameter fixed the straggler issue though! But the total time still didn't really improve by much. I ran two tests, one with 1000 jobs and one with 2000. This time, with 2000 jobs and a 15 sec timeout, it still took 2.5 min to run just the lambda step. I've appended the outputs below:

First test:

$ gg force --jobs 1000 --engine lambda --timeout 10 *.output.txt
→ Loading the thunks...  done (186 ms).
↗ Uploading 4148 files (205.6 MiB)... done (153212 ms).
rmdir /tmp/thunk-execute.6bVRXk: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: TvG3e42nAXIhLImjTFvikEiPj9sIMdJs4sVwIs5i6zf0000002e6
[warning] rate limited: TKxjxyPWSYUzPeLPRcf3doyZYgXYQQWuGpsf2H9sTS4w000002e6
rmdir /tmp/thunk-execute.3dGcNc: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: TV1YlVimHOtPd7mO3IY9bJxQltwF5oL1LjwTU_Mi0eRc000002e6
rmdir /tmp/thunk-execute.2qBDth: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: THxOqTrQaRSIUdVO2wvUvlvMl5xjLnuHfELAOcj1vopE000002ea
[info] duplicating 149 jobs
[info] duplicating 399 jobs
[info] duplicating 69 jobs
[info] duplicating 43 jobs
[info] duplicating 238 jobs
[info] duplicating 79 jobs
[info] duplicating 101 jobs
[info] duplicating 80 jobs
[info] duplicating 201 jobs
[info] duplicating 37 jobs
[info] duplicating 98 jobs
[info] duplicating 109 jobs
[info] duplicating 101 jobs
[info] duplicating 53 jobs
[info] duplicating 74 jobs
[info] duplicating 95 jobs
[info] duplicating 101 jobs
[info] duplicating 45 jobs
[info] duplicating 64 jobs
[info] duplicating 138 jobs
[info] duplicating 59 jobs
[info] duplicating 26 jobs
[info] duplicating 1 job
↘ Downloading output files (17.6 MiB)... done (9187 ms).

Second test (I cancelled it during the upload when I realized that I'd like to update the parameters):

$ gg force --jobs 1000 --engine lambda --timeout 20 *.output.txt
→ Loading the thunks...  done (228 ms).
↗ Uploading 15174 files (206.2 MiB)... ^Cterminate called without an active exception
Aborted (core dumped)
$ gg force --jobs 2000 --engine lambda --timeout 15 *.output.txt
→ Loading the thunks...  done (163 ms).
↗ Uploading 181 files (205.4 MiB)... done (147898 ms).
rmdir /tmp/thunk-execute.zujCy7: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: TypeR0fToxn3kjFRIMbP1r0.UD4Kv9jXcVk6lUAustDc000002e6
[info] duplicating 1159 jobs
rmdir /tmp/thunk-execute.LMIDqS: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: TNvivcWNITQf3SRSDb__rz38.qSw789cMLhiKhJvSd0U000002ea
[info] duplicating 207 jobs
[info] duplicating 161 jobs
[info] duplicating 856 jobs
[info] duplicating 373 jobs
[info] duplicating 152 jobs
[info] duplicating 204 jobs
[info] duplicating 681 jobs
[info] duplicating 347 jobs
[info] duplicating 233 jobs
[info] duplicating 214 jobs
[info] duplicating 98 jobs
[info] duplicating 124 jobs
[info] duplicating 326 jobs
[info] duplicating 147 jobs
[info] duplicating 92 jobs
[warning] socket failure: TYYQB0rx_acg7sYE2cUn2.AlxkzJbew8koPz6iKNyfuo000002ea
[warning] socket failure: T5SZiZGGjE_2QDYUOyrxFBEj.S_EqALdvgVQoOqyvvNc000002ea
[warning] socket failure: TKu3v_OblGbusnBMwVVcItGnMtWyVISZspk0lDWqHFKY000002ea
[warning] socket failure: TQZysrHv9M4spxI5PQoD1ApkVNcTC3zrj7ExsSd.9cew000002ea
[warning] socket failure: TM3SwZvUy2EUuUwYKQbgLTzurjN5nPuIoht8IXzlshAE000002ea
[warning] socket failure: TfpRJzBXByhT3gVg2lA.FBdHc2qBo17_30PPOwhQtu6A000002ea
[warning] socket failure: T6vQej2134h7IxTuLkAr4luV_cVi72_cvY_urWY1JJ14000002ea
[warning] socket failure: TzqAhHqt7HBPIPO0Vtm246MfS4DY6KsXAsSsWKgLxE4A000002ea
[warning] socket failure: TkI5y8PbTTOXitiy2TruAthv7BhLFCQbqDvrYnl_5E8o000002ea
[warning] socket failure: TnJ5SP6aDwZRbJMQNPqKqYJfB0QzSoFGaDXn_SybC2Xw000002ea
[warning] socket failure: TVV7VCpQr_giVcLBLipCxw_eLHWyjV9H2MDYSLVW0JNE000002ea
[warning] socket failure: TvIG5iszbcgQrA4uB5scYmd98u.gl8gSaatngpbpdrUU000002ea
[warning] socket failure: TtcDAubANGN547EpeAUqoGPkF2vJYg644WhjbeFEJMoQ000002ea
[warning] socket failure: Tq4_5mxh..CMnW7.KlBx64WrLmEhOO8_ar6OK6hwFOho000002ea
[warning] socket failure: TgM4bjwvZqsJDbwxWKxj8fJpvmv7GpkjtVOK0r.O1LlM000002ea
[warning] socket failure: T_Qmv_OsMG.ocsO.HWkV87UirG3HD7onvSqy1VSwdaSg000002ea
[warning] socket failure: TwGPj9G_KHCUh0qF8P8LYdPh2KoRIOyaxPRBuMnrrjOQ000002ea
[info] duplicating 8 jobs
[info] duplicating 1 job
↘ Downloading output files (17.6 MiB)... done (11257 ms).

And yes, I could see 2000 lambdas running (the number in red).

[sadjad]

Could you please measure the time necessary for forcing just one thunk on lambda? You can run gg force with --no-download to skip downloading and just time the whole command forcing one thunk (given that the files are already uploaded).

My hunch is that the overhead of each task (launching a Lambda + downloading the inputs + uploading the output + winding down) compared to the actual job is just too high. And you know, that would be okay if you had 15,000 workers (same as the number of thunks you have); all of them start in parallel, take some time to initiate, and the run their task and finish and deliver the results. But now, you need to pay that price multiple times.

Based on what you said, each function needs around 210 MiB of input data—which would take 4-6s to download—and a small output—which will be uploaded quickly. But your link to the cloud can also add to these overheads, and I can't help but wonder if running the job from an EC2 machine would make a different. I think this is something you should definitely try.

For the thunk creation step, I'll try out one of the steps you mentioned - will probably go with the c++ api since you recommended it. Is either step faster than the other?

C++ API is faster, and both are way faster than gg create-thunk.

Hope this helps!

[drunksaint]

Forcing a single thunk without download took ~6s which seems about right.

$ time gg force --engine lambda --no-download output.txt
→ Loading the thunks...  done (0 ms).
↗ Uploading 1 file (148.0 B)... done (228 ms).

real	0m6.332s
user	0m0.022s
sys	0m0.008s

Running it a second time with a different input takes less time (~3s) because it perhaps uses the same instance with the cached binary and big input files:

$ time gg force --engine lambda --no-download output.txt
→ Loading the thunks...  done (0 ms).
↗ Uploading 1 file (97.0 B)... done (167 ms).

real	0m2.761s
user	0m0.021s
sys	0m0.013s

I reran the experiment by splitting the input into 8 line input pieces (<2k files) so that each worker can run just once. This had much better results. Using the old gg infer (with gg create-thunk) took less time (perhaps slightly less than previously expected).

$ time gg infer make -j$(nproc) >/dev/null 2>/dev/null 3>/dev/null

real	0m3.869s
user	0m15.167s
sys	0m5.574s

And forcing the thunks took around 30s which i guess is as expected because of the 15s timeout. perhaps this is due to failures the first time.

$ gg force --jobs 2000 --engine lambda --timeout 15 *.output.txt
→ Loading the thunks...  done (24 ms).
↗ Uploading 1897 files (870.4 KiB)... done (8465 ms).
rmdir /tmp/thunk-execute.opWqyJ: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: TVnRJKH9ZXaZIyXRvdHufeeiug2LxrsKz9g30plGWDBI000002e6
rmdir /tmp/thunk-execute.5OqGFo: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: Tn2odK3pXai6tAkUhEcZ9hkQJmQFlpVbt.VvjZPo64OM000002e6
rmdir /tmp/thunk-execute.So1R9r: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: TmtwB1RVfwN.ww0ZJJvzN.1pE3J0Dt90M_sDfzFqyELM000002e6
rmdir /tmp/thunk-execute.mxHqub: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: T1UYAmQ5w17LbmxgZb__CQ1Z_6A3m3CbabLNTzvGrv.A000002e6
rmdir /tmp/thunk-execute.izEKdX: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: TMyRR0rSha4Rj6i08Rru.WnvjvA447N3jrZgjXzH5YKM000002e6
rmdir /tmp/thunk-execute.GEy3eJ: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: Tucwqf0nF9hSlZs55FikeihfYfcaZNHvmSK6E9FHMrmQ000002ea
rmdir /tmp/thunk-execute.03EnhB: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: T7h_HO_OP1bs7wiGi7h5seMCHxGYbCOx2nTRzf6df5z8000002e6
rmdir /tmp/thunk-execute.gx65Yl: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: Tcbl2q5qhp9a7t9y1YyZnSY9eiuOxRFAQ_TwSSN0sCdk000002ea
rmdir /tmp/thunk-execute.pSg7YC: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: TbxWHEe.CPbwsIEah3rxwVQJrI9tVVi2mKY_vjnGc1Z8000002ea
rmdir /tmp/thunk-execute.wBtBiW: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: ThEiYfYsfBtQwDZ8e04U6wC1f.rRw72Eyv6N3YcyyeWU000002ea
rmdir /tmp/thunk-execute.BtfjXE: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: TDVeqWRjOSYbsnSuKrwVH9pD9OgCCn7Lq5xEECz_eZ4k000002e6
[info] duplicating 88 jobs
↘ Downloading output files (10.1 MiB)... done (2192 ms).

I ran a second test with a 10s timeout and this also took around 30s. Perhaps this time there were failures 2 consecutive times.

$ time gg force --jobs 2000 --engine lambda --timeout 10 *.output.txt
→ Loading the thunks...  done (32 ms).
↗ Uploading 1897 files (870.4 KiB)... done (9648 ms).
invalid response: {"returnCode": -6, "stdout": "terminate called after throwing an instance of 'std::runtime_error'\n  what():  HTTP failure in downloading 'Vr2Cz0xI_PP_57Xd63bssY.5mLFUIO3sE6QJkTYg_Svc000001c7': HTTP/1.1 404 Not Found\n"}
[warning] operational failure: T4l4_41poxM2rXpL9jWbxXe97IIhvoRXwH1_uNZJpP1E000002ea
invalid response: {"returnCode": -6, "stdout": "terminate called after throwing an instance of 'std::runtime_error'\n  what():  HTTP failure in downloading 'Vr2Cz0xI_PP_57Xd63bssY.5mLFUIO3sE6QJkTYg_Svc000001c7': HTTP/1.1 404 Not Found\n"}
[warning] operational failure: T4l4_41poxM2rXpL9jWbxXe97IIhvoRXwH1_uNZJpP1E000002ea
invalid response: {"returnCode": -6, "stdout": "terminate called after throwing an instance of 'std::runtime_error'\n  what():  HTTP failure in downloading 'Vr2Cz0xI_PP_57Xd63bssY.5mLFUIO3sE6QJkTYg_Svc000001c7': HTTP/1.1 404 Not Found\n"}
[warning] operational failure: T4l4_41poxM2rXpL9jWbxXe97IIhvoRXwH1_uNZJpP1E000002ea
invalid response: {"returnCode": -6, "stdout": "terminate called after throwing an instance of 'std::runtime_error'\n  what():  HTTP failure in downloading 'VxDi_0YAklL2LZDOCLRr1glEgxeAIqSMYmkoZPYowWxs000001c6': HTTP/1.1 404 Not Found\n"}
[warning] operational failure: TW7Bk2S1u5_mxUoy8hfKo7uXCrJfCFf9df0B8NXRrT0k000002ea
invalid response: {"returnCode": -6, "stdout": "terminate called after throwing an instance of 'std::runtime_error'\n  what():  HTTP failure in downloading 'Vr2Cz0xI_PP_57Xd63bssY.5mLFUIO3sE6QJkTYg_Svc000001c7': HTTP/1.1 404 Not Found\n"}
[warning] operational failure: T4l4_41poxM2rXpL9jWbxXe97IIhvoRXwH1_uNZJpP1E000002ea
rmdir /tmp/thunk-execute.fQd3KB: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: TfprX2jivLu4FizWi92XWkTR891GpisK_akuiavvvYCs000002e6
rmdir /tmp/thunk-execute.QHhmc6: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: TUA2qaiLT5i_m4sG7Qs2JPjOVO2H52iS.IkRgcwHZJv0000002ea
[info] duplicating 1481 jobs
rmdir /tmp/thunk-execute.K1dc04: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: TyIkFnfUnwEnnGERg8eK5PbknEEh5zxUgfD0LpXVrPCE000002ea
invalid response: {"returnCode": -6, "stdout": "terminate called after throwing an instance of 'std::runtime_error'\n  what():  HTTP failure in downloading 'Vr2Cz0xI_PP_57Xd63bssY.5mLFUIO3sE6QJkTYg_Svc000001c7': HTTP/1.1 404 Not Found\n"}
[warning] operational failure: T4l4_41poxM2rXpL9jWbxXe97IIhvoRXwH1_uNZJpP1E000002ea
[info] duplicating 5 jobs
[info] duplicating 165 jobs
↘ Downloading output files (10.1 MiB)... done (1692 ms).

real	0m44.202s
user	0m6.549s
sys	0m8.387s

Is it possible to debug these error messages? I see the ChildProcess and 404 messages quite frequently.
I'm going to setup the test from ec2 now and see if I see any improvement. Will keep you posted on how that goes.

Unrelated question: Is there a command to clear the thunk cache for just the current gg project from s3 and local?

[drunksaint]

I just ran the test on ec2. Good news! you were right! it took less time despite getting a ton of rate limited warnings.

$ time gg force --jobs 2000 --engine lambda --timeout 10 *.output.txt
→ Loading the thunks...  done (39 ms).
↗ Uploading 1899 files (206.2 MiB)... done (5845 ms).
[warning] rate limited: TQZxVGrlkGDnSkDSvz8kWKLa04UsfRI_JhPC4QlR82Yo000002ea
[warning] rate limited: TA_C9r.pN2YlTIqxS.Wwvnnb2yPCyiojxRLB39DJw2bI000002ea
[warning] rate limited: T6JXnu5O55PH3_RC9EH6rmbWpt.zwRyFLsuTCgjHnoDY000002ea
[warning] rate limited: Twn8tKkiEUfiq22w5UTLCodEGIx.awOMOi2BVUqcnlik000002ea
.
.
.
[warning] rate limited: T_Mo_HWO9m4CRPTOeIejLI73Hcecjk7nOA2_Kl0J0F9I000002ea
[warning] rate limited: T2z3jKYXXQYCEijHpCJjuI..aGJLBNzYviI3G6aBGBcE000002ea
[warning] rate limited: TMiWcy9WeimaHEA8BTmwKX1OnlzEyeqIvdyFTjlr823k000002ea
[info] duplicating 2 jobs
↘ Downloading output files (10.1 MiB)... done (1043 ms).

real	0m25.222s
user	0m10.136s
sys	0m3.295s

Looks like the lambda execution took around 18s (25s total - 6s upload - 1s download). This is despite the ~8000 rate limited warning messages in the output. It also had 6 of the ChildProcess errors. And looks like only 2 jobs failed the first time, so it was more reliable running on ec2. I'm guessing if the rate limiting issue is figured out, this could be sub 10s! Seems like this requires talking to AWS support to get the limit increased. Have you seen these many rate limited messages in your runs?

Inference took much longer using the gg infer way because I was running this on a t3.micro instance. So I'll try out the c++ api as you suggested or maybe gg repl to improve this.

$ time gg infer make -j$(nproc) >/dev/null 2>/dev/null 3>/dev/null
real	0m19.489s
user	0m25.846s
sys	0m12.209s

[drunksaint]

also, is it possible to get gg force to skip the error cases after timeout without retrying?

[drunksaint]

I looked at gg repl and seemed quite easy to use. So I thought I'll try this out and see if the performance is good enough for this use case.
I'm trying to create and execute a thunk for filetest input.txt output.txt. There are 2 issues I had while trying a few things out after looking at the file you linked.

• the description of the same thunk using gg infer has a @{GGHASH:...} argument, but the thunk created using gg repl didn't have this hash value. Do I have to hash the input file outside gg repl and create the hash string to pass in? Or does gg repl handle it?
• the output file didn't exist in the directory after using gg repl, so I am not sure what to gg force

The old way using gg infer:

$ gg infer filetest i.txt o.txt 
$ cat o.txt 
#!/usr/bin/env gg-force-and-run
TSVo_wrmjmC3H7wcmwu8ZGNDldsdc9vb6Iu2ZDxJr2QY00000108
$ gg describe TSVo_wrmjmC3H7wcmwu8ZGNDldsdc9vb6Iu2ZDxJr2QY00000108
{
 "function": {
  "hash": "VYjZ0uPlVFYiusJnE5Saokwsp6k8EuQm_uY8jr3G3hiQ00002160",
  "args": [
   "filetest",
   "@{GGHASH:VziaXgzsiNzeCIBBjDSZ9oHqywVIPYTBP5.ksiphPP_000000016}",
   "o.txt"
  ],
  "envars": []
 },
 "values": [
  "VziaXgzsiNzeCIBBjDSZ9oHqywVIPYTBP5.ksiphPP_000000016"
 ],
 "thunks": [],
 "executables": [
  "VYjZ0uPlVFYiusJnE5Saokwsp6k8EuQm_uY8jr3G3hiQ00002160"
 ],
 "outputs": [
  "o.txt"
 ],
 "links": [],
 "timeout": 0
}

What I tried with gg repl

$ gg repl
push data i.txt
push executable filetest
push arg filetest
push arg i.txt
push arg out.txt
push output out.txt
create thunk
T3818yEAgkwX8vuBV6uhGKduacn4bWNLS8YAScYy5nLI000000d2
exit
$ gg describe T3818yEAgkwX8vuBV6uhGKduacn4bWNLS8YAScYy5nLI000000d2
{
 "function": {
  "hash": "VYjZ0uPlVFYiusJnE5Saokwsp6k8EuQm_uY8jr3G3hiQ00002160",
  "args": [
   "filetest",
   "i.txt",
   "out.txt"
  ],
  "envars": []
 },
 "values": [
  "VziaXgzsiNzeCIBBjDSZ9oHqywVIPYTBP5.ksiphPP_000000016"
 ],
 "thunks": [],
 "executables": [
  "VYjZ0uPlVFYiusJnE5Saokwsp6k8EuQm_uY8jr3G3hiQ00002160"
 ],
 "outputs": [
  "out.txt"
 ],
 "links": [],
 "timeout": 0
}
$ gg force T3818yEAgkwX8vuBV6uhGKduacn4bWNLS8YAScYy5nLI000000d2
gg-force: open (T3818yEAgkwX8vuBV6uhGKduacn4bWNLS8YAScYy5nLI000000d2): No such file or directory
$ gg force out.txt
gg-force: open (out.txt): No such file or directory

sending push arg @{GGHASH:VziaXgzsiNzeCIBBjDSZ9oHqywVIPYTBP5.ksiphPP_000000016} instead of push arg i.txt created the same thunk, but this required running gg hash i.txt outside gg repl to get the hash. But this still didn't create the out.txt file with the thunk id that I could force to get the output.
I'm obviously missing something here. I'd appreciate your help with this. Thanks :)

[drunksaint]

From AWS Lambda console:

Looks like my account was allowed just 1k lambdas eventhough the number in red from the gg force output was 2k. I've reached out to them to get this increased.

[sadjad]

Hi @drunksaint -- apologies for my delayed response.

Is it possible to debug these error messages? I see the ChildProcess and 404 messages quite frequently.

The ChildProcess error is something that I'm wondering about, too, and I guess it has something to do with a race condition in kernel. We use this class to fork the process and execute the thunk, and there's a check that makes sure the parent process is not a multi-threaded program. But for some reason, that check sometimes fails, although no other threads are running... @keithw: Any ideas why this is happening?

re: 404 messages, are you sure those blobs exist in the S3 bucket? That usually means that the object is missing and can happen when there's a mismatch between the local index and the remote bucket.

Unrelated question: Is there a command to clear the thunk cache for just the current gg project from s3 and local?

Sadly, no, not yet. But it's a good thing to consider.

Seems like this requires talking to AWS support to get the limit increased. Have you seen these many rate limited messages in your runs?

We also talked to AWS support to get our limit increased. If you wanna get 2,000 concurrent Lambdas, you need them to increase your limit to around 10,000 concurrent requests (expected duration × expected requests per second).

also, is it possible to get gg force to skip the error cases after timeout without retrying?

Unfortunately, no. All the thunks has to be completed for a job to finish.

the description of the same thunk using gg infer has a @{GGHASH:...} argument, but the thunk created using gg repl didn't have this hash value. Do I have to hash the input file outside gg repl and create the hash string to pass in? Or does gg repl handle it?

gg repl doesn't directly support that. Instead, you can try pushing a link; which creates a symlink in the execution directory to the blob. For example, in your case, after push data i.txt, you can do a push link NAME TARGET, and then your program can just access NAME without any of those @{GGHASH:...} shenanigans.

the output file didn't exist in the directory after using gg repl, so I am not sure what to gg force

Normally, gg repl only creates the thunk in the blobs directory. If you want to see the output file to use with gg force, you need to pass create placeholder OUTPUT_TAG PLACEHOLDER_PATH command to gg repl.

Looks like my account was allowed just 1k lambdas eventhough the number in red from the gg force output was 2k. I've reached out to them to get this increased.

Nice, yeah, I think you need to get your limited increased.

Please let me know if there's anything else I can help with!

--Sadjad

[drunksaint]

Hey @sadjad! Thanks for your response!

re: 404 messages, are you sure those blobs exist in the S3 bucket? That usually means that the object is missing and can happen when there's a mismatch between the local index and the remote bucket.

I saw 404s initially, but the execution went through eventually. And the ouput file was correct and had everything in place. So I'm guessing this might be due to some sort of race condition.

I got gg repl to work on my local machine after the changes you suggested! And it's much faster! But I've encountered a new issue. The uploading step from ec2 to lambda is getting killed for some reason. The exact same thing works when I run it from my local machine to lambda or locally on ec2 though.

user@ec2$ gg force --engine lambda --timeout 10 outputpieces/*.out.txt
→ Loading the thunks...  done (0 ms).
↗ Uploading 2 files (205.4 MiB)... Killed
user@ec2$

me@localmachine$ gg force --engine lambda --timeout 10 outputpieces/*.out.txt
→ Loading the thunks...  done (0 ms).
↗ Uploading 4 files (205.4 MiB)... done (148358 ms).
↘ Downloading output files (2.4 KiB)... done (145 ms).
me@localmachine$

user@ec2$ gg force --timeout 10 outputpieces/*.out.txt
→ Loading the thunks...  done (1 ms).
user@ec2$

Do you know why this might be happening? Is there a way I can debug this?

[sadjad]

Hi @drunksaint,

My best guess is that the EC2 machine runs out of memory, and then the process is killed. Could you please check if that is the case?

--Sadjad

[drunksaint]

Yup, you're right! I'll move to a fatter instance. I was surprised because the machine has more than 750MB free for ~200MB files.

[drunksaint]

Got it working end to end on ec2! The whole thing (16k lines in 2k tasks - 8 lines per task) now takes around 23s for the full upload+execution+download cycle, of which upload+download is around 5s. So execution is around 18s (with a timeout of 5s and 2k lambdas).

A single task when run independently takes around 2.5s to run from ec2 though. I must've run close to 20 tests now. It always takes around 23s-24s. I always see >50 of the Childprocess and 404 errors - ratio and number vary between runs. But the runs always complete with a valid output. So the 404s are most likely a race condition. The ChildProcess I'm not sure about though. Is it possible that these two errors are leading to the higher than expected execution times?

The two errors:

rmdir /tmp/thunk-execute.mCj4Vp: Directory not empty
gg-execute-static: ChildProcess constructed in multi-threaded program

[warning] operational failure: T4OFoGB7grd8LhkRCXfwp.WiNGYcBeCMkTpUQVQ3c3Gg00000308

and

invalid response: {"returnCode": -6, "stdout": "terminate called after throwing an instance of 'std::runtime_error'\n  what():  HTTP failure in downloading 'VlKG0pQdehrKSj79PAvohBCG7YoEDXXnJevvcOVxirkk000001d6': HTTP/1.1 404 Not Found\n"}
[warning] operational failure: TCfcC03B0eKzA4V9V90irN.FswxhvZy9Tif_BiHKbmFE00000300

[sadjad]

That's still seems too much... I wonder if we're hitting some limits within S3. Would you be willing to get me some timing logs from an actual run? Here are the steps:

1. Set the GG_TIMELOG=1 environment variable, before running gg force
2. After the job is done, run aws s3 cp --recursive s3://[YOUR_BUCKET]/timelog $1/ to download all the timing logs.

If the job failed for any reason, please get rid of timelog directory in your bucket first and then try again.