Optimizing Retrieval of Git Objects

[kahmali]

Here's a message I wrote back on June 4th:

It looks like our bottleneck is running the git commands. I've included the output of a profiling of the method for getting a git graph. It took ~46 seconds for the entire function to run, and ~43.5 seconds were spent in the GitTerminal.show_git_object_contents() method. We may want to try some other libraries, or I just need to learn how the hell to unpack or read those damn packfiles manually. Here's the output from the profiler:

*** PROFILER RESULTS ***
get_commit_graph (/home/kahmali/Development/Projects/VisualGit/src/git/LocalRepository.py:64)
function called 1 times

    66829792 function calls in 45.968 seconds

Ordered by: cumulative time, internal time, call count
List reduced from 146 to 40 due to restriction <40>

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.195 0.195 45.968 45.968 LocalRepository.py:64(get_commit_graph)
7755 0.295 0.000 43.991 0.006 LocalRepository.py:132(get_commit_object)
7755 0.086 0.000 43.403 0.006 GitTerminal.py:100(show_git_objects_contents)
7755 6.169 0.001 43.306 0.006 GitTerminal.py:113(execute_command)
32401031 8.336 0.000 13.718 0.000 subprocess.py:958(poll)
7755 11.867 0.002 11.978 0.002 {method 'read' of '_io.TextIOWrapper' objects}
7755 0.453 0.000 9.950 0.001 subprocess.py:742(init)
7755 0.776 0.000 9.094 0.001 subprocess.py:1322(_execute_child)
32408784 5.360 0.000 5.384 0.000 subprocess.py:1464(_internal_poll)
15509 0.036 0.000 4.549 0.000 subprocess.py:484(_eintr_retry_call)
15509 4.408 0.000 4.408 0.000 {built-in method read}
7755 3.567 0.000 3.567 0.000 {built-in method fork_exec}
15510 0.055 0.000 1.756 0.000 init.py:1238(debug)
15510 0.054 0.000 1.640 0.000 init.py:1374(_log)
7754 0.088 0.000 1.492 0.000 subprocess.py:908(communicate)
7754 0.279 0.000 1.395 0.000 subprocess.py:1544(_communicate)

[kahmali]

Since June 4th, I really dug into the packfiles using the Git Community Book by Scott Chacon and for even better detail, but less beauty, the git documentation. I've now managed to parse the git objects directly from the packfiles. That code is up on master now.

2 problems remain:

1. I still have to fall back on using the GitTerminal when I bump into any errors when trying to decompress the packed object contents. This happens on about 0-3% of the commits I parse, and on larger repos with a higher rate of occurrence this can add 1 or more seconds each time the method is called (see output below)
   • I can get around the error (at least in the couple cases I checked), which appears to be an incorrect zlib header on the git object, by moving the cursor ahead two bytes to what appears to be the beginning of a zlib header [byte b'\x9c'], but that gives back a partially decompressed object which contains non-UTF-8 characters, and therefore fails with a UnicodeDecodeError when decoding the decompressed object
2. Even without any interruptions from bad zlib objects, the method still appears to be about %x slower than gitk. I roughly estimated gitk to run at about 10000 commits per second, and without any interruptions, the algorithm as is can do about 2000 commits per second. We need to be better.

Here is the output from a repo (https://github.com/quilljs/quill) with about 2000 commits and no noticeable interruption by the GitTerminal. As you'll see in the cumtime for the entire get_commit_graph method, 2100 commits took 1.05 seconds, which is exactly 2000 commits per second. Not too shabby! Definitely an upgrade from abusing the GitTerminal:

And here is the output of a repo (https://github.com/divio/django-cms) with a higher rate of dependency on the GitTerminal. As you can see, the 285 out of 9100 commits (3%) took about 1 out of 5.6 seconds (17%). That's about 9/10 of a second longer than it would have taken it we did not have to rely on the GitTerminal:

And finally a repo (https://github.com/atom/atom) almost double the size of the last (16000 commits) with no noticeable interference from the GitTerminal, just to show that the error is not consistently present in all repositories: