Thread Safety

[alicebibaud]

Is your feature request related to a problem? Please describe.
I am currently importing large amounts of commits at once for analysis. Grabbing every detail of every commit at once occasionally means that imports take a LONG time - processing every property of every commit in the Linux Kernel, for example, takes about three weeks if you run pydriller consistently.

Describe the solution you'd like
I would love to be able to run pydriller in the context of a ThreadPoolExecutor block so as to parallelize (and ideally speed up) this process. Currently, when I run pydriller in a ThreadPoolExecutor block, segments of whatever commit I'm running on get chopped up and get passed in as the wrong values. For example, I've seen a python function passed in as commit.author.name, when it should be in commit.methods.

Additional context
Not a bug - I bet this was just not thought about in the original scope of the project. Thank you for all that you do!

[IP1102]

I believe commit traversal is already done inside ThreadPoolExecutor as we can see here.

Currently, when I run pydriller in a ThreadPoolExecutor block, segments of whatever commit I'm running on get chopped up and get passed in as the wrong values. For example, I've seen a python function passed in as commit.author.name, when it should be in commit.methods.

For this part, I am not sure. Python has GIL (assuming you didn't disable it) so multi-threading in Python is truly not concurrent. From what I understand, you need to make your implementation thread safe. The internal threads in PyDriller must be losing context leading to such wrong values. Mase sure to use thread locks based on your use case. You can also try multiprocessing or run PyDriller in each thread separately (not sure of your use case).

P.S: PyDriller is amazing, but for some of my larger projects which involve iterating over 100k commits I use native git implementation, especially go-git. If you are familiar with Golang or suits your use case, definitely would recommend it. You will see a massive improvement in the turn around time.

[ishepard]

Hi! Not sure what could cause this issue TBH. You can run Pydriller with multiple threads, just specify num_workers. This will execute in a ThreadPoolExecutor (https://github.com/ishepard/pydriller/blob/master/pydriller/repository.py#L239).

Just be careful that if you need commit ordering, it won't work.

Also, from my testing, git doesn't like to be parallelized. I think it uses a lot of caching to make things faster, and using multiple processes/threads is not ideal and can break things.