test ENOBUF workaround

[AnqiHuangQiQi]

#151
For issue detail

We have 615 packages to run,
1 Added { maxBuffer: 1024 * 1024 * 5 } - doesn't help much. Takes too long time.
2 using BATCH_SIZE=512 - still takes too long to run.

[varungandhi-src]

I've created a separate PR with a more general fix here: #193

[varungandhi-src]

I've created a fix in #193 -- That avoids hard-coding a specific value; growing it dynamically instead. Can you please try it out?

As for the point about pip show performance, I think it's an upstream issue. Here are some rough experimental numbers:

In particular, if you have more packages installed, doesn't matter if the build uses fewer packages, pip show still becomes slower.

[AnqiHuangQiQi]

I've created a fix in #193 -- That avoids hard-coding a specific value; growing it dynamically instead. Can you please try it out?

As for the point about pip show performance, I think it's an upstream issue. Here are some rough experimental numbers:

In particular, if you have more packages installed, doesn't matter if the build uses fewer packages, pip show still becomes slower.

Thank you @varungandhi-src for the quick solution!

I did a test with the latest scip-python.
We have 605 dependencies.
But got this error after waiting for 1 hour.

That was on 4CPU/18GMem Docker container on a m7a.4xlarge Ec2 machine.
CPU usage is always 100% during the process.

1 hours is too much delay. As the repo could have many commits checked-in during peak hours.
If the later commit is ready on Sourcegraph, but older commit is still not done indexing yet.
I tried 16 CPU/ 32G Mem. So far it has taken over 40 minutes and not done yet.

It makes sense to use the re-try.
But for our case, it will have to wait until the 1st ENOBUF error thrown and then re-try. Maybe wait for another failure and then re-try.
It feels not very efficient to wait for the ENOBUF.
For each round of increasing the maxBuf, it takes a long time.

Maybe it's because we are running it from a Docker Container which caused the slowness.

I think it's an upstream issue.

What do you mean by this? Do you have any suggestions on how we could fix from our side?

[varungandhi-src]

What do you mean by this? Do you have any suggestions on how we could fix from our side?

I mean that pip show -f itself is terribly slow, even when you have a good SSD. So you can't fix this by yourself.

I'll look into if we can get rid of the dependency on the pip show command, and use a different way of getting package information.

But got this error after waiting for 1 hour.

Hmm, that seems like a bug being triggered downstream during package parsing. We'll take a look.

[AnqiHuangQiQi]

Thank you @varungandhi-src

I was printing the lines from PythonPackage.ts

for (let line of output.split('\n')) {
            console.log("line:", line);

I saw the line with MIT License before the error

[varungandhi-src]

@AnqiHuangQiQi can you please try out the v0.6.6 release? That one should avoid the pip show call which seems linked to the problem you're facing. (via #196)

If you're still having issues with the 0.6.6 release, can you please reach out through the #ext-...-sourcegraph-support Slack channel with more details about whatever errors are getting logged?

[AnqiHuangQiQi]

Hi @varungandhi-src

Thanks for the fix. It definitely works much better than previous version. It's no more stuck at the pip show step.
And the time taken is more reasonable.

But we still got this following error. This is all trace we got.
This runs on 16CPU, 32G Memory. And "--max-old-space-size=25000"

(21:26:33)   4620 / 15671
(21:26:43)   5002 / 15671
(21:26:53)   5617 / 15671

#
# Fatal error in , line 0
# Check failed: (result.ptr) != nullptr.
#
#
#
#FailureMessage Object: 0x7ffe38c55280
----- Native stack trace -----

Trace/breakpoint trap (core dumped)

[varungandhi-src]

@AnqiHuangQiQi can you please share the full stack trace/error/logs? Based on what you shared, it's not clear if that's a nodejs bug or an OOM or something else.

We can discuss this on Slack in your company's private slack message with Sourcegraph if needed, as I wrote in my earlier messages. Having the full error message would be useful for debugging.

[AnqiHuangQiQi]

I have posted it the slack channel as well. @varungandhi-src