gevent support

[chicagobuss]

We'd like to use aerospike at a high rate of inserting with our asynchronous code but currently it doesn't work with gevent well.

[rbotzer]

This is still being researched and discussed. See this discussion thread: https://discuss.aerospike.com/t/gevent-compatibility-or-async-api/1001

[ramrengaswamy]

Any update here on what the plan is ?

[RonRothman]

+1 for async client. We're processing many thousands of requests concurrently--are we supposed to block them all every time we call out to Aerospike?

[rbotzer]

The likely chain of events is that the C client adds async support, which is then used for the clients built around it. Currently the Python client has been measured doing 4.5Ktps per-process, and in aggregate around 200Ktps. Among your options is to use the fairly fast synchronous client, contribute to the async effort, or wait.

[ramrengaswamy]

The 4.5K TPS roughly translates to ~200us for an operation.
So under what kind of a controlled setup was the 4.5K TPS achieved ? What was the variance ?
We are using Aerospike on EC2 machines running in a VPC where this seems highly unlikely to achieve consistently.

And more pertinent to this discussion is that using gevent in high performance asynchronous environment is a very common pattern which the python wrapper around the c client does not permit.

It would be good to have a sense of what the timing is around the planned async support in C client so that we can decide if we should wait or try something else.
Thanks !

[rbotzer]

That was measured on enterprise grade bare-metal servers. I'm not sure which instance type of EC2 you're on, but yes, I'd guess its performance will lie somewhere between that on the upper end, and the 1.5Ktps that I get on my laptop.

We're looking into adding async to the C client, but the next part will be solving how to integrate this functionality with the Python client. Unfortunately Python has a myriad of async approaches. Getting it to work with gevent is something I would like to achieve. As for the timeline, I don't want to make promises I can't keep, so I won't. It's progressing.

[andersonbd1]

Ronan,
I thought the libevent client essentially was the asynchronous C client. Are you saying that you're looking to add async to the C client apart from the libevent client?

[rbotzer]

The libevent client is asynchronous, but only does basic key-value operations. Yes, we are looking into adding async to the C client, alongside the existing synchronous functionality (similar to the Java and C# clients).

[RonRothman]

Thanks Ronen, we appreciate your responsiveness and candor.

Quick question: We noticed that the Python client does not release the GIL before calling out to Aerospike. Is this by design?

Also, just to clarify: Your benchmark of 4.5k TPS is impressive, but TBH you've missed the point.

4.5k per second averages to 220 usecs per call, but what is the latency distribution? Does every call take exactly 220 usecs? (No.) There is a tail, and at the end are some requests that lag. And when those requests lag, your synchronous Python client gums up all the thousands of requests that are in the system at that moment. Every request being serviced suffers because of a few slow Aerospike requests. This is not an acceptable risk. Hope this helps explain the urgency on our end.

[rbotzer]

I understand that the network roundtrip time that can be reclaimed, but you are still using Python and your code is going to execute significantly slower than the extension code. There are many good reasons to use Python, but. 😁

Good point on releasing the GIL. That modification can be tested well before the C client work is in place. I'm going to look into it.

[rbotzer]

@RonRothman @ramrengaswamy In 1.0.56 we're releasing the GIL around the calls to the server, other than ones that involve callbacks (aerospike.Query.foreach and aerospike.Scan.foreach). Let me know how that works for you with gevent.

[RonRothman]

Thanks, that's great. Our current (multiprocess) version has been quite stable but I'd certainly prefer to keep the entire client in-process. When 1.0.56 is released and I can find a few hours, I'll switch to a multithreaded implementation and let you know how it goes. Thanks again.

[rbotzer]

@RonRothman I'd love to hear your feedback when you get the chance.