Migrate to Kinesis & Lambda (serverless)

[hadoopjax]

In conversation with @ASRagab, @bstarling and @nataliaking over the past several weeks we've contemplated migrating the Discursive application to a 'serverless' architecture supported by Kinesis and Lambda. In terms of desired functionality (delivering Twitter data to researchers) the master branch works just fine, albeit requiring a level of infrastructure expertise our research colleagues may not possess. To that end, we would like to gather community feedback regarding whether this migration to a serverless architecture is something we should pursue. Please provide feedback if you have it, we'd love to hear from you!

[hazel-bohon]

I like it. Is there a reason to prefer AWS Kinesis over Apache Kafka?

[wwymak]

I think it's a good idea -- having it being able to run in a serverless architecture means that people who just want an easy way to get the data has less of hurdle to overcome. As for @zacherybohon comment about kinesis vs Kafka-- if you use Kafka you'd have to manage your own architecture whereas the point of Kinesis is that all of that faff of making sure everything scales etc is managed for you ;)

[nataliaking]

I have never used Lambda with AWS Elasticsearch service but taking a peek at docs, it is doable (http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-aws-integrations.html)

Also agree with Kinesis idea 👍

I just came back from vacation the other day but getting back into work and stuff now! Sorry for lack of activity recently. Will catch up.

[nataliaking]

One important thing I wanted to note now is in AWS Lambda, there are some limitations with using Python that will affect us.

• C libraries for Python modules can’t be installed ahead of time (my apologies if too much info, but wanted to share what I learned about this through blood & tears: I tried compiling on my Mac and adding to my code for future import, but it doesn't quite work. The problem has to do with the .so file which is complied on my local computer's architecture. Solution is to spin up an ec2 instance and create virtualenv with the necessary dependencies. This will cause it to compile with the correct architecture as used by AWS lambda. Then download virtualenv from the ec2 instance and use that for Lambda. This will be same whether we use just AWS Lambda deployment or AWS Lambda via wonderful Serverless framework, same limitations apply);
• There is a size limit (50MB at this time http://docs.aws.amazon.com/lambda/latest/dg/limits.html) of the zipped code, including libraries;

[bstarling]

Anyone have experience configuring this in AWS? I was able to get a proof concept working on lambda but I was having issues accessing both public internet (twitter API) and S3 within the same lambda process. Some googling turned up this is probably related to VPC configuration but didn't get much past that.

I could have lambda return the tweets in as a json blob
Or
I could have a lambda function that saved a random blob to S3 but when I tried to combine it I always received timeout error.

Very new to lambda but I found this little package pretty nice for bundling and shipping your lambda functions.

[bstarling]

Ugh, I knew it was something simple. @wwymak Helped me out. My function was running out of memory. Toy example now working. note I was experimenting to get familiar with Lambda, not trying to claim this issue :)

[ASRagab]

@nataliaking thanks for sharing, and yeah, it's never too much! So, as noted this is a limitation in part due to Python, and part due to the target architecture that the interpreter is running on locally. Is there something unique about your setup that created this architecture mismatch, or is the general case that an EC2 instance is required?

Is there any advantage to moving to another language (i.e. Java or C#), or any appetite?

Also, I guess I want to understand the use cases a bit more that we think Kinesis/Serverless might solve, and which if any would we be interested in targeting:

1. I as a backend/infra engineer can create new data pipelines for downstream consumption quickly and in a scaleable way
2. I as curation specialist can kick off new data searches/streams without trifling with git repos, s3 buckets, etc.
3. I as a data analyst can read available data streams and choose to deposit data locally or analyze the stream as such (a la PySpark)
4. Others?

[bstarling]

Regarding the architecture it has to do with platform the C extensions are compiled for. If you use a library with a C extension and compile on mac then try to run on AWS linux that's what causes the issue mentioned. Currently our code does not require any C extensions. I was able package and upload directly without issue.

RE: Language: I think most people that would be working with tools (and potentially interested in contributing) are familiar with python which is the reason I would suggest we stick with it if possible. If there is a proposal to build pieces in another language that we simply cannot or would not want to implement in python we would just want to make sure we 1) Have the people familiar enough with chosen language to build it 2) Does not impact our usability since for the most part python seems to be the least common denominator for this sort of thing.

RE: Infra, IMO:

1. Thumbs up!
2. Hopefully one day. A incremental first step I think we should shoot for "Without requiring intimate knowledge of AWS" I.E. easy process for adding streams to existing infrastructure with tutorials, deploy scripts etc.
3. Sounds reasonable

A little off topic of the actual issue but In my mind there should be a range of engaging

• Entry point - CLI that just works EX: python twitterstream --track myhashtag --output db.sqlite for exploratory side of things
• "Production" - Once the details of what to capture have been ironed out those interested in data there is the ability to keep longrunning streams alive and saved to permanent more scalable datastore

Maybe these are completely separate but I think we can work together for now.

[metame]

So while poking around in Lambda docs turns out that for now it only works with Python 2.7. Might be another reason to hold off on a python3 refactor for the project.

[zachmueller]

Just randomly jumping in here, but I've done a good amount of work with Lambda (even using libraries with C extensions, specifically pandas/numpy/scipy) so could help answer questions on that front. I've also briefly dabbled in linking kinesis and Lambda. That said, I don't have much bandwidth to build this out myself, so I'm limited to answering specific questions. Overall I think it'd be worth trying out.

[hadoopjax]

@metame - totally forgot I ran into that the other day while hacking together a prototype. Great point.

@zachmueller cool, thanks!

Later today I'm going to create a discursive-serverless branch and dump what I have out there for us all to poke around on (if nothing else, it's a starting point).

[alexcasalboni]

Hi everyone,

just adding a few thoughts to the conversation.

AWS Lambda:

• You can refer to this article to use C/Fortan libraries in AWS Lambda
• By using the newly announced AWS CodeBuild, we may improve the deployment package generation (pip install & co.), and even achieve CI via GitHub triggers
• I wouldn't count on hacks to use Python3, as it might be natively supported soon

Amazon Kinesis:

• I love it, it's a great service to handle high-volume streaming with multiple consumers and serverless processing
• Let's not forget that it's not covered by the AWS Free Tier (details here)
• If cost is an issue, we may consider SQS as well, as it's included in the Free Tier. On the downside, it does not support serverless batch processing (yet?) and multiple consumers (once a message is consumed, it's also removed from the queue)

[hadoopjax]

Thanks everyone for reaching out on this issue I think it generated a lot of great ideas. As we decompose Discursive components into other repos (Assemble, Collect-Social) we'll migrate our infrastructure discussions into those spaces.