building firebeat fails with "Generator response status code was 504"

[mykmelez]

Per @mwargers in bug 989041, attempting to generate an APK for Firebeat fails with the error "Generator response status code was 504":

07-31 14:40 > git clone git@github.com:loreV/firebeat.git
Cloning into 'firebeat'...
remote: Counting objects: 1037, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 1037 (delta 0), reused 0 (delta 0), pack-reused 1035
Receiving objects: 100% (1037/1037), 45.02 MiB | 1.16 MiB/s, done.
Resolving deltas: 100% (215/215), done.
Checking connectivity... done.
07-31 14:40 > mozilla-apk-cli firebeat firebeat.apk
Building with https://controller-review.apk.firefox.com
Generator response status code was 504

I note that a ZIP of the files in the repository takes up almost 2MB. Could this be an issue with the amount of data being transmitted?

[mykmelez]

Note that zipping the contents of the firebeat repository and uploading the package to https://marketplace.firefox.com/developers/validator succeeds with "Your app passed validation with no errors and 10 warnings."

[mykmelez]

https://marketplace.firefox.com/developers/upload/2c051ec57b4f4e77a4d1e4be405f256a

[ozten]

I am unable to reproduce this locally.

mozilla-apk-cli --endpoint http://localhost:8080 firebeat firebeat.apk

This command completes and I have a properly formatted apk, no 504.

When reproducing using your steps, it's interesting that it takes 1 minute and 46 seconds before it fails.

The 504 could be coming from an nginx gateway timeout issue, perhaps timeouts could be examined and put into this bug. I would set them high as some apks take a long time to build and transmit over the network.

Another piece of the puzzel is S3 which is used in storing and transmitting the app data.

I would look at the server logs for

• apk-controller.log
• apk-generator.log
• apk-signer.log

I don't think this apk is exceptionally large, did anything in the infrastructure change? Perhaps nginx rules were reset, the network topology has a lot more latency now (say moving part of the cluster to another hosting provider), etc.

Also:
I've updated the README of apk-factory-service with couple minor updates mozilla/apk-factory-service#87

[mykmelez]

I don't think this apk is exceptionally large, did anything in the infrastructure change? Perhaps nginx rules were reset, the network topology has a lot more latency now (say moving part of the cluster to another hosting provider), etc.

@andymckay, @dannycoates, @kumar303: Do y'all have any insight into what might be going on here?