Can't deploy via GitHubActions to Dublin VM: SSH connection issues

[webwarrior-ws]

Steps to reproduce

1. Deploy to a Dublin VM using Pulumi running on GitHub actions runner.

Expected behavior

Deployment succeeds.

Actual behavior

Deployment fails with

error: after 10 failed attempts: dial tcp <VM's ip address>:22: i/o timeout

The failure is somewhere in the middle of VM setup process. The same deployment succeeded before. I also can SSH to the VM from my machine.

[webwarrior-ws]

Also deployment to VM in Quebec works fine.

[knocte]

@v0l hey, we're blocked because of this, can't deploy for 2 days already; let us know if we can provide any more useful info?

[v0l]

Did it start working again? I think actions had connectivity issues

[knocte]

We will test today again, but it didn't look like it was a flakiness issue, given that 100% of deploys were failing (not just the majority of them but all of them) during these 3 days.

[knocte]

@v0l We're not sure anymore if this is actually related to SSH connectivity of the VM, we started getting closer and closer today to narrow down the issue and we suspect it might have something to do with the LNVPS https://api.lnvps.net API /api/v1/ssh-key HTTP POST operation that seems to be occasionally timing out (Dublin more often, if not always, than Quebec). Does this ring any bell? If not, we'll keep investigating on Monday.

[v0l]

Thats pretty much a straight crud operation on the db, i wouldnt expect it to timeout

[webwarrior-ws]

When deploying using GitHub actions runner, /api/v1/ssh-key HTTP POST request sometimes times out:

warning: [CustomHttpClient] Request 'HttpReqMsg(https://api.lnvps.net/api/v1/ssh-key){Method:POST,Content:{ key_data =
   "...snip..."
  name = "setupUser" }}' failed with an exception 'System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   ...snip... ' (attempt 7/8). Retrying in 5s...

And when that happens, it keeps happening even after several retries until max retries is reached and the deployment fails.

[v0l]

Yes this is due to DDoS mitigation, I've also been having trouble building the services to deploy updates due to my docker registry being inside LNVPS network

[knocte]

So:

• who controls that DDoS approach?
• why is it affecting Dublin only?
• how can we solve this problem as a LNVPS customer other than choosing Quebec over Dublin?
• could TailScale (at our deployment level or at LNVPS level) help here?

Please we need answers to move forward. Thanks!

[v0l]

Hi,

The ddos mitigation is managed by AVS (GSL) networks for the dublin region. The reason Dublin is the only network affected because the other regions are hosted on 3rd party servers (OVH) which has its own comprehensive DDoS mitigation systems.

Im curious how its causing so much issues? I added a retry to my build workflows and its been working fine?

Im happy to add an alternative route for you guys if you need it, you can email me and we can talk about it.

[v0l]

@knocte Try this URL instead, its going via CF Tunnel so it might be more reliable: https://api-tunnel.lnvps.net/

[knocte]

We decided some days ago to move to nonDublin VMs (London & Quebec) because of the DDoS problems we were having when deploying to Dublin ones (even when using api-tunnel.lnvps.net), but suddenly today we started getting this with a Quebec VM:

command:remote:CopyToRemote (copyFileCommand-xrayCertRenewScript):
    error: after 10 failed attempts: dial tcp 15.235.3.XXX:2222: i/o timeout

Has the DDOS issues come to other regions as well???

[knocte]

Has the DDOS issues come to other regions as well???

Disregard this last msg, actually London and Quebec were working fine. We will keep monitoring Dublin to see if it improves.