From 18cdb69263616b4382bfac88962312d0476c88fe Mon Sep 17 00:00:00 2001
From: "W. Trevor King" <wking@tremily.us>
Date: Fri, 7 Dec 2018 22:14:54 -0800
Subject: [PATCH] data/aws/variables-aws: Bump master volume to 500 GiB for I/O

We'd bumped this to 120 GiB in bca6a92f (aws: Increase default master
disk size to 120GB for IO, 2018-12-27), but in a recent CI cluster [1]
the master-0 etcd-member still cycled:

  [core@ip-10-0-6-251 ~]$ sudo crictl ps -a | grep etcd-member
  4612be4ebdd15  94bc3af972c98ce73f99d70bd72144caa8b63e541ccc9d844960b7f0ca77d7c4                             27 minutes ago  Running  etcd-member  1
  81b927a307579  quay.io/coreos/etcd@sha256:688e6c102955fe927c34db97e6352d0e0962554735b2db5f2f66f3f94cfe8fd1  43 minutes ago  Exited   etcd-member  0
  [core@ip-10-0-6-251 ~]$ sudo crictl inspect 81b927a307579
  {
    "status": {
      "id": "81b927a307579bab5675851d1ffa3e37c288a252c125ffe0a9e7b4a772fae0a7",
      "metadata": {
        "attempt": 0,
        "name": "etcd-member"
      },
      "state": "CONTAINER_EXITED",
      "createdAt": "2018-12-08T05:36:05.119762774Z",
      "startedAt": "2018-12-08T05:36:05.143173509Z",
      "finishedAt": "2018-12-08T05:51:04.889449401Z",
      "exitCode": 255,
      "image": {
        "image": "quay.io/coreos/etcd:v3.2.14"
      },
      "imageRef": "quay.io/coreos/etcd@sha256:688e6c102955fe927c34db97e6352d0e0962554735b2db5f2f66f3f94cfe8fd1",
      "reason": "Error",
      "message": "",
      "labels": {
        "io.kubernetes.container.name": "etcd-member",
        "io.kubernetes.pod.name": "etcd-member-ip-10-0-6-251.ec2.internal",
        "io.kubernetes.pod.namespace": "kube-system",
        "io.kubernetes.pod.uid": "73a817bd724bb22e2bd6e4ff0ccee478"
      },
      "annotations": {
        "io.kubernetes.container.hash": "f3672bc7",
        "io.kubernetes.container.ports": "[{\"name\":\"peer\",\"hostPort\":2380,\"containerPort\":2380,\"protocol\":\"TCP\"},{\"name\":\"server\",\"hostPort\":2379,\"containerPort\":2379,\"protocol\":\"TCP\"}]",
        "io.kubernetes.container.restartCount": "0",
        "io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
        "io.kubernetes.container.terminationMessagePolicy": "File",
        "io.kubernetes.pod.terminationGracePeriod": "30"
      },
      "mounts": [],
      "logPath": "/var/log/pods/73a817bd724bb22e2bd6e4ff0ccee478/etcd-member/0.log"
    }
  }
  [core@ip-10-0-6-251 ~]$ sudo crictl logs --tail=10 81b927a307579
  2018-12-08 05:48:01.220052 W | wal: sync duration of 1.551958035s, expected less than 1s
  2018-12-08 05:48:05.013263 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 11; CANCEL")
  2018-12-08 05:48:12.491950 W | wal: sync duration of 1.756141212s, expected less than 1s
  2018-12-08 05:49:04.229575 W | etcdserver: apply entries took too long [120.740107ms for 1 entries]
  2018-12-08 05:49:04.229602 W | etcdserver: avoid queries with large range/delete range!
  2018-12-08 05:49:04.712619 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 3; CANCEL")
  2018-12-08 05:49:06.346062 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 3; CANCEL")
  2018-12-08 05:49:06.722589 W | etcdserver/api/v3rpc: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 3; CANCEL")
  2018-12-08 05:49:11.048170 W | etcdserver: apply entries took too long [151.877306ms for 1 entries]
  2018-12-08 05:49:11.048202 W | etcdserver: avoid queries with large range/delete range!

On gp2's sliding IOPS scale [2], 500 GiB gets us lots more space (1.5k
IOPS baseline), but the etcd docs recommend m4.large with max 3.6k
concurrent IOPS for a 50-node kubernetes cluster [3].  So we'll see if
it helps in CI.

[1]: https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_installer/841/pull-ci-openshift-installer-master-e2e-aws/2064/build-log.txt
[2]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html#EBSVolumeTypes_gp2
[3]: https://github.com/etcd-io/etcd/blob/v3.3.10/Documentation/op-guide/hardware.md#small-cluster
---
 data/data/aws/variables-aws.tf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/data/data/aws/variables-aws.tf b/data/data/aws/variables-aws.tf
index df672aeaff4..7fb982dc8d1 100644
--- a/data/data/aws/variables-aws.tf
+++ b/data/data/aws/variables-aws.tf
@@ -120,7 +120,7 @@ variable "aws_master_root_volume_type" {
 
 variable "aws_master_root_volume_size" {
   type        = "string"
-  default     = "120"
+  default     = "500"
   description = "The size of the volume in gigabytes for the root block device of master nodes."
 }