[Question]: Upgrade implementation vs recommendations in docs #363

Activity

1. ivareri

   added

   questionA question or discussion, not a taskA question or discussion, not a task

   on Jan 26, 2025
2. ivareri commented on Jan 27, 2025

   

   ivareri

   on Jan 27, 2025

   ContributorAuthor

   More actions

   Ok, so by replacing this test

   ansible-collection-elasticstack/roles/elasticsearch/tasks/elasticsearch-rolling-upgrade.yml

   Line 102 in 87a7dc6

   until: "response.json.status == 'green'"

   with

    until:
           - "response.json.relocating_shards == 0"
           - "response.json.initializing_shards == 0"
           - "response.json.status == 'green' or
              response.json.status == 'yellow'"
   

   it solved the deadlock with no eligble nodes. It should probably verify this status a couple of times before procceding.

   @widhalmt: Do you want PRs for this against main or #349
3. widhalmt commented on Feb 4, 2025

   

   widhalmt

   on Feb 4, 2025

   Member

   More actions

   Hi,

   As you know from our collaboration "life getting in the way" Is a thing just know all too well. Your contributions are very welcome and I'm personally extremely thankful for the work you put into it.

   To be honest, the "old implementation" came from a very old documentation where updates where done mostly manually. Until now we have not encountered a problem with the update procedure. That doesn't mean I don't believe it exists, we only work with a limited count of different setups.

   Please provide PRs against #349 as this will be the new way for updates/upgrades.

   I haven't encountered an upgrade where the system was left without eligible nodes. My personal approach would be that this is an fatal exception and needs manual interaction. On the other hand with your code you provided I don't see a problem with using it in the future. All the more since you can support the idea of proceeding with official documentation.
4. afeefghannam89 commented on Aug 21, 2026

   

   afeefghannam89

   on Aug 21, 2026

   Member

   More actions

   Related question that came up while fixing #546: should the logstash, kibana and beats roles get
   the same fresh-install guard that the elasticsearch role now has on its package tasks?

   They should not. The guard is only safe in the elasticsearch role because there is an
   alternative path - a blocked plain install falls through to elasticsearch-rolling-upgrade.yml.
   The other three roles have no such path, so the plain package: state=present task is their
   upgrade mechanism. Guarding it would mean the package is installed once and then never updated
   again, and raising elasticstack_version would silently do nothing.

   What those roles are missing instead is ordering. The upgrade docs put ingest components after
   the cluster:

   Once you've successfully upgraded your deployment, you can upgrade your ingest components,
   such as Logstash, Elastic Agent, or Beats.

   Nothing currently makes sure the Elasticsearch rolling upgrade has finished before logstash,
   kibana and beats are upgraded. If it aborts halfway, the other components are upgraded anyway
   and end up ahead of the cluster.

   For logstash there is a second, smaller point: with more than one instance they all restart at
   the same time, which pauses ingest for that window. Beats buffer and track offsets, so this is
   an availability dip rather than data loss, but serializing it the way the elasticsearch role
   does would avoid it.
5. afeefghannam89 commented on Aug 24, 2026

   

   afeefghannam89

   on Aug 24, 2026

   Member

   More actions

   Two additions after comparing the current implementation with the docs.

   The until change proposed above matches the documentation. The
   rolling upgrade guide
   says to proceed once the shards have settled, not once the cluster is green:

   You can proceed once there are no initializing or relocating shards.

   and it expects yellow during the upgrade, because of the mixed-version rule on the same page:

   Primary shards assigned to a node running the new version cannot have their replicas assigned
   to a node with the old version.

   If it is not possible to assign the replica shards to another node (there is only one upgraded
   node in the cluster), the replica shards remain unassigned and status stays yellow.

   So waiting for green between nodes cannot succeed in a small cluster, it can only time out.

   One gap that is not in the list above: node order. Per
   Upgrade Elasticsearch
   the upgrade runs in three groups - data nodes tier by tier (data_frozen, data_cold,
   data_warm, data_hot, then the remaining data nodes), then the nodes that are neither data nor
   master-eligible (ml, ingest, transform, remote_cluster_client, coordinating-only), then
   master-eligible last:

   Upgrade the master-eligible nodes last.

   Upgraded nodes can join a cluster with an earlier version master, but earlier version nodes
   cannot always join a cluster with an upgraded master.

   tasks/main.yml loops over groups[elasticstack_elasticsearch_group_name], so the position of a
   master-eligible node in the inventory decides when it is upgraded. If it goes first, every
   remaining old node has to join an upgraded master, which the docs say may fail.

   The role already builds elasticsearch_role_master / _data / _other with group_by, but only
   uses them to count masters and to render cluster.initial_master_nodes - the upgrade loop could
   use the same groups. One caveat: that block is gated on elasticsearch_node_types is defined, so
   with the default configuration (no node.roles written, so
   every node is master-eligible)
   the groups do not exist and the elected master would have to be read from _cat/master instead.

   The tier order has its own reason, also from the guide:

   Upgrade the data nodes first, tier-by-tier, starting with the frozen tier, then the cold tier,
   then the warm tier, then the hot tier.

   This ensures ILM can continue to move data through the tiers during the upgrade.

   For reference: #349 in its current state moves the tasks into separate files but keeps both
   points, "cluster.routing.allocation.enable": "none" and until: response.json.status == 'green'.
6. afeefghannam89

   added

   component:elasticsearchroles/elasticsearchroles/elasticsearch

   on Aug 25, 2026
7. afeefghannam89

   added this to the 1.0.0 milestone on Aug 26, 2026