ETCD

• 1: Configure
• 2: Parameter
• 3: Administration
• 4: Playbook
• 5: Monitor
• 6: FAQ

    Define etcd cluster at different size

    Customize etcd components with 10 parameters

    Adding / Removing etcd members, refresh endpoints...

    Ansible playbooks that can be used in etcd module

    Dashboards, metrics, record & alerting rules.

    Frequently Asked Questions about etcd module

1 - Configure

You have to define the etcd cluster in the config inventory before deploying it.

Usually you can choose an etcd cluster with:

• One Node, no high availability, just the functionality of etcd, suitable for dev, test & demo purpose.
• Three Nodes, basic high availability, tolerate one node failure, suitable for medium production env.
• Five Nodes, better high availability, tolerate two-node failure, suitable for large production env.

Use even number of etcd nodes is meaningless, and more than five nodes is not common.

One Node

Define the group etcd in the inventory, It will create a singleton etcd instance.

# etcd cluster for ha postgres
etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

This line exists almost in all single-node config template, where the placeholder IP address 10.10.10.10 will be replaced with the current admin node IP.

The only necessary parameters are etcd_seq and etcd_cluster, which uniquely identify the cluster and each instance.

Three Nodes

Three-node etcd cluster is quite common, which tolerates one-node failure, suitable for most cases.

The trio and safe config templates use a three-node etcd cluster, as shown below:

etcd: # dcs service for postgres/patroni ha consensus
  hosts:  # 1 node for testing, 3 or 5 for production
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq required
    10.10.10.11: { etcd_seq: 2 }  # assign from 1 ~ n
    10.10.10.12: { etcd_seq: 3 }  # use odd numbers
  vars: # cluster level parameter override roles/etcd
    etcd_cluster: etcd  # mark etcd cluster name etcd
    etcd_safeguard: false # safeguard against purging

Five Nodes

Five nodes etcd cluster can tolerate two node failure, suitable for large prod env.

There’s a five-node etcd cluster example in the prod template:

etcd:
  hosts:
    10.10.10.21 : { etcd_seq: 1 }
    10.10.10.22 : { etcd_seq: 2 }
    10.10.10.23 : { etcd_seq: 3 }
    10.10.10.24 : { etcd_seq: 4 }
    10.10.10.25 : { etcd_seq: 5 }
  vars: { etcd_cluster: etcd    }

You can use even more nodes, but 3 or 5 nodes are recommended.

Etcd Usage

These are the services that currently use Etcd:

• patroni: Use etcd as a consensus backend for PostgreSQL HA
• vip-manager: Read leader info from Etcd to bind an optional L2 VIP on the PostgreSQL cluster

You’ll have to reload etcd config after any permanent change to the etcd cluster members.

e.g, update patroni reference to etcd endpoints:

./pgsql.yml -t pg_conf                                  # re-gen patroni config
./pgsql.yml -t patroni_reload -e patroni_reload=true    # reload patroni config

e.g., update vip-manager reference to etcd endpoints (if you are using PGSQL L2 VIP):

./pgsql.yml -t pg_vip # reload vip-manager config

2 - Parameter

There are 12 parameters about the ETCD module.

ETCD: 9 Parameters:

ETCD_REMOVE: 3 Parameters:

Defaults

The default parameters are defined in roles/etcd/defaults/main.yml

Additional removal parameters are defined in roles/etcd_remove/defaults/main.yml

#-----------------------------------------------------------------
# ETCD
#-----------------------------------------------------------------
#etcd_seq: 1                      # etcd instance identifier, explicitly required
etcd_cluster: etcd                # etcd cluster & group name, etcd by default
etcd_data: /data/etcd             # etcd data directory, /data/etcd by default
etcd_learner: false               # etcd instance run as learner? false by default
etcd_port: 2379                   # etcd client port, 2379 by default
etcd_peer_port: 2380              # etcd peer port, 2380 by default
etcd_init: new                    # etcd initial cluster state, new or existing
etcd_election_timeout: 1000       # etcd election timeout, 1000ms by default
etcd_heartbeat_interval: 100      # etcd heartbeat interval, 100ms by default

ETCD_REMOVE Parameters

#-----------------------------------------------------------------
# ETCD_REMOVE
#-----------------------------------------------------------------
etcd_safeguard: false             # prevent accidental removal?
etcd_rm_data: true                # remove etcd data during removal?
etcd_rm_pkg: false                # uninstall etcd packages during removal?

etcd_seq

name: etcd_seq, type: int, level: I

etcd instance identifier, REQUIRED

no default value, you have to specify it explicitly. Here is a 3-node etcd cluster example:

etcd: # dcs service for postgres/patroni ha consensus
  hosts:  # 1 node for testing, 3 or 5 for production
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq required
    10.10.10.11: { etcd_seq: 2 }  # assign from 1 ~ n
    10.10.10.12: { etcd_seq: 3 }  # use odd numbers
  vars: # cluster level parameter override roles/etcd
    etcd_cluster: etcd  # mark etcd cluster name etcd
    etcd_safeguard: false # safeguard against purging

etcd_cluster

name: etcd_cluster, type: string, level: C

etcd cluster & group name, etcd by default

default values: etcd, which is a fixed group name, can be useful when you want to use deployed some extra etcd clusters

etcd_learner

name: etcd_learner, type: bool, level: I

Init etcd instance as learner? default value is false

When set to true, the etcd instance will be init as leaner, so it will not be able to vote in the etcd cluster. You can promote it to a full member later with etcdctl member promote command.

etcd_data

name: etcd_data, type: path, level: C

etcd data directory, /data/etcd by default

etcd_port

name: etcd_port, type: port, level: C

etcd client port, 2379 by default

etcd_peer_port

name: etcd_peer_port, type: port, level: C

etcd peer port, 2380 by default

etcd_init

name: etcd_init, type: enum, level: C

etcd initial cluster state, new or existing

default values: new, which will create a standalone new etcd cluster.

The value existing is used when trying to append new member to existing etcd cluster.

etcd_election_timeout

name: etcd_election_timeout, type: int, level: C

etcd election timeout, 1000 (ms) by default

etcd_heartbeat_interval

name: etcd_heartbeat_interval, type: int, level: C

etcd heartbeat interval, 100 (ms) by default

ETCD_REMOVE

This section contains parameters for the etcd_remove role, which are action flags for the etcd-rm.yml playbook.

etcd_safeguard

name: etcd_safeguard, type: bool, level: G/C/A

prevent purging etcd instance? default value is false

If enabled, running etcd instance will not be purged by etcd-rm.yml playbook.

etcd_rm_data

name: etcd_rm_data, type: bool, level: G/C/A

remove etcd data during removal? default value is true

When enabled, the etcd-rm.yml playbook will remove etcd data directories and configuration files during cluster or member removal.

etcd_rm_pkg

name: etcd_rm_pkg, type: bool, level: G/C/A

uninstall etcd packages during removal? default value is false

When enabled, the etcd-rm.yml playbook will uninstall etcd packages during cluster or member removal.

3 - Administration

Here are some administration SOP for etcd:

• Create Cluster
• Remove Cluster
• CLI Environment
• Reload Config
• Management Scripts
• Append Member
• Remove Member

Check ETCD: FAQ for more questions.

Create Cluster

To create an etcd cluster, define the etcd cluster in inventory first:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars: { etcd_cluster: etcd   }

Then run the etcd.yml playbook.

./etcd.yml   # init etcd module on group 'etcd'

If there’s an existing etcd cluster, this playbook will update the config and restart all the etcd instances

Pigsty has a safeguard mechanism to prevent accidental purge. etcd_safeguard is false by default,

For provisioned etcd cluster in prod env, you can enable safeguard to prevent accidental clean.

Remove Cluster

To remove an existing etcd cluster, you can use the dedicated etcd-rm.yml:

./etcd-rm.yml   # remove the default etcd cluster

bin/etcd-rm     # remove the default etcd cluster

If the etcd_safeguard is set to true, the playbook will abort.

CLI Environment

Pigsty use etcd v3 API by default. (v2 support is dropped since v3.6.0)

Here’s an example of client environment config.

alias e="etcdctl"
alias em="etcdctl member"
export ETCDCTL_ENDPOINTS=https://10.10.10.10:2379
export ETCDCTL_CACERT=/etc/pki/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

You can do CRUD with the following commands after setting up the envs:

e put a 10 ; e get a; e del a ; # V3 API

Reload Config

In case of permanent etcd cluster membership changes, You’ll have to refresh the 4 etcd endpoints references:

• config file of existing etcd members and client env var
• patroni dcs endpoint config
• vip-manager dcs endpoint config

To refresh etcd config file /etc/etcd/etcd.conf on existing members and client env vars:

./etcd.yml -t etcd_conf         # refresh /etc/etcd/etcd.conf with latest status
./etcd.yml -t etcd_launch -f 1  # restart etcd instances one by one

Update patroni reference to etcd endpoints:

./pgsql.yml -t pg_conf                                  # re-gen patroni config
./pgsql.yml -t patroni_reload -e patroni_reload=true    # reload patroni config

Update vip-manager reference to etcd endpoints (if you are using PGSQL L2 VIP):

./pgsql.yml -t pg_vip # reload vip-manager config

Append Member

ETCD Reference: Add a member

Pigsty can perform etcd cluster expansion with bin/etcd-add script or the etcd.yml playbook.

bin/etcd-add <ip>

You can add new members to existing etcd cluster in 5 steps:

1. issue etcdctl member add command to tell existing cluster that a new member is coming (use learner mode)
2. update inventory group etcd with new instance
3. init the new member with etcd_init=existing, to join the existing cluster rather than create a new one (VERY IMPORTANT)
4. promote the new member from leaner to follower
5. update etcd endpoints reference with reload-config

Manual Approach

etcdctl member add <etcd-?> --learner=true --peer-urls=https://<new_ins_ip>:2380
./etcd.yml -l <new_ins_ip> -e etcd_init=existing
etcdctl member promote <new_ins_server_id>

Automated Approach (Recommended)

Use the bin/etcd-add script to simplify the process:

# Add new members to inventory first, then:
bin/etcd-add <ip1> <ip2> ...  # append specific members to existing cluster

The etcd-add script will:

• Validate IP addresses
• Execute the etcd.yml playbook with appropriate parameters
• Provide safety warnings and countdown timers
• Guide you through post-operation configuration updates

Remove Member

To remove a member from existing etcd cluster, you have two approaches:

Automated Approach (Recommended)

Use the bin/etcd-rm script for simplified removal:

bin/etcd-rm <ip> ...    # remove specific members

Or use the dedicated removal playbook:

./etcd-rm.yml -l <ip>         # remove specific member

Manual Approach

For manual removal, it usually takes 3 steps:

1. remove/uncomment it from inventory and reload config
2. remove it with etcdctl member remove <server_id> command and kick it out of the cluster
3. use the etcd-rm.yml playbook to clean up the instance

Removal Parameters

The etcd_remove role supports several configuration options:

• etcd_safeguard=true: Prevents accidental removal
• etcd_rm_data=true: Removes etcd data directories (default: true)
• etcd_rm_pkg=false: Uninstalls etcd packages (default: false)

Example with custom parameters:

./etcd-rm.yml -l <ip> -e etcd_rm_pkg=true  # also remove packages

4 - Playbook

There’s a built-in playbook: etcd.yml for etcd cluster installation.

etcd.yml

To create a new etcd cluster, run the following playbook:

./etcd.yml    # install etcd cluster on group 'etcd'

bin/etcd-add    # create entire etcd cluster

Here are available sub tasks:

• etcd_assert : generate etcd identity
• etcd_install : install etcd rpm packages
• etcd_dir : create etcd data & conf dir
• etcd_config : generate etcd config
  • etcd_conf : generate etcd main config
  • etcd_cert : generate etcd ssl cert
• etcd_launch : launch etcd service
• etcd_register : register etcd to prometheus

If you want to append a new member to an existing etcd cluster, you have to add it to the config inventory, and run the playbook with etcd_init = existing against the new member:

./etcd.yml -l <new_instance> -e etcd_init=existing

bin/etcd-add <ip> # append new members to existing etcd cluster

It’s usually OK to re-run the playbook, it will update the etcd cluster config and restart etcd instances.

etcd-rm.yml

To remove the etcd cluster, run the following playbook:

./etcd-rm.yml    # remove etcd cluster

Here are available sub tasks:

• etcd_safeguard : check safeguard and abort if enabled
• prometheus : remove etcd targets registration from prometheus
• etcd_leave : try graceful leaving etcd cluster before purge
• etcd_stop : stop and disable etcd service with systemd
• etcd_data : remove etcd data (disable with etcd_rm_data=false)
• etcd_pkg : uninstall etcd packages (enable with etcd_rm_pkg=true)

To remove a member from an existing etcd cluster, you can run the playbook

./etcd-rm.yml -l <ip>

bin/etcd-rm <ip>    # remove specific members from etcd cluster
bin/etcd-rm         # remove entire etcd cluster

The removal playbook uses the new etcd_remove role with configurable parameters:

• etcd_safeguard: Prevents accidental removal when set to true
• etcd_rm_data: Controls whether ETCD data is deleted (true by default)
• etcd_rm_pkg: Controls whether ETCD packages are uninstalled (false by default)

5 - Monitor

Dashboards

The ETCD module provides a monitoring dashboard: Etcd Overview.

ETCD Overview: Overview of the ETCD cluster

This dashboard provides key information about the ETCD status, with the most notable being ETCD Aliveness, which displays the overall service status of the ETCD cluster.

Red bands indicate periods when instances are unavailable, while the blue-gray bands below show when the entire cluster is unavailable.

Alert Rules

Pigsty provides the following two alert rules for the INFRA module:

You can modify or add new etcd alert rules in files/prometheus/rules/etcd.yml.

#==============================================================#
#                         Aliveness                            #
#==============================================================#
# etcd server instance down
- alert: EtcdServerDown
  expr: etcd_up < 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: etcd }
  annotations:
    summary: "CRIT EtcdServerDown {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      etcd_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value }} < 1
      http://g.pigsty/d/etcd-overview

#==============================================================#
#                         Error                                #
#==============================================================#
# Etcd no Leader triggers a P0 alert immediately
# if dcs_failsafe mode is not enabled, this may lead to global outage
- alert: EtcdNoLeader
  expr: min(etcd_server_has_leader) by (cls) < 1
  for: 15s
  labels: { level: 0, severity: CRIT, category: etcd }
  annotations:
    summary: "CRIT EtcdNoLeader: {{ $labels.cls }} {{ $value }}"
    description: |
      etcd_server_has_leader[cls={{ $labels.cls }}] = {{ $value }} < 1
      http://g.pigsty/d/etcd-overview?from=now-5m&to=now&var-cls={{$labels.cls}}

#==============================================================#
#                        Saturation                            #
#==============================================================#
- alert: EtcdQuotaFull
  expr: etcd:cls:quota_usage > 0.90
  for: 1m
  labels: { level: 1, severity: WARN, category: etcd }
  annotations:
    summary: "WARN EtcdQuotaFull: {{ $labels.cls }}"
    description: |
      etcd:cls:quota_usage[cls={{ $labels.cls }}] = {{ $value | printf "%.3f" }} > 90%

#==============================================================#
#                         Latency                              #
#==============================================================#
# etcd network peer rt p95 > 200ms for 1m
- alert: EtcdNetworkPeerRTSlow
  expr: etcd:ins:network_peer_rt_p95_5m > 0.200
  for: 1m
  labels: { level: 2, severity: INFO, category: etcd }
  annotations:
    summary: "INFO EtcdNetworkPeerRTSlow: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      etcd:ins:network_peer_rt_p95_5m[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 200ms
      http://g.pigsty/d/etcd-instance?from=now-10m&to=now&var-cls={{ $labels.cls }}

# Etcd wal fsync rt p95 > 50ms
- alert: EtcdWalFsyncSlow
  expr: etcd:ins:wal_fsync_rt_p95_5m > 0.050
  for: 1m
  labels: { level: 2, severity: INFO, category: etcd }
  annotations:
    summary: "INFO EtcdWalFsyncSlow: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      etcd:ins:wal_fsync_rt_p95_5m[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 50ms
      http://g.pigsty/d/etcd-instance?from=now-10m&to=now&var-cls={{ $labels.cls }}

6 - FAQ

What is the role of the etcd in pigsty?

etcd is a distributed, reliable key-value store used to store the most critical config / consensus data in the deployment. Pigsty uses etcd as the DCS (Distributed Configuration Store) service for Patroni, which will store the high availability status information of the PostgreSQL cluster.

How many etcd instances should I choose?

If more than (include) half of the etcd instances are down, the etcd cluster, and its service will be unavailable.

For example, a 3-node etcd cluster can tolerate at most one node failure, and the other two nodes can still work normally; while a 5-node etcd cluster can tolerate 2 node failures.

Beware that the learner instances in the etcd cluster do not count in the member number. So in a 3-node etcd cluster, if there is a learner instance, the actual member count is 2, so no node failure can be tolerated.

It is advisable to choose an odd number of etcd instances to avoid split-brain scenarios. It is recommended to use 3 or 5 nodes for the production environment.

What is the impact of etcd failure?

If etcd cluster is unavailable, it will affect the control plane of Pigsty, but not the data plane — the existing PostgreSQL cluster will continue to serve, but admin operations through Patroni will not work.

During etcd failure, PostgreSQL HA is unable to perform automatic failover; and most of the Patroni operations will be blocked, such as edit-config, restart, switchover, etc… Admin tasks through Ansible playbooks are usually not affected by etcd failure, such as create database, create user, reload HBA and Service, etc…, and you can always operate the PostgreSQL cluster directly to achieve most of the patroni functions.

Beware that the above description is only applicable to newer versions of Patroni (>=3.0, Pigsty >= 2.0). If you are using an older version of Patroni (<3.0, corresponding to Pigsty version 1.x), etcd / consul failure will cause a serious impact: All PostgreSQL clusters will be demoted and reject write requests, and etcd failure will be amplified as a global PostgreSQL failure. After Patroni 3.0’s DCS Failsafe feature, this situation has been significantly improved.

What data is stored in the etcd cluster?

etcd is only used for PostgreSQL HA consensus in Pigsty, no other data is stored in etcd by default.

These consensus data are managed by Patroni, and when these data are lost in etcd, Patroni will automatically rebuild them.

Thus, by default, the etcd in Pigsty can be regarded as a “stateless service” that is disposable, which brings great convenience to maintenance work.

If you use etcd for other purposes, such as storing metadata for Kubernetes, or storing other data, you need to back up the etcd data yourself and restore the data after the etcd cluster is restored.

How to recover from etcd failure?

Since etcd is disposable in Pigsty, you can quickly stop the bleeding by “restarting” or “redeploying” etcd in case of failure.

To Restart the etcd cluster, you can use the following Ansible command (or systemctl restart etcd):

./etcd.yml -t etcd_launch

To Reset the etcd cluster, you can run this playbook, it will nuke the etcd cluster and redeploy it:

./etcd-rm.yml       # nuke the existing cluster
./etcd.yml          # create a new cluster

Beware that if you use etcd to store other data, don’t forget to backup etcd data before nuking the etcd cluster.

Is any maintenance work for etcd cluster?

In short: do not use all the quota of etcd.

etcd has a default quota for database size of 2GB, if your etcd database size exceeds this limit, etcd will reject write requests. Meanwhile, as etcd’s data model illustrates, each write will generate a new version (a.k.a. revision), so if your etcd cluster writes frequently, even with very few keys, the etcd database size may continue to grow, and may fail when it reaches the quota limit.

You can achieve this by Auto Compact, Manual Compact, Defragmentation, and Quota Increase, etc., please refer to the etcd official maintenance guide.

Pigsty has auto compact enabled by default since v2.6, so you usually don’t have to worry about etcd full. For versions before v2.6, we strongly recommend enabling etcd’s auto compact feature in the production environment.

Fill etcd may lead to PostgreSQL failure!

For Pigsty v2.0 - v2.5 users, we strongly recommend upgrading to a newer version, or following the instructions below to enable etcd auto compaction!

How to enable etcd auto compaction?

If you are using an earlier version of Pigsty (v2.0 - v2.5), we strongly recommend that you enable etcd’s auto compaction feature in the production environment.

Edit the etcd config template in roles/etcd/templates/etcd.conf with these 3 new lines:

auto-compaction-mode: periodic
auto-compaction-retention: "24h"
quota-backend-bytes: 17179869184

You can set all the PostgreSQL cluster to maintenance mode and then redeploy the etcd cluster with ./etcd.yml to apply the these changes.

It will increase the etcd default quota from 2 GiB to 16 GiB, and ensure that only the most recent day’s write history is retained, avoiding the infinite growth of the etcd database size.

Where does the PostgreSQL HA data store in etcd?

Patroni will use the pg_namespace (default is /pg) as the prefix for all metadata keys in etcd, followed by the PostgreSQL cluster name.

For example, a PG cluster named pg-meta, its metadata keys will be stored under /pg/pg-meta, which may look like this:

/pg/pg-meta/config
{"ttl":30,"loop_wait":10,"retry_timeout":10,"primary_start_timeout":10,"maximum_lag_on_failover":1048576,"maximum_lag_on_syncnode":-1,"primary_stop_timeout":30,"synchronous_mode":false,"synchronous_mode_strict":false,"failsafe_mode":true,"pg_version":16,"pg_cluster":"pg-meta","pg_shard":"pg-meta","pg_group":0,"postgresql":{"use_slots":true,"use_pg_rewind":true,"remove_data_directory_on_rewind_failure":true,"parameters":{"max_connections":100,"superuser_reserved_connections":10,"max_locks_per_transaction":200,"max_prepared_transactions":0,"track_commit_timestamp":"on","wal_level":"logical","wal_log_hints":"on","max_worker_processes":16,"max_wal_senders":50,"max_replication_slots":50,"password_encryption":"scram-sha-256","ssl":"on","ssl_cert_file":"/pg/cert/server.crt","ssl_key_file":"/pg/cert/server.key","ssl_ca_file":"/pg/cert/ca.crt","shared_buffers":"7969MB","maintenance_work_mem":"1993MB","work_mem":"79MB","max_parallel_workers":8,"max_parallel_maintenance_workers":2,"max_parallel_workers_per_gather":0,"hash_mem_multiplier":8.0,"huge_pages":"try","temp_file_limit":"7GB","vacuum_cost_delay":"20ms","vacuum_cost_limit":2000,"bgwriter_delay":"10ms","bgwriter_lru_maxpages":800,"bgwriter_lru_multiplier":5.0,"min_wal_size":"7GB","max_wal_size":"28GB","max_slot_wal_keep_size":"42GB","wal_buffers":"16MB","wal_writer_delay":"20ms","wal_writer_flush_after":"1MB","commit_delay":20,"commit_siblings":10,"checkpoint_timeout":"15min","checkpoint_completion_target":0.8,"archive_mode":"on","archive_timeout":300,"archive_command":"pgbackrest --stanza=pg-meta archive-push %p","max_standby_archive_delay":"10min","max_standby_streaming_delay":"3min","wal_receiver_status_interval":"1s","hot_standby_feedback":"on","wal_receiver_timeout":"60s","max_logical_replication_workers":8,"max_sync_workers_per_subscription":6,"random_page_cost":1.1,"effective_io_concurrency":1000,"effective_cache_size":"23907MB","default_statistics_target":200,"log_destination":"csvlog","logging_collector":"on","log_directory":"/pg/log/postgres","log_filename":"postgresql-%Y-%m-%d.log","log_checkpoints":"on","log_lock_waits":"on","log_replication_commands":"on","log_statement":"ddl","log_min_duration_statement":100,"track_io_timing":"on","track_functions":"all","track_activity_query_size":8192,"log_autovacuum_min_duration":"1s","autovacuum_max_workers":2,"autovacuum_naptime":"1min","autovacuum_vacuum_cost_delay":-1,"autovacuum_vacuum_cost_limit":-1,"autovacuum_freeze_max_age":1000000000,"deadlock_timeout":"50ms","idle_in_transaction_session_timeout":"10min","shared_preload_libraries":"timescaledb, pg_stat_statements, auto_explain","auto_explain.log_min_duration":"1s","auto_explain.log_analyze":"on","auto_explain.log_verbose":"on","auto_explain.log_timing":"on","auto_explain.log_nested_statements":true,"pg_stat_statements.max":5000,"pg_stat_statements.track":"all","pg_stat_statements.track_utility":"off","pg_stat_statements.track_planning":"off","timescaledb.telemetry_level":"off","timescaledb.max_background_workers":8,"citus.node_conninfo":"sslm
ode=prefer"}}}
/pg/pg-meta/failsafe
{"pg-meta-2":"http://10.10.10.11:8008/patroni","pg-meta-1":"http://10.10.10.10:8008/patroni"}
/pg/pg-meta/initialize
7418384210787662172
/pg/pg-meta/leader
pg-meta-1
/pg/pg-meta/members/pg-meta-1
{"conn_url":"postgres://10.10.10.10:5432/postgres","api_url":"http://10.10.10.10:8008/patroni","state":"running","role":"primary","version":"4.0.1","tags":{"clonefrom":true,"version":"16","spec":"8C.32G.125G","conf":"tiny.yml"},"xlog_location":184549376,"timeline":1}
/pg/pg-meta/members/pg-meta-2
{"conn_url":"postgres://10.10.10.11:5432/postgres","api_url":"http://10.10.10.11:8008/patroni","state":"running","role":"replica","version":"4.0.1","tags":{"clonefrom":true,"version":"16","spec":"8C.32G.125G","conf":"tiny.yml"},"xlog_location":184549376,"replication_state":"streaming","timeline":1}
/pg/pg-meta/status
{"optime":184549376,"slots":{"pg_meta_2":184549376,"pg_meta_1":184549376},"retain_slots":["pg_meta_1","pg_meta_2"]}

How to use existing external etcd cluster?

The hard-coded group, etcd, will be used as DCS servers for PGSQL. You can initialize them with etcd.yml or assume it is an existing external etcd cluster.

To use an existing external etcd cluster, define them as usual and make sure your current etcd cluster certificate is signed by the same CA as your self-signed CA for PGSQL.

How to add a new member to the existing etcd cluster?

Check Add a member to etcd cluster

etcdctl member add <etcd-?> --learner=true --peer-urls=https://<new_ins_ip>:2380 # on admin node
./etcd.yml -l <new_ins_ip> -e etcd_init=existing                                 # init new etcd member
etcdctl member promote <new_ins_server_id>                                       # on admin node

How to remove a member from an existing etcd cluster?

Check Remove member from etcd cluster

etcdctl member remove <etcd_server_id>   # kick member out of the cluster (on admin node)
./etcd.yml -l <ins_ip> -t etcd_purge     # purge etcd instance