# Administration

> Manage your etcd cluster
---

Here are some administration SOP for etcd:

- [Create Cluster](#create-cluster)
- [Remove Cluster](#remove-cluster)
- [CLI Environment](#cli-environment)
- [Reload Config](#reload-config)
- [Management Scripts](#management-scripts)
- [Append Member](#append-member)
- [Remove Member](#remove-member)

Check [ETCD: FAQ](/docs/etcd/faq) for more questions.


------

## Create Cluster

To create an etcd cluster, define the `etcd` cluster in [inventory](/docs/config/inventory) first:

```yaml
etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars: { etcd_cluster: etcd   }
```

Then run the [`etcd.yml`](/docs/etcd/playbook/#etcdyml) playbook.

```bash
./etcd.yml   # init etcd module on group 'etcd'
```

If there's an existing etcd cluster, this playbook will update the config and **restart** all the etcd instances



Pigsty has a safeguard mechanism to prevent accidental purge. [`etcd_safeguard`](/docs/etcd/param#etcd_safeguard) is `false` by default,

For provisioned etcd cluster in prod env, you can enable safeguard to prevent accidental clean.

**Architecture Change: Pigsty v3.6+**

Since Pigsty v3.6+, the **etcd.yml** playbook and **etcd** role are focused solely on cluster installation and member addition. All removal operations have been moved to the dedicated **etcd-rm.yml** playbook using the **etcd_remove** role.



------

## Remove Cluster

To remove an existing etcd cluster, you can use the dedicated [`etcd-rm.yml`](/docs/etcd/playbook/#etcdyml):

```bash tab="playbook"
./etcd-rm.yml   # remove the default etcd cluster
```
```bash tab="script"
bin/etcd-rm     # remove the default etcd cluster
```

If the [`etcd_safeguard`](/docs/etcd/param#etcd_safeguard) is set to `true`, the playbook will abort.





------

## CLI Environment

Pigsty use etcd v3 API by default. (`v2` support is dropped since v3.6.0)

Here’s an example of client environment config.

```bash
alias e="etcdctl"
alias em="etcdctl member"
export ETCDCTL_ENDPOINTS=https://10.10.10.10:2379
export ETCDCTL_CACERT=/etc/pki/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key
```

You can do CRUD with the following commands after setting up the envs:

```bash
e put a 10 ; e get a; e del a ; # V3 API
```



------

## Reload Config

In case of permanent etcd cluster membership changes, You’ll have to refresh the 4 etcd endpoints references:

- config file of existing etcd members and client env var
- patroni dcs endpoint config
- vip-manager dcs endpoint config

To refresh etcd config file `/etc/etcd/etcd.conf` on existing members and client env vars:

```bash
./etcd.yml -t etcd_conf         # refresh /etc/etcd/etcd.conf with latest status
./etcd.yml -t etcd_launch -f 1  # restart etcd instances one by one
```

Update patroni reference to etcd endpoints:

```bash
./pgsql.yml -t pg_conf                                  # re-gen patroni config
./pgsql.yml -t patroni_reload -e patroni_reload=true    # reload patroni config
```

Update vip-manager reference to etcd endpoints (if you are using PGSQL L2 VIP):

```bash
./pgsql.yml -t pg_vip # reload vip-manager config
```





------

## Append Member

ETCD Reference: [Add a member](https://etcd.io/docs/v3.6/op-guide/runtime-configuration/#add-a-new-member)

Pigsty can perform etcd cluster expansion with [`bin/etcd-add`](https://github.com/pgsty/pigsty/blob/v3.7.0/bin/etcd-add) script or the [`etcd.yml`](/docs/etcd/playbook#etcdyml) playbook.

```bash
bin/etcd-add <ip>
```

You can add new members to existing etcd cluster in 5 steps:

1. issue `etcdctl member add` command to tell existing cluster that a new member is coming (use learner mode)
2. update inventory group `etcd` with new instance
3. init the new member with `etcd_init=existing`, to join the existing cluster rather than create a new one (**VERY IMPORTANT**)
4. promote the new member from leaner to follower
5. update etcd endpoints reference with [reload-config](#reload-config)

**Manual Approach**

```bash
etcdctl member add <etcd-?> --learner=true --peer-urls=https://<new_ins_ip>:2380
./etcd.yml -l <new_ins_ip> -e etcd_init=existing
etcdctl member promote <new_ins_server_id>
```

**Automated Approach (Recommended)**

Use the `bin/etcd-add` script to simplify the process:

```bash
# Add new members to inventory first, then:
bin/etcd-add <ip1> <ip2> ...  # append specific members to existing cluster
```

The `etcd-add` script will:
- Validate IP addresses
- Execute the etcd.yml playbook with appropriate parameters
- Provide safety warnings and countdown timers
- Guide you through post-operation configuration updates


------

## Remove Member

To remove a member from existing etcd cluster, you have two approaches:

**Automated Approach (Recommended)**

Use the `bin/etcd-rm` script for simplified removal:

```bash
bin/etcd-rm <ip> ...    # remove specific members
```

Or use the dedicated removal playbook:

```bash
./etcd-rm.yml -l <ip>         # remove specific member
```

**Manual Approach**

For manual removal, it usually takes 3 steps:

1. remove/uncomment it from inventory and [reload config](#reload-config)
2. remove it with `etcdctl member remove <server_id>` command and kick it out of the cluster
3. use the etcd-rm.yml playbook to clean up the instance

**Removal Parameters**

The etcd_remove role supports several configuration options:

- `etcd_safeguard=true`: Prevents accidental removal
- `etcd_rm_data=true`: Removes etcd data directories (default: true)
- `etcd_rm_pkg=false`: Uninstalls etcd packages (default: false)

Example with custom parameters:

```bash
./etcd-rm.yml -l <ip> -e etcd_rm_pkg=true  # also remove packages
```