# Multi-Node > how to install pigsty on multiple nodes --- There is a configuration [Tutorial](/docs/config/tutorial) for expanding Pigsty from one node to multiple nodes. While the easiest way is always pre-planing everything and provisioning them in one-pass. -------- ## 1-node Setup We already illustrated the 1-node installation in the [Quick Start](/docs/install/start) section, which may be the simplest setup. | ID | IP Address | [NODE](/docs/node) | [PGSQL](/docs/pgsql) | [INFRA](/docs/infra) | [**ETCD**](/docs/etcd) | |:--:|:-------------:|:-------------:|:---------------:|:---------------:|:-----------------:| | 1 | `10.10.10.10` | `meta` | `pg-meta-1` | `infra-1` | `etcd-1` | It is not advised to put all the eggs in one basket, but even this one-node setup can be used for production, as long as an external [MinIO](/docs/minio) / S3 / NFS... is configured for [PG](/docs/pgsql/backup) as remote backup repo. There are lots of 1-node [config templates](/docs/config/template) for your reference. And provision VM with [Vagrant](/zh/docs/prepare/vagrant) [`meta.rb`](https://github.com/pgsty/pigsty/blob/v3.7.0/vagrant/spec/meta.rb) or [Terraform](/zh/docs/prepare/terraform) [`meta.tf`](https://github.com/pgsty/pigsty/blob/v3.7.0/terraform/spec/aliyun-meta.tf). --------- ## 2-node Setup > The semi-HA setup A two-node setup enables database replication and semi-HA capabilities: | ID | IP Address | [NODE](/docs/node) | [PGSQL](/docs/pgsql) | [INFRA](/docs/infra) | [**ETCD**](/docs/etcd) | |:--:|:-------------:|:-------------:|:---------------:|:---------------:|:-----------------:| | 1 | `10.10.10.10` | `meta` | `pg-meta-1` | `infra-1` | `etcd-1` | | 2 | `10.10.10.11` | `node-1` | `pg-meta-2` | | | While more robust than a single node setup, HA has limitations: - No automatic failover if `node-1` fails - manual promotion of `node-2` required - Automatic failover works if `node-2` fails - `node-1` gets promoted automatically This "semi-HA" setup can only auto-recover from specific node failures. You can use the [`dual.yml`](https://github.com/pgsty/pigsty/blob/v3.7.0/conf/ha/dual.yml) config template, and provision required VM with: [Vagrant](/docs/prepare/vagrant) [`dual.rb`](https://github.com/pgsty/pigsty/blob/v3.7.0/vagrant/spec/dual.rb) to provision this environment. --------- ## 3-node Setup > The true HA A true HA setup that can automatically recover from any single node failure: | ID | IP Address | [NODE](/docs/node) | [PGSQL](/docs/pgsql) | [INFRA](/docs/infra) | [**ETCD**](/docs/etcd) | |:--:|:-------------:|:-------------:|:---------------:|:---------------:|:-----------------:| | 1 | `10.10.10.10` | `node-1` | `pg-meta-1` | `infra-1` | `etcd-1` | | 2 | `10.10.10.11` | `node-2` | `pg-meta-2` | `infra-2` | `etcd-2` | | 3 | `10.10.10.12` | `node-3` | `pg-meta-3` | `infra-3` | `etcd-3` | You can use the [`trio.yml`](https://github.com/pgsty/pigsty/blob/v3.7.0/conf/ha/trio.yml) config template, and provision required VM with: [Vagrant](/docs/prepare/vagrant) [`trio.rb`](https://github.com/pgsty/pigsty/blob/v3.7.0/vagrant/spec/trio.rb) to provision this environment. --------- ## 4-node Sandbox This is a [sandbox demo](/docs/prepare/sandbox) environment used in Pigsty, which has one infra node and three extra data nodes: | ID | IP Address | [NODE](/docs/node) | [PGSQL](/docs/pgsql) | [INFRA](/docs/infra) | [**ETCD**](/docs/etcd) | [**MINIO**](/docs/minio) | |:--:|:-------------:|:-------------:|:---------------:|:---------------:|:-----------------:|:--------------------| | 1 | `10.10.10.10` | `meta` | `pg-meta-1` | `infra-1` | `etcd-1` | `minio-1` | | 2 | `10.10.10.11` | `node-1` | `pg-test-1` | | | | | 3 | `10.10.10.12` | `node-2` | `pg-test-1` | | | | | 4 | `10.10.10.13` | `node-3` | `pg-test-1` | | | | You can use the [`full.yml`](https://github.com/pgsty/pigsty/blob/v3.7.0/conf/ha/full.yml) config template, and provision required VM with: [Vagrant](/docs/prepare/vagrant) [`full.rb`](https://github.com/pgsty/pigsty/blob/v3.7.0/vagrant/spec/full.rb) or [Terraform](/docs/prepare/terraform) [`full.tf`](https://github.com/pgsty/pigsty/blob/v3.7.0/terraform/spec/aliyun-full.tf) to provision this environment. --------- ## 5-node Building This [`pro.yml`](https://github.com/pgsty/pigsty/blob/v3.7.0/conf/build/pro.yml) is a 5-node building environment which contains supported [Linux distros](/docs/prepare/linux). | ID | IP Address | [NODE](/docs/node) | [PGSQL](/docs/pgsql) | [INFRA](/docs/infra) | [**ETCD**](/docs/etcd) | |:--:|:-------------:|:-------------:|:---------------:|:---------------:|:-----------------:| | 1 | `10.10.10.8` | `el8` | `el8-1` | `infra-1` | | | 2 | `10.10.10.9` | `el9` | `el9-1` | `infra-2` | `etcd-2` | | 3 | `10.10.10.12` | `u12` | `u12-1` | `infra-3` | | | 4 | `10.10.10.22` | `u22` | `u22-1` | `infra-4` | | | 5 | `10.10.10.24` | `u24` | `u24-1` | `infra-5` | | You can use the [Vagrant](/docs/prepare/vagrant) [`pro.rb`](https://github.com/pgsty/pigsty/blob/v3.7.0/vagrant/spec/pro.rb) or [Terraform](/docs/prepare/terraform) [`pro.tf`](https://github.com/pgsty/pigsty/blob/v3.7.0/terraform/spec/aliyun-pro.tf) to provision this environment. --------- ## 36-node Simulation A production simulation environment ([`simu.yml`](https://github.com/pgsty/pigsty/blob/v3.7.0/conf/ha/simu.yml)) with 36 nodes covering all Pigsty components | IP Address | SPEC | [NODE](/docs/node) | [PGSQL](/docs/pgsql) | [INFRA](/docs/infra) | [**ETCD**](/docs/etcd) | [**MINIO**](/docs/minio) | [**REDIS**](/docs/redis) | |:--------------|:-----:|:-------------:|:---------------:|:---------------:|:-----------------:|:-------------------:|:-------------------:| | `10.10.10.10` | 8C32G | `meta1` | `pg-meta-1` | `infra-1` | | | | | `10.10.10.11` | 8C32G | `meta2` | `pg-meta-2` | `infra-2` | | | | | `10.10.10.12` | 2C4G | `pg12` | `pg-v12-1` | | | | | | `10.10.10.13` | 2C4G | `pg13` | `pg-v13-1` | | | | | | `10.10.10.14` | 2C4G | `pg14` | `pg-v14-1` | | | | | | `10.10.10.15` | 2C4G | `pg15` | `pg-v15-1` | | | | | | `10.10.10.16` | 2C4G | `pg16` | `pg-v16-1` | | | | | | `10.10.10.17` | 2C4G | `pg17` | `pg-v17-1` | | | | | | `10.10.10.18` | 2C4G | `proxy1` | | | | | | | `10.10.10.19` | 2C4G | `proxy2` | | | | | | | `10.10.10.21` | 2C4G | `minio1` | | | `etcd-1` | `minio-1` | `redis-meta-1` | | `10.10.10.22` | 2C4G | `minio2` | | | `etcd-2` | `minio-2` | `redis-meta-2` | | `10.10.10.23` | 2C4G | `minio3` | | | `etcd-3` | `minio-3` | `redis-meta-3` | | `10.10.10.24` | 2C4G | `minio4` | | | `etcd-4` | `minio-4` | `redis-meta-4` | | `10.10.10.25` | 2C4G | `minio5` | | | `etcd-5` | `minio-5` | `redis-meta-5` | | `10.10.10.40` | 1C2G | `node40` | `pg-pitr-1` | | | | | | `10.10.10.41` | 1C2G | `node41` | `pg-test-1` | | | | `redis-test-1` | | `10.10.10.42` | 1C2G | `node42` | `pg-test-2` | | | | `redis-test-2` | | `10.10.10.43` | 1C2G | `node43` | `pg-test-3` | | | | `redis-test-3` | | `10.10.10.44` | 1C2G | `node44` | `pg-test-4` | | | | `redis-test-4` | | `10.10.10.45` | 1C2G | `node45` | `pg-src-1` | | | | `redis-src-1` | | `10.10.10.46` | 1C2G | `node46` | `pg-src-2` | | | | `redis-src-2` | | `10.10.10.47` | 1C2G | `node47` | `pg-src-3` | | | | `redis-src-3` | | `10.10.10.48` | 1C2G | `node48` | `pg-dst-1` | | | | `redis-dst-1` | | `10.10.10.49` | 1C2G | `node49` | `pg-dst-2` | | | | `redis-dst-2` | | `10.10.10.50` | 1C2G | `node50` | `pg-citus0-1` | | | | | | `10.10.10.51` | 1C2G | `node51` | `pg-citus0-2` | | | | | | `10.10.10.52` | 1C2G | `node52` | `pg-citus1-1` | | | | | | `10.10.10.53` | 1C2G | `node53` | `pg-citus1-2` | | | | | | `10.10.10.54` | 1C2G | `node54` | `pg-citus2-1` | | | | | | `10.10.10.55` | 1C2G | `node55` | `pg-citus2-2` | | | | | | `10.10.10.56` | 1C2G | `node56` | `pg-citus3-1` | | | | | | `10.10.10.57` | 1C2G | `node57` | `pg-citus3-2` | | | | | | `10.10.10.58` | 1C2G | `node58` | `pg-citus4-1` | | | | | | `10.10.10.59` | 1C2G | `node59` | `pg-citus4-2` | | | | | | `10.10.10.88` | 4C8G | `test` | | | | | | You can use the [Vagrant](/docs/prepare/vagrant) [`simu.rb`](https://github.com/pgsty/pigsty/blob/v3.7.0/vagrant/spec/simu.rb) to provision this environment. You can run the entire simulation on a real server (72C / 256G) with `libvirt` as VM provider with vagrant. - 2 infra nodes, monitoring each other - 2 dedicated proxy nodes that run haproxy - 5-node etcd cluster which tolerates 2 node failures, and 5-node redis sentinel cluster - 5-node minio cluster with 4 disks on each node - 10 postgres clusters, `pg13` - `pg15`, `pg-src`, `pg-dst`, `pg-pitr`, `pg-test` - 10-node citus cluster with 5 shards - redis standalone cluster `redis-src` and `redis-dst`, and native cluster `redis-test`