# Observability Infra

> Modern observability stack with 3000+ metrics, 30+ dashboards, and enterprise-grade monitoring
---

Pigsty delivers **unparalleled observability** with a modern monitoring stack built on industry best practices.
Automatically monitor every component with **3000+ metrics**, **30+ dashboards**.

![](/img/pigsty/dashboard.gif)

**Note**

**Complete Insight**: Monitor everything from high-level cluster health to individual table statistics. Get complete insight into the past, present, and future of your infrastructure.


## Architecture Overview

![](/img/pigsty/infra.png)

Pigsty's observability infrastructure leverages battle-tested open-source components in a cohesive, production-ready stack:


**Grafana Visualization Engine**


Dashboards with advanced interactive visualizations



**Prometheus Metrics Database**


Time-series storage with powerful query language



**Loki Logging Platform**


Centralized logging with label-based indexing



**AlertManager**


Alert aggregate, management and escalation





### Service Architecture

```mermaid
graph TB
    subgraph "Observability Stack"
        Grafana[Grafana :3000]
        Prometheus[Prometheus :9058]
        Loki[Loki :3100]
        AlertManager[AlertManager :9059]
        Pushgateway[Pushgateway :9091]
        Blackbox[Blackbox :9115]
    end

    subgraph "Data Sources"
        PG[(PostgreSQL)]
        Node[Node Metrics]
        Redis[(Redis)]
        MinIO[(MinIO)]
    end

    subgraph "Exporters"
        PGExp[pg_exporter]
        NodeExp[node_exporter]
        RedisExp[redis_exporter]
        MinIOExp[minio_exporter]
    end

    PG --> PGExp
    Node --> NodeExp
    Redis --> RedisExp
    MinIO --> MinIOExp

    PGExp --> Prometheus
    NodeExp --> Prometheus
    RedisExp --> Prometheus
    MinIOExp --> Prometheus

    Prometheus --> Grafana
    Prometheus --> AlertManager
    Loki --> Grafana
```

--------

## Monitoring Dashboards

### Multi-Level Dashboard Hierarchy

Pigsty provides **26+ PostgreSQL dashboards** organized in a logical drill-down hierarchy:

**Overview Level**

```yaml
# Global overview dashboards
dashboards:
  - Home: Global cluster overview and key metrics
  - INFRA: Infrastructure services status
  - NODES: Node-level resource utilization
  - Alert: Active alerts and notification status
```
**Purpose**: High-level operational visibility across entire environment
**Audience**: Operations teams, management dashboards

**Cluster Level**

```yaml
# Cluster-focused dashboards
dashboards:
  - PGSQL Cluster: Cluster health and replication status
  - PGSQL Service: Service endpoints and load balancing
  - PGSQL Activity: Connection pools and query activity
  - PGSQL Replication: Streaming replication metrics
```
**Purpose**: Cluster-wide PostgreSQL performance and health
**Audience**: Database administrators, SRE teams

**Instance Level**

```yaml
# Instance-specific dashboards
dashboards:
  - PGSQL Instance: Detailed PostgreSQL server metrics
  - PGSQL Persist: WAL, checkpoints, and persistence
  - PGSQL Proxy: Pgbouncer connection pooling metrics
  - PGSQL Session: Active sessions and lock analysis
```
**Purpose**: Deep-dive into individual PostgreSQL instances
**Audience**: Database developers, performance engineers

**Database Level**

```yaml
# Database and object-level dashboards
dashboards:
  - PGSQL Database: Database-specific performance metrics
  - PGSQL Table: Table statistics and access patterns
  - PGSQL Query: Query performance and optimization
  - PGSQL Slow: Slow query analysis and tuning
```
**Purpose**: Application-level database performance analysis
**Audience**: Application developers, database analysts

### Dashboard Features


**Drill-Down Navigation**


**Seamless exploration** from overview to granular details with contextual linking



**Time Range Controls**


**Flexible time windows** from real-time to historical analysis over months



**Multi-Dimensional Filtering**


**Dynamic filtering** by cluster, instance, database, or custom labels



**Alert Integration**


**Visual alert correlation** with metrics and direct links to alert details





--------

## Grafana Deployment

### Enhanced Grafana Stack

Pigsty extends Grafana with powerful plugins and data sources for advanced analytics:

**Core Plugins**

```yaml
# Essential Grafana plugins
grafana_plugins:
  - grafana-piechart-panel        # Pie chart visualizations
  - grafana-polystat-panel        # Multi-value status panels
  - grafana-worldmap-panel        # Geographic visualizations
  - grafana-clock-panel           # Time display widgets
```
**Purpose**: Essential visualization capabilities for monitoring dashboards

**Visualization Plugins**

```yaml
# Advanced visualization plugins
grafana_plugins:
  - echarts-panel                 # Apache ECharts integration
  - volkovlabs-echarts-panel      # Enhanced ECharts support
  - volkovlabs-form-panel         # Interactive forms
  - volkovlabs-variable-panel     # Dynamic variables
```
**Purpose**: Rich, interactive visualizations for complex data analysis

**Data Sources**

```yaml
# Extended data source support
grafana_datasources:
  - infinity-datasource           # REST API and file data sources
  - redis-datasource              # Redis data source
  - clickhouse-datasource         # ClickHouse integration
  - postgres-datasource           # Enhanced PostgreSQL support
```
**Purpose**: Connect to diverse data sources beyond traditional metrics

**Custom Extensions**

```yaml
# Pigsty-specific customizations
custom_features:
  - pigsty-theme                  # Custom branding and colors
  - dashboard-provisioning       # Automated dashboard deployment
  - alert-templates               # Pre-configured alert rules
  - data-link-automation          # Context-aware navigation
```
**Purpose**: Tailored user experience optimized for PostgreSQL environments

### Configuration & Customization

```yaml
# Advanced Grafana configuration
grafana_config:
  # Authentication
  auth.anonymous.enabled: true
  auth.anonymous.org_role: Viewer
  auth.disable_login_form: false

  # Security
  security.allow_embedding: true
  security.cookie_secure: true
  security.cookie_samesite: strict

  # Performance
  database.max_open_conn: 300
  database.max_idle_conn: 300
  database.conn_max_lifetime: 14400

  # Alerting
  alerting.enabled: true
  alerting.execute_alerts: true
  unified_alerting.enabled: true

  # Custom panels
  panels.enable_alpha: true
  feature_toggles.enable: ngalert,live,publicDashboards
```

--------

## Prometheus Stack

### Complete Monitoring Ecosystem

Pigsty deploys the full Prometheus ecosystem for comprehensive observability:



### Step 1
### Prometheus Server
**Core metrics database** with advanced querying and storage capabilities
```yaml
# Prometheus configuration highlights
prometheus_config:
  global:
    scrape_interval: 15s          # Default scrape frequency
    evaluation_interval: 15s      # Rule evaluation frequency
    external_labels:
      cluster: '{{ pg_cluster }}'

  rule_files:
    - "/etc/prometheus/rules/*.yml"

  scrape_configs:
    - job_name: 'node'            # Node-level metrics
    - job_name: 'postgres'        # PostgreSQL metrics
    - job_name: 'redis'           # Redis metrics
    - job_name: 'pushgateway'     # Batch job metrics
```

### Step 2
### AlertManager
**Intelligent alert routing** with suppression, grouping, and escalation
```yaml
# AlertManager routing configuration
alertmanager_routes:
  - match:
      severity: critical
    receiver: pagerduty-critical
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h

  - match:
      severity: warning
    receiver: slack-warnings
    group_wait: 1m
    group_interval: 10m
    repeat_interval: 24h
```

### Step 3
### Pushgateway
**Batch job metrics collection** for ephemeral workloads and cron jobs
```bash
# Example: Backup job metrics
echo "backup_duration_seconds $(date +%s)" | curl --data-binary @- \
  http://pushgateway:9091/metrics/job/pg-backup/instance/pg-test
```

### Step 4
### Blackbox Exporter
**Network connectivity monitoring** with HTTP, TCP, and ICMP probes
```yaml
# Blackbox probe configuration
blackbox_probes:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]

  tcp_connect:
    prober: tcp
    timeout: 5s
```



### Pre-configured Alert Rules

```yaml
# Sample PostgreSQL alert rules
alert_rules:
  - alert: PostgreSQLDown
    expr: pg_up == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "PostgreSQL instance {{ $labels.instance }} is down"

  - alert: PostgreSQLHighConnections
    expr: pg_stat_database_numbackends / pg_settings_max_connections > 0.8
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High connection usage on {{ $labels.instance }}"

  - alert: PostgreSQLReplicationLag
    expr: pg_replication_lag_seconds > 300
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Replication lag > 5 minutes on {{ $labels.instance }}"
```

--------

## pg_exporter: Advanced PostgreSQL Monitoring

### Custom Metrics Engine

Pigsty's [pg_exporter](https://github.com/pgsty/pg_exporter) is a highly customizable PostgreSQL metrics collector supporting **all PostgreSQL versions** with fine-grained metric control:

**Core Features**

```yaml
# pg_exporter key capabilities
features:
  - auto_discovery: true          # Automatic database discovery
  - custom_queries: true          # User-defined metric queries
  - version_aware: true           # PostgreSQL version detection
  - rds_compatible: true          # Cloud database support
  - label_customization: true     # Flexible metric labeling
  - connection_pooling: true      # Efficient connection reuse
```
**Advantages**: Flexible, lightweight, and highly configurable

**Version Compatibility**

```yaml
# PostgreSQL version support matrix
supported_versions:
  - postgresql_9_6: legacy_metrics_set
  - postgresql_10: enhanced_metrics_set
  - postgresql_11: advanced_metrics_set
  - postgresql_12: modern_metrics_set
  - postgresql_13: extended_metrics_set
  - postgresql_14: latest_metrics_set
  - postgresql_15: cutting_edge_metrics_set
  - postgresql_16: next_gen_metrics_set
```
**Benefit**: Single exporter for heterogeneous PostgreSQL environments

**Custom Queries**

```yaml
# Custom metric definition example
custom_queries:
  pg_custom_business_metrics:
    query: |
      SELECT
        schemaname,
        tablename,
        n_tup_ins as inserts_total,
        n_tup_upd as updates_total,
        n_tup_del as deletes_total
      FROM pg_stat_user_tables
    metrics:
      - inserts_total:
          usage: COUNTER
          description: "Total number of inserts"
      - updates_total:
          usage: COUNTER
          description: "Total number of updates"
```

**RDS Support**

```yaml
# RDS monitoring configuration
rds_monitoring:
  connection_string: "postgres://monitor:password@rds.region.rds.amazonaws.com:5432/postgres"
  metrics_subset: rds_safe        # RDS-compatible metrics only
  auto_discovery: false           # Manual database specification
  query_timeout: 30s              # Conservative timeout

  # RDS-specific metrics
  included_databases: [production, staging]
  excluded_schemas: [information_schema, pg_catalog]
```

### Metrics Configuration

```yaml
# Comprehensive pg_exporter configuration
pg_exporter_config:
  # Connection settings
  data_source_name: "postgres://dbuser_monitor:password@localhost:5432/postgres"

  # Metric collection
  auto_discover_databases: true
  exclude_databases: [template0, template1]
  include_databases: [postgres, business_db]

  # Query customization
  query_path: "/etc/pg_exporter/queries"
  metric_prefix: "pg"

  # Performance tuning
  parallel_scrape: true
  scrape_timeout: 30s
  max_connections: 3

  # Security
  ssl_mode: require
  ssl_cert: "/etc/ssl/pg-client.crt"
  ssl_key: "/etc/ssl/pg-client.key"
```

--------

## Host & Infrastructure Monitoring

Pigsty v3.7 installs `node_exporter` on managed nodes and registers Node,
HAProxy, Keepalived, Nginx, Etcd, MinIO, Redis, PostgreSQL, PgBouncer, and
pgBackRest targets with Prometheus when the corresponding modules are
enabled. Component ports and switches are defined by the tagged role
defaults and documented on each module's parameter page.

## External Database Monitoring

Existing PostgreSQL instances and cloud RDS can be registered with
`pgsql-monitor.yml`. Supply a monitor connection URL with only the required
`pg_monitor` privileges; Pigsty registers pg_exporter and the Grafana
datasource without provisioning or changing that database.

```bash
./pgsql-monitor.yml -e pg_exporters='{"rds":{"pg_exporter_url":"postgres://dbuser_monitor:DBUser.Monitor@10.10.10.10:5432/postgres"}}'
```

## Data Analytics & Visualization Platform

Grafana can query both Prometheus metrics and PostgreSQL data. The bundled
dashboards use variables and URL links for drill-down navigation; ECharts
and other packaged plugins can visualize application or business data.

## Low-Code Application Development

Grafana panels may be combined with PostgreSQL queries to build internal
operational views. This is a visualization capability, not a separate
Pigsty deployment module or application API.

## Reusable Infrastructure

`infra.yml` can deploy INFRA independently from PGSQL. Nginx, DNSMasq,
Prometheus, AlertManager, Grafana, and Loki are controlled by their
documented `*_enabled` parameters, allowing Pigsty to reuse or coexist with
existing infrastructure.

## Best Practices

- Control metric cardinality and set retention according to available disk.
- Back up Grafana configuration and keep alert notification channels tested.
- Protect monitoring endpoints with access controls and HTTPS when exposed
  outside the trusted network.
- Monitor the monitoring stack itself and review storage growth regularly.

## Limitations & Considerations

Prometheus and Loki in v3.7 are single-node services by default. Very large
metric cardinality, long retention, or complex dashboards require capacity
planning. External long-term storage and HA monitoring architectures are
manual integrations rather than built-in v3.7 modules.