Observability Infra

Pigsty delivers unparalleled observability with a modern monitoring stack built on industry best practices. Automatically monitor every component with 3000+ metrics, 30+ dashboards.

Architecture Overview

Pigsty’s observability infrastructure leverages battle-tested open-source components in a cohesive, production-ready stack:

Service Architecture

graph TB
    subgraph "Observability Stack"
        Grafana[Grafana :3000]
        Prometheus[Prometheus :9058]
        Loki[Loki :3100]
        AlertManager[AlertManager :9059]
        Pushgateway[Pushgateway :9091]
        Blackbox[Blackbox :9115]
    end

    subgraph "Data Sources"
        PG[(PostgreSQL)]
        Node[Node Metrics]
        Redis[(Redis)]
        MinIO[(MinIO)]
    end

    subgraph "Exporters"
        PGExp[pg_exporter]
        NodeExp[node_exporter]
        RedisExp[redis_exporter]
        MinIOExp[minio_exporter]
    end

    PG --> PGExp
    Node --> NodeExp
    Redis --> RedisExp
    MinIO --> MinIOExp

    PGExp --> Prometheus
    NodeExp --> Prometheus
    RedisExp --> Prometheus
    MinIOExp --> Prometheus

    Prometheus --> Grafana
    Prometheus --> AlertManager
    Loki --> Grafana

Monitoring Dashboards

Multi-Level Dashboard Hierarchy

Pigsty provides 26+ PostgreSQL dashboards organized in a logical drill-down hierarchy:

# Global overview dashboards
dashboards:
  - Home: Global cluster overview and key metrics
  - INFRA: Infrastructure services status
  - NODES: Node-level resource utilization
  - Alert: Active alerts and notification status

# Cluster-focused dashboards
dashboards:
  - PGSQL Cluster: Cluster health and replication status
  - PGSQL Service: Service endpoints and load balancing
  - PGSQL Activity: Connection pools and query activity
  - PGSQL Replication: Streaming replication metrics

# Instance-specific dashboards
dashboards:
  - PGSQL Instance: Detailed PostgreSQL server metrics
  - PGSQL Persist: WAL, checkpoints, and persistence
  - PGSQL Proxy: Pgbouncer connection pooling metrics
  - PGSQL Session: Active sessions and lock analysis

# Database and object-level dashboards
dashboards:
  - PGSQL Database: Database-specific performance metrics
  - PGSQL Table: Table statistics and access patterns
  - PGSQL Query: Query performance and optimization
  - PGSQL Slow: Slow query analysis and tuning

Dashboard Features

Grafana Deployment

Enhanced Grafana Stack

Pigsty extends Grafana with powerful plugins and data sources for advanced analytics:

# Essential Grafana plugins
grafana_plugins:
  - grafana-piechart-panel        # Pie chart visualizations
  - grafana-polystat-panel        # Multi-value status panels
  - grafana-worldmap-panel        # Geographic visualizations
  - grafana-clock-panel           # Time display widgets

# Advanced visualization plugins
grafana_plugins:
  - echarts-panel                 # Apache ECharts integration
  - volkovlabs-echarts-panel      # Enhanced ECharts support
  - volkovlabs-form-panel         # Interactive forms
  - volkovlabs-variable-panel     # Dynamic variables

# Extended data source support
grafana_datasources:
  - infinity-datasource           # REST API and file data sources
  - redis-datasource              # Redis data source
  - clickhouse-datasource         # ClickHouse integration
  - postgres-datasource           # Enhanced PostgreSQL support

# Pigsty-specific customizations
custom_features:
  - pigsty-theme                  # Custom branding and colors
  - dashboard-provisioning       # Automated dashboard deployment
  - alert-templates               # Pre-configured alert rules
  - data-link-automation          # Context-aware navigation

Configuration & Customization

# Advanced Grafana configuration
grafana_config:
  # Authentication
  auth.anonymous.enabled: true
  auth.anonymous.org_role: Viewer
  auth.disable_login_form: false

  # Security
  security.allow_embedding: true
  security.cookie_secure: true
  security.cookie_samesite: strict

  # Performance
  database.max_open_conn: 300
  database.max_idle_conn: 300
  database.conn_max_lifetime: 14400

  # Alerting
  alerting.enabled: true
  alerting.execute_alerts: true
  unified_alerting.enabled: true

  # Custom panels
  panels.enable_alpha: true
  feature_toggles.enable: ngalert,live,publicDashboards

Prometheus Stack

Complete Monitoring Ecosystem

Pigsty deploys the full Prometheus ecosystem for comprehensive observability:

# Prometheus configuration highlights
prometheus_config:
  global:
    scrape_interval: 15s          # Default scrape frequency
    evaluation_interval: 15s      # Rule evaluation frequency
    external_labels:
      cluster: '{{ pg_cluster }}'

  rule_files:
    - "/etc/prometheus/rules/*.yml"

  scrape_configs:
    - job_name: 'node'            # Node-level metrics
    - job_name: 'postgres'        # PostgreSQL metrics
    - job_name: 'redis'           # Redis metrics
    - job_name: 'pushgateway'     # Batch job metrics

# AlertManager routing configuration
alertmanager_routes:
  - match:
      severity: critical
    receiver: pagerduty-critical
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h

  - match:
      severity: warning
    receiver: slack-warnings
    group_wait: 1m
    group_interval: 10m
    repeat_interval: 24h

# Example: Backup job metrics
echo "backup_duration_seconds $(date +%s)" | curl --data-binary @- \
  http://pushgateway:9091/metrics/job/pg-backup/instance/pg-test

# Blackbox probe configuration
blackbox_probes:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]

  tcp_connect:
    prober: tcp
    timeout: 5s

Pre-configured Alert Rules

# Sample PostgreSQL alert rules
alert_rules:
  - alert: PostgreSQLDown
    expr: pg_up == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "PostgreSQL instance {{ $labels.instance }} is down"

  - alert: PostgreSQLHighConnections
    expr: pg_stat_database_numbackends / pg_settings_max_connections > 0.8
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High connection usage on {{ $labels.instance }}"

  - alert: PostgreSQLReplicationLag
    expr: pg_replication_lag_seconds > 300
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Replication lag > 5 minutes on {{ $labels.instance }}"

pg_exporter: Advanced PostgreSQL Monitoring

Custom Metrics Engine

Pigsty’s pg_exporter is a highly customizable PostgreSQL metrics collector supporting all PostgreSQL versions with fine-grained metric control:

# pg_exporter key capabilities
features:
  - auto_discovery: true          # Automatic database discovery
  - custom_queries: true          # User-defined metric queries
  - version_aware: true           # PostgreSQL version detection
  - rds_compatible: true          # Cloud database support
  - label_customization: true     # Flexible metric labeling
  - connection_pooling: true      # Efficient connection reuse

# PostgreSQL version support matrix
supported_versions:
  - postgresql_9_6: legacy_metrics_set
  - postgresql_10: enhanced_metrics_set
  - postgresql_11: advanced_metrics_set
  - postgresql_12: modern_metrics_set
  - postgresql_13: extended_metrics_set
  - postgresql_14: latest_metrics_set
  - postgresql_15: cutting_edge_metrics_set
  - postgresql_16: next_gen_metrics_set

# Custom metric definition example
custom_queries:
  pg_custom_business_metrics:
    query: |
      SELECT
        schemaname,
        tablename,
        n_tup_ins as inserts_total,
        n_tup_upd as updates_total,
        n_tup_del as deletes_total
      FROM pg_stat_user_tables
    metrics:
      - inserts_total:
          usage: COUNTER
          description: "Total number of inserts"
      - updates_total:
          usage: COUNTER
          description: "Total number of updates"

# RDS monitoring configuration
rds_monitoring:
  connection_string: "postgres://monitor:[email protected]:5432/postgres"
  metrics_subset: rds_safe        # RDS-compatible metrics only
  auto_discovery: false           # Manual database specification
  query_timeout: 30s              # Conservative timeout

  # RDS-specific metrics
  included_databases: [production, staging]
  excluded_schemas: [information_schema, pg_catalog]

Metrics Configuration

# Comprehensive pg_exporter configuration
pg_exporter_config:
  # Connection settings
  data_source_name: "postgres://dbuser_monitor:password@localhost:5432/postgres"

  # Metric collection
  auto_discover_databases: true
  exclude_databases: [template0, template1]
  include_databases: [postgres, business_db]

  # Query customization
  query_path: "/etc/pg_exporter/queries"
  metric_prefix: "pg"

  # Performance tuning
  parallel_scrape: true
  scrape_timeout: 30s
  max_connections: 3

  # Security
  ssl_mode: require
  ssl_cert: "/etc/ssl/pg-client.crt"
  ssl_key: "/etc/ssl/pg-client.key"

Host & Infrastructure Monitoring

Pigsty v3.7 installs node_exporter on managed nodes and registers Node, HAProxy, Keepalived, Nginx, Etcd, MinIO, Redis, PostgreSQL, PgBouncer, and pgBackRest targets with Prometheus when the corresponding modules are enabled. Component ports and switches are defined by the tagged role defaults and documented on each module’s parameter page.

External Database Monitoring

Existing PostgreSQL instances and cloud RDS can be registered with pgsql-monitor.yml. Supply a monitor connection URL with only the required pg_monitor privileges; Pigsty registers pg_exporter and the Grafana datasource without provisioning or changing that database.

./pgsql-monitor.yml -e pg_exporters='{"rds":{"pg_exporter_url":"postgres://dbuser_monitor:[email protected]:5432/postgres"}}'

Data Analytics & Visualization Platform

Grafana can query both Prometheus metrics and PostgreSQL data. The bundled dashboards use variables and URL links for drill-down navigation; ECharts and other packaged plugins can visualize application or business data.

Low-Code Application Development

Grafana panels may be combined with PostgreSQL queries to build internal operational views. This is a visualization capability, not a separate Pigsty deployment module or application API.

Reusable Infrastructure

infra.yml can deploy INFRA independently from PGSQL. Nginx, DNSMasq, Prometheus, AlertManager, Grafana, and Loki are controlled by their documented *_enabled parameters, allowing Pigsty to reuse or coexist with existing infrastructure.

Best Practices

• Control metric cardinality and set retention according to available disk.
• Back up Grafana configuration and keep alert notification channels tested.
• Protect monitoring endpoints with access controls and HTTPS when exposed outside the trusted network.
• Monitor the monitoring stack itself and review storage growth regularly.

Limitations & Considerations

Prometheus and Loki in v3.7 are single-node services by default. Very large metric cardinality, long retention, or complex dashboards require capacity planning. External long-term storage and HA monitoring architectures are manual integrations rather than built-in v3.7 modules.