Redis

• 1: Configure
• 2: Parameter
• 3: Administration
• 4: Playbook
• 5: Monitor
• 6: FAQ

    Configure redis module, and use multiple redis nodes.

    Customize redis components with 21 parameters

    Create, remove, expand, redis cluster

    Ansible playbooks that can be used in this module

    Dashboards, metrics, record & alerting rules.

    Frequently Asked Questions about redis module

1 - Configure

The entity model of Redis is almost the same as that of PostgreSQL, which also includes the concepts of Cluster and Instance. The Cluster here does not refer to the native Redis Cluster mode.

The core difference between the REDIS module and the PGSQL module is that Redis uses a single-node multi-instance deployment rather than the 1:1 deployment: multiple Redis instances are typically deployed on a physical/virtual machine node to utilize multicore CPUs fully. Therefore, the ways to configure and administer Redis instances are slightly different from PGSQL.

In Redis managed by Pigsty, nodes are entirely subordinate to the cluster, which means that currently, it is not allowed to deploy Redis instances of two different clusters on one node. However, this does not affect deploying multiple independent Redis primary replica instances on one node.

Redis Identity

Redis identity parameters are required parameters when defining a Redis cluster.

Redis Mode

There are three redis_mode available in Pigsty:

• standalone: setup Redis in standalone (master-slave) mode
• cluster: setup this Redis cluster as a Redis native cluster
• sentinel: setup Redis as a sentinel for standalone Redis HA

Here are three examples:

• A 1-node, one master & one slave Redis Standalone cluster: redis-ms
• A 1-node, 3-instance Redis Sentinel cluster: redis-sentinel
• A 2-node, 6-instance Redis Cluster: redis-cluster

redis-ms: # redis classic primary & replica
  hosts: { 10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } } }
  vars: { redis_cluster: redis-ms ,redis_password: 'redis.ms' ,redis_max_memory: 64MB }

redis-meta: # redis sentinel x 3
  hosts: { 10.10.10.11: { redis_node: 1 , redis_instances: { 26379: { } ,26380: { } ,26381: { } } } }
  vars:
    redis_cluster: redis-meta
    redis_password: 'redis.meta'
    redis_mode: sentinel
    redis_max_memory: 16MB
    redis_sentinel_monitor: # primary list for redis sentinel, use cls as name, primary ip:port
      - { name: redis-ms, host: 10.10.10.10, port: 6379 ,password: redis.ms, quorum: 2 }

redis-test: # redis native cluster: 3m x 3s
  hosts:
    10.10.10.12: { redis_node: 1 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
    10.10.10.13: { redis_node: 2 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
  vars: { redis_cluster: redis-test ,redis_password: 'redis.test' ,redis_mode: cluster, redis_max_memory: 32MB }

Limitation

• A Redis node can only belong to one Redis cluster, which means you cannot assign a node to two different Redis clusters simultaneously.
• On each Redis node, you need to assign a unique port number to the Redis instance to avoid port conflicts.
• Typically, the same Redis cluster will use the same password, but multiple Redis instances on a Redis node cannot set different passwords (because redis_exporter only allows one password).
• Redis Cluster has built-in HA, while standalone HA requires manually configured in Sentinel because we are unsure if you have any sentinels available. Fortunately, configuring standalone Redis HA is straightforward: Configure HA with sentinel.

2 - Parameter

There are 21 parameters in the redis module.

Defaults

The default parameters are defined in roles/redis/defaults/main.yml

#redis_cluster:        <CLUSTER> # redis cluster name, required identity parameter
#redis_node: 1            <NODE> # redis node sequence number, node int id required
#redis_instances: {}      <NODE> # redis instances definition on this redis node
redis_fs_main: /data              # redis main data mountpoint, `/data` by default
redis_exporter_enabled: true      # install redis exporter on redis nodes?
redis_exporter_port: 9121         # redis exporter listen port, 9121 by default
redis_exporter_options: ''        # cli args and extra options for redis exporter
redis_safeguard: false            # prevent purging running redis instance?
redis_clean: true                 # purging existing redis during init?
redis_rmdata: true                # remove redis data when purging redis server?
redis_mode: standalone            # redis mode: standalone,cluster,sentinel
redis_conf: redis.conf            # redis config template path, except sentinel
redis_bind_address: '0.0.0.0'     # redis bind address, empty string will use host ip
redis_max_memory: 1GB             # max memory used by each redis instance
redis_mem_policy: allkeys-lru     # redis memory eviction policy
redis_password: ''                # redis password, empty string will disable password
redis_rdb_save: ['1200 1']        # redis rdb save directives, disable with empty list
redis_aof_enabled: false          # enable redis append only file?
redis_rename_commands: {}         # rename redis dangerous commands
redis_cluster_replicas: 1         # replica number for one master in redis cluster
redis_sentinel_monitor: []        # sentinel master list, works on sentinel cluster only

redis_cluster

name: redis_cluster, type: string, level: C

redis cluster name, required identity parameter.

no default value, you have to define it explicitly.

Comply with regexp [a-z][a-z0-9-]*, it is recommended to use the same name as the group name and start with redis-

redis_node

name: redis_node, type: int, level: I

redis node sequence number, unique integer among redis cluster is required

You have to explicitly define the node id for each redis node. integer start from 0 or 1.

redis_instances

name: redis_instances, type: dict, level: I

redis instances definition on this redis node

no default value, you have to define redis instances on each redis node using this parameter explicitly.

Here is an example for a native redis cluster definition

redis-test: # redis native cluster: 3m x 3s
  hosts:
    10.10.10.12: { redis_node: 1 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
    10.10.10.13: { redis_node: 2 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
  vars: { redis_cluster: redis-test ,redis_password: 'redis.test' ,redis_mode: cluster, redis_max_memory: 32MB }

The port number should be unique among the node, and the replica_of in value should be instance member of the same redis cluster.

redis_instances:
    6379: {}
    6380: { replica_of: '10.10.10.13 6379' }
    6381: { replica_of: '10.10.10.13 6379' }

redis_fs_main

name: redis_fs_main, type: path, level: C

redis main data mountpoint, /data by default

default values: /data, and /data/redis will be used as the redis data directory.

redis_exporter_enabled

name: redis_exporter_enabled, type: bool, level: C

install redis exporter on redis nodes?

default value is true, which will launch a redis_exporter on this redis_node

redis_exporter_port

name: redis_exporter_port, type: port, level: C

redis exporter listen port, 9121 by default

default values: 9121

redis_exporter_options

name: redis_exporter_options, type: string, level: C/I

cli args and extra options for redis exporter, will be added to /etc/default/redis_exporter.

default value is empty string

redis_safeguard

name: redis_safeguard, type: bool, level: G/C/A

prevent purging running redis instance?

default value is false, if set to true, and redis instance is running, init / remove playbook will abort immediately.

redis_clean

name: redis_clean, type: bool, level: G/C/A

purging existing redis during init?

default value is true, which will remove redis server during redis init or remove.

redis_rmdata

name: redis_rmdata, type: bool, level: G/C/A

remove redis data when purging redis server?

default value is true, which will remove redis rdb / aof along with redis instance.

redis_mode

name: redis_mode, type: enum, level: C

redis mode: standalone,cluster,sentinel

default values: standalone

• standalone: setup redis as standalone (master-slave) mode
• cluster: setup this redis cluster as a redis native cluster
• sentinel: setup redis as sentinel for standalone redis HA

redis_conf

name: redis_conf, type: string, level: C

redis config template path, except sentinel

default values: redis.conf, which is a template file in roles/redis/templates/redis.conf.

If you want to use your own redis config template, you can put it in templates/ directory and set this parameter to the template file name.

Note that redis sentinel are using a different template file, which is roles/redis/templates/redis-sentinel.conf

redis_bind_address

name: redis_bind_address, type: ip, level: C

redis bind address, empty string will use inventory hostname

default values: 0.0.0.0, which will bind to all available IPv4 address on this host

PLEASE bind to intranet IP only in production environment, i.e. set this value to ''

redis_max_memory

name: redis_max_memory, type: size, level: C/I

max memory used by each redis instance, default values: 1GB

redis_mem_policy

name: redis_mem_policy, type: enum, level: C

redis memory eviction policy

default values: allkeys-lru, check redis eviction policy for more details

• noeviction: New values aren’t saved when memory limit is reached. When a database uses replication, this applies to the primary database
• allkeys-lru: Keeps most recently used keys; removes least recently used (LRU) keys
• allkeys-lfu: Keeps frequently used keys; removes least frequently used (LFU) keys
• volatile-lru: Removes least recently used keys with the expire field set to true.
• volatile-lfu: Removes least frequently used keys with the expire field set to true.
• allkeys-random: Randomly removes keys to make space for the new data added.
• volatile-random: Randomly removes keys with expire field set to true.
• volatile-ttl: Removes keys with expire field set to true and the shortest remaining time-to-live (TTL) value.

redis_password

name: redis_password, type: password, level: C/N

redis password, empty string will disable password, which is the default behavior

Note that due to the implementation limitation of redis_exporter, you can only set one redis_password per node. This is usually not a problem, because pigsty does not allow deploying two different redis cluster on the same node.

PLEASE use a strong password in production environment

redis_rdb_save

name: redis_rdb_save, type: string[], level: C

redis rdb save directives, disable with empty list, check redis persist for details.

the default value is ["1200 1"]: dump the dataset to disk every 20 minutes if at least 1 key changed:

redis_aof_enabled

name: redis_aof_enabled, type: bool, level: C

enable redis append only file? default value is false.

redis_rename_commands

name: redis_rename_commands, type: dict, level: C

rename redis dangerous commands, which is a dict of k:v old: new

default values: {}, you can hide dangerous commands like FLUSHDB and FLUSHALL by setting this value, here’s an example:

{
  "keys": "op_keys",
  "flushdb": "op_flushdb",
  "flushall": "op_flushall",
  "config": "op_config"
}

redis_cluster_replicas

name: redis_cluster_replicas, type: int, level: C

replica number for one master/primary in redis cluster, default values: 1

redis_sentinel_monitor

name: redis_sentinel_monitor, type: master[], level: C

This can only be used when redis_mode is set to sentinel.

List of redis master to be monitored by this sentinel cluster. each master is defined as a dict with name, host, port, password, quorum keys.

redis_sentinel_monitor:  # primary list for redis sentinel, use cls as name, primary ip:port
  - { name: redis-src, host: 10.10.10.45, port: 6379 ,password: redis.src, quorum: 1 }
  - { name: redis-dst, host: 10.10.10.48, port: 6379 ,password: redis.dst, quorum: 1 }

The name and host are mandatory, port, password, quorum are optional, quorum is used to set the quorum for this master, usually large than half of the sentinel instances.

3 - Administration

Here are some common administration tasks for Redis. Check FAQ: Redis for more details.

Init Redis

Init Cluster/Node/Instance

# init all redis instances on group <cluster>
./redis.yml -l <cluster>      # init redis cluster

# init redis node
./redis.yml -l 10.10.10.10    # init redis node

# init one specific redis instance 10.10.10.11:6379
./redis.yml -l 10.10.10.11 -e redis_port=6379 -t redis

You can also use the wrapper script:

bin/redis-add redis-ms          # create redis cluster 'redis-ms'
bin/redis-add 10.10.10.10       # create redis node '10.10.10.10'
bin/redis-add 10.10.10.10 6379  # create redis instance '10.10.10.10:6379'

Remove Redis

Remove Cluster/Node/Instance

# Remove cluster `redis-test`
redis-rm.yml -l redis-test

# Remove cluster `redis-test`, and uninstall packages
redis-rm.yml -l redis-test -e redis_uninstall=true

# Remove all instance on redis node 10.10.10.13
redis-rm.yml -l 10.10.10.13

# Remove one specific instance 10.10.10.13:6379
redis-rm.yml -l 10.10.10.13 -e redis_port=6379

You can also use the wrapper script:

bin/redis-rm redis-ms          # remove redis cluster 'redis-ms'
bin/redis-rm 10.10.10.10       # remove redis node '10.10.10.10'
bin/redis-rm 10.10.10.10 6379  # remove redis instance '10.10.10.10:6379'

Reload Redis

You can partially run redis.yml tasks to re-configure redis.

./redis.yml -l <cluster> -t redis_config,redis_launch

Beware that redis cannot be reloaded online; you have to restart redis to make config effective.

Use Redis CLI

Access redis instance with redis-cli:

$ redis-cli -h 10.10.10.10 -p 6379 # <--- connect with host and port
10.10.10.10:6379> auth redis.ms    # <--- auth with password
OK
10.10.10.10:6379> set a 10         # <--- set a key
OK
10.10.10.10:6379> get a            # <--- get a key back
"10"

Redis also has a redis-benchmark which can be used for benchmark and generate load on redis server:

redis-benchmark -h 10.10.10.13 -p 6379

Replicate Redis

https://redis.io/commands/replicaof/

# promote a redis instance to primary
> REPLICAOF NO ONE
"OK"

# make a redis instance replica of another instance
> REPLICAOF 127.0.0.1 6799
"OK"

HA with Sentinel

You have to enable HA for redis standalone m-s cluster manually with your redis sentinel.

Take the 4-node sandbox as an example, a redis sentinel cluster redis-meta is used to manage the redis-ms standalone cluster.

# for each sentinel, add redis master to the sentinel with:
$ redis-cli -h 10.10.10.11 -p 26379 -a redis.meta
10.10.10.11:26379> SENTINEL MONITOR redis-ms 10.10.10.10 6379 1
10.10.10.11:26379> SENTINEL SET redis-ms auth-pass redis.ms      # if auth enabled, password has to be configured

If you wish to remove a redis master from sentinel, use SENTINEL REMOVE <name>.

You can configure multiple redis master on sentinel cluster with redis_sentinel_monitor.

redis_sentinel_monitor: # primary list for redis sentinel, use cls as name, primary ip:port
  - { name: redis-src, host: 10.10.10.45, port: 6379 ,password: redis.src, quorum: 1 }
  - { name: redis-dst, host: 10.10.10.48, port: 6379 ,password: redis.dst, quorum: 1 }

And refresh the master list on sentinel cluster with:

./redis.yml -l redis-meta -t redis-ha   # replace redis-meta if your sentinel cluster has different name

4 - Playbook

There are two playbooks for redis:

• redis.yml: create redis cluster / node / instance
• redis-rm.yml: remove redis cluster /docs/node /instance

redis.yml

The playbook redis.yml will init redis cluster/node/instance:

redis_node        : init redis node
  - redis_install : install redis & redis_exporter
  - redis_user    : create os user redis
  - redis_dir     : create redis redis fhs
redis_exporter    : config and launch redis_exporter
  - redis_exporter_config  : generate redis_exporter config
  - redis_exporter_launch  : launch redis_exporter
redis_instance    : config and launch redis cluster/node/instance
  - redis_check   : check redis instance existence
  - redis_clean   : purge existing redis instance
  - redis_config  : generate redis instance config
  - redis_launch  : launch redis instance
redis_register    : register redis to prometheus
redis_ha          : setup redis sentinel
redis_join        : join redis cluster

redis-rm.yml

The playbook redis-rm.yml will remove redis cluster/node/instance:

- register       : remove monitor target from prometheus
- redis_exporter : stop and disable redis_exporter
- redis          : stop and disable redis cluster/node/instance
- redis_data     : remove redis data (rdb, aof)
- redis_pkg      : uninstall redis & redis_exporter packages

5 - Monitor

Dashboard

There are three dashboards for REDIS module.

Redis Overview

Redis Overview: Overview of all Redis Instances

Redis Cluster

Redis Cluster : Overview of one single redis cluster

Redis Instance

Redis Instance : Overview of one single redis instance

Alert Rules

There are 6 predefined alert rules for Redis, defined in files/prometheus/rules/redis.yml.

#==============================================================#
#                         Error                                #
#==============================================================#
# redis down triggers a P0 alert
- alert: RedisDown
  expr: redis_up < 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: redis }
  annotations:
    summary: "CRIT RedisDown: {{ $labels.ins }} {{ $labels.instance }} {{ $value }}"
    description: |
      redis_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value }} == 0
      http://g.pigsty/d/redis-instance?from=now-5m&to=now&var-ins={{$labels.ins}}

# redis reject connection in last 5m
- alert: RedisRejectConn
  expr: redis:ins:conn_reject > 0
  labels: { level: 0, severity: CRIT, category: redis }
  annotations:
    summary: "CRIT RedisRejectConn: {{ $labels.ins }} {{ $labels.instance }} {{ $value }}"
    description: |
      redis:ins:conn_reject[cls={{ $labels.cls }}, ins={{ $labels.ins }}][5m] = {{ $value }} > 0
      http://g.pigsty/d/redis-instance?from=now-10m&to=now&viewPanel=88&fullscreen&var-ins={{ $labels.ins }}



#==============================================================#
#                         Latency                              #
#==============================================================#
# redis avg query response time > 160 µs
- alert: RedisRTHigh
  expr: redis:ins:rt > 0.00016
  for: 1m
  labels: { level: 1, severity: WARN, category: redis }
  annotations:
    summary: "WARN RedisRTHigh: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      pg:ins:query_rt[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 160µs
      http://g.pigsty/d/redis-instance?from=now-10m&to=now&viewPanel=97&fullscreen&var-ins={{ $labels.ins }}



#==============================================================#
#                        Saturation                            #
#==============================================================#
# redis cpu usage more than 70% for 1m
- alert: RedisCPUHigh
  expr: redis:ins:cpu_usage > 0.70
  for: 1m
  labels: { level: 1, severity: WARN, category: redis }
  annotations:
    summary: "WARN RedisCPUHigh: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      redis:ins:cpu_all[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 60%
      http://g.pigsty/d/redis-instance?from=now-10m&to=now&viewPanel=43&fullscreen&var-ins={{ $labels.ins }}

# redis mem usage more than 70% for 1m
- alert: RedisMemHigh
  expr: redis:ins:mem_usage > 0.70
  for: 1m
  labels: { level: 1, severity: WARN, category: redis }
  annotations:
    summary: "WARN RedisMemHigh: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      redis:ins:mem_usage[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 80%
      http://g.pigsty/d/redis-instance?from=now-10m&to=now&viewPanel=7&fullscreen&var-ins={{ $labels.ins }}

#==============================================================#
#                         Traffic                              #
#==============================================================#
# redis qps more than 32000 for 5m
- alert: RedisQPSHigh
  expr: redis:ins:qps > 32000
  for: 5m
  labels: { level: 2, severity: INFO, category: redis }
  annotations:
    summary: "INFO RedisQPSHigh: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      redis:ins:qps[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 16000
      http://g.pigsty/d/redis-instance?from=now-10m&to=now&viewPanel=96&fullscreen&var-ins={{ $labels.ins }}

6 - FAQ

ABORT due to existing redis instance

use redis_clean = true and redis_safeguard = false to force clean redis data

This happens when you run redis.yml to init a redis instance that is already running, and redis_clean is set to false.

If redis_clean is set to true (and the redis_safeguard is set to false, too), the redis.yml playbook will remove the existing redis instance and re-init it as a new one, which makes the redis.yml playbook fully idempotent.

ABORT due to redis_safeguard enabled

This happens when removing a redis instance with redis_safeguard set to true.

You can disable redis_safeguard to remove the Redis instance. This is what redis_safeguard is for.

How to add a single new redis instance on this node?

Use bin/redis-add <ip> <port> to deploy a new redis instance on node.

How to remove a single redis instance from the node?

bin/redis-rm <ip> <port> to remove a single redis instance from node