Redis is a powerful in-memory data store thats blazing fast. It's performant, scalable, supports a wide range of data structures, has built in caching mechanisms, and is simple to set up for Cycle users.
This post will take you from deploying your first Redis instance on Cycle, through deploying highly available, stateful Redis instances that are monitored by Redis Sentinel.
The companion repo for this article with configuration files and settings can be found here.
Redis is a mature product, and with that maturity comes choice, flexibility, and configurations that could feel overwhelming at first glance. Let's take a look at some of these things.
There are two main ways to achieve high availability with Redis.
Redis has a built-in functionality called Redis Cluster that enables you to group Redis nodes together to store and retrieve data. The cluster as a whole can handle massive amounts of data and heavy traffic since each node in the cluster only stores a portion of the data. In addition, Redis Cluster offers automated failover so that in the event that one node in the cluster fails, the remaining nodes can take over and keep the cluster operational.
There's also Redis Sentinel. Sentinel is a program that watches over a collection of Redis instances and runs as a separate service. It has lots of configurable capabilities, including monitoring, notifications, automatic failover, and the capacity to run several Redis instances in a cluster. It merely keeps track of and manages the Redis instances rather than actually storing the data. Both a single Redis instance and a Redis cluster can employ Redis Sentinel.
When dealing with Redis configurations, Sentinel configurations are somewhat different from other "Redis Servers". This becomes apparent in two places:
The root command for starting a Redis service is:
redis-server
.
And if you look at the official Docker image repo, you can see that the default command used is just that.
Extending this startup command becomes essential once you start modifying the default configuration, and the redis-server command will extend to become:
redis-server /path/to/config, telling Redis "when you start this server process, use the following config file".
If you'd like to see the difference between the default Redis configuration and the default Sentinel configuration files you can download the example Sentinel config here.
The way to signal to Redis server that you'd like to start the service in Sentinel mode is by appending --sentinel to the redis-server command.
Ex: redis-server /path/to/sentinel.conf --sentinel
There's a huge amount of customization you can wade through in the Redis config file. For today, we'll be focused on a very small portion of the file that will provide us a great getting started point with our HA Redis deployment.
In the redis config I've updated the following settings: bind, protected-mode, appendonly.
Bind all is set to all IPv4 and all IPv6.
bind 0.0.0.0 ::*
For this deployment we'll drop protected-mode because we want to be able to connect between clients without setting up authentication.
protected-mode no
With appendonly mode, each time Redis gets a change to the data, it appends to the .aof file. Restarting Redis with this setting in place will cause Redis to restore data from whats stored in the AOF backup.
appendonly yes
The configuration for the secondary instance is the same as the main instance but there is one major difference: replicaof.
The replicaof setting tells the redis instance that it is a replica of an existing redis instance (in this case the main).
replicaof redis-main 6379
The sentinel instances have a different base config (noted above). The first line you'll want to look for is sentinel monitor. This line sets the main instance to monitor, the port, and the quorum. You can change this line to:
sentinel monitor mymaster redis-one 6379 2
The other setting changed in this config file is mostly for demonstration purposes: sentinel down-after-milliseconds will tell the Redis sentinel how long to wait after failing to communicate with the Redis main node before holding a new election. In this case we'll set it to 3 seconds:
sentinel down-after-milliseconds 3000
The companion repo for this post has a cycle.json file in it that describes all the containers we want to set up.
The first container called "redis-main" is the main Redis instance. If you look in the directory "main" you'll see a config file and a Dockerfile. These files, together with the "docker-file" image source type in the stack will let Cycle know how to build the image for this container.
If you inspect the Dockerfile, you'll see that the startup command will be redis-server /var/lib/config/redis.conf, which is a file located in a volume with permissions 0660. The reason we want to have that directory as a volume is due to the Redis sentinel service. The Redis sentinel service will actually re-write the config files based on what happens.
For example: if the main instance were to go offline for a period long enough for a vote to be held and quorum to be met that it is in fact no longer reachable, sentinel would elect a new leader (main). That new leader would need to have a config file that no longer says its a "replicaof", so sentinel rewrites the config on the fly and manages the change in service. When the old main comes back online, it is now a secondary and must become a "replicaof" the other instance. Again here sentinel rewrites the config to facilitate.
The second container is called "redis-secondary" and is all but the same as the main instance, but the config carries the "replicaof" setting so that on the containers first start it becomes a replica.
The other three containers in the stack file represent the three sentinel instances. They each use the same image and have the same volume permissions for their config files, because these instances too will rewrite their own config data after they're started the first time.
Deployment will take place over 3 simple steps:
To create the stack, head over to the stacks page in portal and:
To generate a stack build simply click the "Create Build" button.
Once the stack has generated a build, head over to the Environments page and start the creation of a new environment. When selecting "Default" vs "From A Stack" select "From A Stack" and then the stack/build that we just created. Create the environment and then on the newly created environment dashboard, use the global container start button to start all the containers.
You may notice that the containers start at slightly different times (just a few seconds). If you go back to the cycle.json file you'll notice that some of the instances have settings for startup: {delay: time}. This tells Cycle that when the start command is given to this container, wait a certain amount of time (in seconds) before starting. I used this setting to make sure that the main instance would start before the secondary instance and that both the main and secondary instance would be online before the sentinel instances.
To run a failover test, pull up the instance console for any of the sentinel instances. Next, open up another tab with the containers view and select the redis-main container. Stop just the redis-main container and then watch the console of the sentinel instance you'd previously opened. What you'll see is after a few seconds of the redis-main instance being stopped an election will be held and a new main instance will be elected.
Now start the redis-main container again and use the two-way console to log into it. After running redis-cli, run INFO replication and you'll see that this instance is now acting as the secondary instance. The sentinel has properly reconfigured your servers to handle a failover event.
💡 Interested in trying the Cycle platform? Create your account today! Want to drop in and have a chat with the Cycle team? We'd love to have you join our public Cycle Slack community!