An Afternoon of Code Golf (in Lua) to Achieve 4x performance in Redis
Amplitude collects tens of thousands of user events per second. Processing all these events involves a multistep processing pipeline that has dozens of steps and dependencies. We use a variety of tools (Datadog, Pagerduty, Pingdom, New Relic, and some of our own) to monitor the health of our system. Despite all that, pinpointing the exact root cause of issues can still be a little tricky. Ideally all the systems we work with are designed from the ground up with scalability in mind, and have clear metrics that allow us to anticipate scaling requirements. Most of our systems at Amplitude are like that, but it’s a given that little features get added here and there that later grow into groups of features that interact and scale in not-so-clear ways. This is where some quick debugging skills come in handy. I’d like to walk through the investigation and resolution of one such incident involving Redis and our processing pipeline, where I spent a couple hours investigating and changing a small bit of code to avoid a much longer and tedious process of scaling one of our Redis clusters.
We use Redis a lot at Amplitude, mostly as a cache but sometimes for less transient data as well. One of these less transient use cases is our “redis-metrics” cluster, which stores event counts for applications on our platform bucketed at various aggregations. One day, our ingestion pipeline seemed to be struggling during peak traffic. From our processing pipeline’s metrics in datadog, I could see a processing step that interacted exclusively with redis-metrics was unusually slow, making redis-metrics the likely culprit.
Most of our Redis clusters are behind Twemproxy so for the clusters serving as caches, we could simply add nodes behind Twemproxy to accommodate more traffic. However, because this particular cluster contained non-transient data, scaling up would have been more involved because we’d need to preserve the old data in the new scaled up cluster. This would involve cutting the Twem nodes over to a new scaled-up cluster, then copying all the data from the old cluster to the new cluster. Not an impossible task, but one that would require focus and planning and maybe a day or two to execute just to make sure we did it right. So we figured it’d be worth investigating other ways to reduce load on this cluster to avoid the hassle.
Looking at this cluster’s metrics in datadog didn’t really reveal what was going on; CPU usage on the Redis and Twem nodes seemed fine (this can sometimes be misleading because Redis and Twem are single threaded, but things seemed fine even factoring that in). This could imply the issue might be network related (i.e. amount of data to/from Redis). I figured I’d try looking at the code that interacts with this cluster to find ways to reduce load. Basically, we store event counts on an app by app basis in Redis’s hash structure:
“<aggr_size>:<aggr_bucket>” -> { “<app_id_1>” -> count … “<app_id_n>” -> count}
We batch up these counts over a period of time before emitting the batches using Redis’s pipelined request. There wasn’t anything obviously wrong with this picture.
Next I went to one of the redis-metrics machines. I had recently learned about the redis-cli monitor command which dumps all the commands that Redis is executing. I figured maybe I could see something unusual in the pattern of the commands. So I ran:
$ redis-cli monitor > commands$ cat commands | awk -F” ” ‘{print $4}’ | sort | uniq -c | sort -n3987 “SMEMBERS”5063 “HGETALL”17619 “EXPIREAT”17626 “EVAL”21207 “INCRBY”21213 “EXPIRE”43087 “GET”60929 “HINCRBY”
That’s a lot of HINCRBY commands, but not totally surprising because that’s how we update the event counts. I wanted to see if all those HINCRBY commands were what I expected so I ran the following command to help me find any unexpected HINCRBY commands or unexpected usage patterns.
cat some_commands | grep HINCRBY | awk ‘{print $5}’ | sort | uniq -c | sort -n2015 “id:second_part:20170221225752”2508 “second_part:20170221225756”2799 “second_app:20170221225744”2843 “second_app:20170221225746”2937 “second_part:20170221225754”2952 “second_app:20170221225750”2967 “second_app:20170221225752”3031 “second_part:20170221225749”
What stands out to me here is that there are a lot of updates to the same top level hash keys. This made me suspect a bug at first, but it was actually correct because we store counts bucketed by time so all the counts end up falling under the same buckets. Still this seemed excessive so I wanted to find a way to collapse all these updates into a single command. In my experience so far, it seemed that Redis had a command for everything; HSET sets a field value pair under a key, and HMSET sets many field value pairs under a key so surely there is the equivalent HMINCRBY for HINCRBY, but after looking around I found this closed issue which made it clear I’d have to make my own solution.
I decided I’d try writing a Lua script of my own for HMINCRBY. Redis provides two options for evaluating custom scripts:
- Load a script on the Redis node and identify it with the SHA hash of the script
- Send the entire script to Redis each time you want to run the command.
I figured I’d go with the first option so I wrote up a verbose script with error handling and everything. However, further consideration of the deployment process revealed this approach had some complications: it would tie this code into the deployment code that managed the Redis cluster itself which is just not something we currently did. Also, it would be difficult to manage in different environments; our production, staging and shared integration environments are managed with salt, but we don’t use salt for our local development environments so an ad hoc solution would be needed, further complicating this change. I decided to try to make the second option (sending the script with the EVAL command) more viable by playing some code golf. I would now be sending the script instead of the SHA1 hash of the script so 40 characters was the target length I had in mind. Eventually I got the script down to 77 characters:
for i=0,#ARGV/2–1 do redis.call(‘HINCRBY’,KEYS[1],ARGV[i*2+1],ARGV[i*2+2])endThis length is sufficient because we’re really replacing a sequence of commands that look like:
“HINCRBY” “second_app:20170221225744” <app_id_1> <count_1>“HINCRBY” “second_app:20170221225744” <app_id_2> <count_2>…“HINCRBY” “second_app:20170221225744” <app_id_N> <count_N>
With:
EVAL <script> “second_app:20170221225744” <app_id_1> <count_1> … <app_id_N> <count_N>Some rough calculation shows that the amount of data sent with the old method scales 4x the rate of the new solution.
After applying the change I monitored our datadog graphs for the Redis cluster and saw a 22% reduction in number of commands and 16% reduction in data sent to the cluster. Over the following days we applied the same change to other sections of code that interacted with the same cluster and saw similar reductions in network and number of commands.
One last thing, there’s another (maybe obvious) improvement to be made before scaling up the cluster, if you know what it is, we’d love to hear from you.
