Purging local Memcached

Memcached is a ubiquitous caching solution, most commonly used to speed things up in web application backends. You deploy it as a separate binary and have your application servers talk to it before querying a database, calling a third party API over HTTP, or performing some other time-consuming I/O operation. Since it stores data in memory, as its name obviously suggests, it can be orders of magnitude faster than anything that may have to hit a spinning disk (like a database) or an unreliable, external network.

From the point of view of a developer, using Memcached is pretty simple. There exist numerous libraries that wrap its protocol in a neat, language-appropriate API. It does put another requirement on the development environment, of course: you need to have a working memcached deamon in the background if you want the local server to hit code paths where it retrieves data from memcache¹. Thankfully, it’s an extremely popular piece of software, present in basically all package repositories, so having it up and running is just one apt-get install or brew install away.

How to flush

It’s a little dated and finicky piece of software, too. Once you have its local instance used for a while, there comes a time when you’d like to purge all its contents and have your server(s) fill it up again. Rather than restarting the daemon (which is system-specific procedure that may require root privileges), you’d like to just issue the flush_all command to it.

This should be easy. Unlike Redis, however, memcached doesn’t come with a dedicated CLI client. In theory, it doesn’t have to, because you can talk to it over just telnet:

$ telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
flush_all
OK
^]

telnet> quit
Connection closed.

But going through this rigmarole every time we want to flush memcache gets a bit tedious after a while. An obvious attempt at automating it will, however, fail:

$ echo "flush_all\n" | telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.

Turns out the you cannot just bombard memcached with flush_all (or any other command) while the connection is being established. You can of course try adding a sleep to allow it some time, but this is inherently unreliable, especially when connecting to remote servers.

The most successful approach I’ve managed to find is to automate not the raw data exchange, but the interactive telnet session itself. By that I mean invoking a telnet command, observing its output and reacting to it — just like a human would do, but in a scripted, automated way.

Nobody expects the Telnet instrumentation

There is a dedicated Unix program for scripting just this kind of CLI interactions. It’s decidedly more obscure than mere echo or pipes, but should nonetheless be available for any modern (or ancient) Unix system, from Linuxes to OSX. Its name is expect and — if installed² — the usual place you can find it is /usr/bin or /usr/local/bin.

expect accepts commands and scripts written in Tcl — a scripting language whose syntax looks a bit like a mix of PHP and Objective-C. It’s a fully featured language with all the usual programming constructs you’d normally want but expect enhances it further with instructions dedicated to spawning external processes, reading their output, and sending input in response.

In fact, the main commands of the expect program are spawn, expect, and send. Mashing them together, we can easily nail the crucial part of our memcache-purging script:

#!/usr/bin/expect

spawn telnet localhost 11211;
expect "Connected to localhost";
send "flush_all\n";

This performs the actual flush but the telnet session/process is kept open afterwards. Once the expect program ends (after finishing our script), that process may get orphaned and hog system resources. To prevent that, we should behave like a good Unix citizen and wait for our child processes to terminate:

wait;

But obviously, this will never happen, because we’ve never instructed the telnet process to end!

Escape artist

Your first instinct may be to send ^C (SIGINT) or ^D (end-of-file) to the telnet process but this won’t work either. Everything we type inside an active Telnet session is sent to the remote server, and that includes those two key chords.

Authors of the telnet program were of course aware of this problem. As a countermeasure, they introduced the concept of an escape character that allows the user to control the Telnet connection itself — including, but not limited to, its termination. When encountered in the input stream, the escape character causes telnet to temporally suspend its normal operation, “escaping” from the client-server session to a simple command shell of the telnet program itself. (This is indicated by the telnet> prompt).

The default escape character is ^], which can be easily typed on a keyboard using the key combination Ctrl+]. The expect program, however, only simulates typing real characters. Coupled with some syntactical quirks of the Tcl language, this makes the usage of ^] as an escape character rather cumbersome, to say the least.

Fortunately, this default can be changed rather easily with an -e flag to the telnet application. After setting it to something more common but still outside of the memcache protocol — such as the exclamation mark — we are now able to send it without an issue:

#!/usr/bin/expect

set escapechar !;

spawn telnet -e $escapechar localhost 11211;
expect "Connected to localhost";
send "flush_all\n";

send $escapechar;
expect "telnet>";
send "quit\n";
wait;

The script will also terminate despite waiting for the telnet process to finish, which means everything has been shut down gracefully.

Production-ready

As a final touch, it’d be nice to make our solution work with any memcache server and not just localhost. You can see it done in this gist: the script accepts two optional arguments (host & port) and uses it when spawning the telnet process.