Varnish and MovableType for content management and application integration - Cold Hard Code, Purveyors of Fine Internet Properties

Varnish and MovableType for content management and application integration.

I encountered an interesting problem, without any clear solution in front of me. To save you from a lot of reading, I just deleted a 3 paragraph rant about the software I'm replacing.

Now, what I have is a 3 part system. First is the Catalyst application that runs in a secure environment and handles purchases and registrations. The second larger component is the content management, which is comprised of the legacy system and the new Movable Type based system.

MovableType was a fantastic choice, for the static publishing and easy user interface. While the asset management could be better, I can always fix that for our needs after the fact (which I will be, most likely).

The problem is getting 3 applications to live together harmonously when they're very disjointed. I don't really want to server the MovableType CGIs, so I can block any .cgi scripts at the frontend.

MovableType would be able to handle everything together, except for one caveat: A user logged in, with a valid membership, is presented with different content.

This, quite obviously, is a fantastic case for ESI. I think most people in the MovableType world would have their templates omit PHP and put the generated HTML files behind Apache and PHP.

Hell no.

There's 3 reasons for this:

I hate PHP
I don't want the bloat of dynamic pages when it isn't necessary
I already have an application that has all the snippets of templates, and uses a unified CSS structure

So, the only thing I'm missing is a controller that renders the atomic units of the logged in snippets.

This primarily consists of a few member benefit pages (showing redemption instructions) and menus, as is the typical case with ESI.

Now, this doesn't make sense in just words. So here's a picture that also doesn't make sense:

Using ESI in front of MovableType

In a nutshell, all that is really happening is a server-side include. It's similar to PHP, except you have a proxy-based load balanced backend cluster that can be accessed via an internal network (or on the same server, if you're a single-server setup). The ESI processing can be very simple to setup, and all your application needs to do is spit out the little snippet of a page.

The downside of this is your application processes one request for every ESI include, which can honestly get a bit costly but there are some better perks here over letting a browser do it. First, Varnish can use a keep-alive socket and stream the requests better, as well as providing caching on the individual ESI components and store the individual components inside of its main cache.

The Real Meat: Varnish Configuration

The VCL configuration for this is actually dead-simple. It utilizes the concept of 'restarting' the processing.

The first step is to get the Movable Type generated files served from Varnish. Rather than trying anything fancy, I'm just having it point to the standalone Apache server that hosts the MovableType CGI scripts and the generated files. This is on a separate server from the legacy content management system.

Just create the two backend segments (or you could use directors):


backend mt_cms {
    .host = "192.168.1.2";
    .port = "80";
}

backend legacy_cms {
    .host = "192.168.1.3";
    .port = "80";
}

Using restart; for a 404 fallback

Before even working with ESI, we need to fallthrough-on-404 behavior working. The key here is checking restarts, and then picking the backend based on that:


sub vcl_recv {
        if ( req.restarts == 0 ) {
            set req.backend = mt_cms;
        } else {
            set req.backend = legacy_cms;
        }
}

sub vcl_fetch {
    if ( obj.status == 404 ) {
        restart;
    }
}

You have to be careful of the max_restarts configuration variable here, which defaults to '4'. This means that during a legitimate 404 request that Varnish will attempt, on this configuration, 4 different requests to the backend servers that will all be 404. You can safely set it to '2', since no requests should fail, right? Right!

Now, when a request comes in it first attempts to connect to the MovableType server and on a 404 falls back to the legacy site. Awesome, almost there.

Adding ESI
Ready to be underwhelmed? Great, here:


sub vcl_fetch {
        # Restart on a 404, look at vcl_recv to see how this is handled.  The request just
        # traverses backends
        if ( obj.status == 404 ) {
             restart;
        }
        # We have something, and it's a GET request that is HTML based (html or xhtml)
        elseif (req.request == "GET" && obj.http.Content-Type ~ "html" ) {
            esi;  # Do ESI processing.  Yup, that's it.
            set obj.ttl = 24 h; # And cache the resulting page (not the ESI segments) for a day       
        }
 }

I warned you, overwhelming. Just adding esi; makes all the magic happen. Now, you just modify your MovableType templates (or whatever else you want to use) to have the various <esi:include src="/some/path"/> tags. Varnish 2.0 lets your ESIs even span across domains, which is neat (although somewhat scary, so use caution).

So, now you can have a fallthrough-on-404 setup that makes use of ESIs to populate your statically generated content with private information from your app server.

Enjoy!

Comments

View the entire comment thread.