Open edX: Using LXC Containers
25 Sep 2014

This page is extremely rough. It’s just my notes with very little editing or checking. Be warned!

Instead of running Open edX directly on your VPS, consider running it within an LXC container.



What is LXC?

LXC is containers for the Linux kernel. What that means is that you can run multiple Linux userspace instances under one kernel. They’re isolated but can still share resources efficiently.

As LXC isn’t full machine virtualisation (like VMware, Parallels or VirtualBox), you can run it underneath another VM instance, such as one rented from a VPS host.

How do I set up the LXC host?

LXC features are built into most modern kernels. The tools are available back to Ubuntu 12.04 LTS (and probably further). I’ve had significantly better results under Ubuntu 14.04 than 12.04.

If you’re running Ubuntu, just install the lxc package.

How do I set up LXC guests?

If you want to set up (say) an Ubuntu guest, you would run:

lxc-create -n <container name> -t ubuntu

<container name> can be anything; I use ‘edxstaging’ and ‘edxprod’.

The files for the guest will appear under /var/lib/lxc/<container name>/rootfs

The first time you set up a particular template/release combination, packages will probably be downloaded. Be prepared to wait a little. Subsequent creations of the same template/release should be very fast.

It’s useful to be able to specify exactly which release of Ubuntu will be installed. For edX, you probably want the 12.04 AMD64 release, so run:

lxc-create -n <container name> -t ubuntu -- -r precise -a amd64

(I don’t know what happens if you try to run amd64 on a 32-bit host; it probably won’t work.)

Start the guest with:

lxc-start -d -n <container name>

The -d starts the container in the background. If you leave this off, the container will run in your terminal like a program and it’ll die when you close the terminal (unless you were already in a tmux session).

You can then get a console on the container with:

lxc-console -n <container name>

For the ubuntu template, you can log in at the console with username ubuntu and password ubuntu. You can then set up Open edX as you would a normal VM, per the deployment checklist.

Later on, you might like to stop the guest with:

lxc-stop -n <container name>

or delete the guest with:

lxc-destroy -n <container name>

Let the guest access the network

Depending on your release and configuration, the guest might not be able to access the network by default. Assuming that you’re running Ubuntu 12.04, you’ll need to set up an IP address on the guest and then use iptables to share the host’s network with the guest.

On the guest, as root, edit /etc/network/interfaces to look like:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address <guest IP address>

Change <guest IP address> to something unique for that guest. The default uses for the host, so use something like for the guest. Each guest must have its own IP, obviously.

On the guest, run /etc/init.d/networking restart to apply these changes. You should then be able to access the Internet through the guest.

I’m not sure why the default config uses a 10.x.x.x IP but only a /24 subnet; doesn’t hurt anything, though.

On the host, run:

/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
/sbin/iptables -I FORWARD 1 -i eth0 -o lxcbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -I FORWARD 1 -i lxcbr0 -o eth0 -j ACCEPT

Check that the updated iptables rules make sense with:

/sbin/iptables -L -v

To commit the rules for the next boot:

iptables-save > /etc/iptables.up-rules

Let the Internet see your edX instance

You probably want to make the edX instances accessible to the Internet. If you want all of them (or other websites) accessible on port 80 but with different hostnames, use nginx on the host per

You’ll end up with stacked nginx proxies (the host nginx talks to the guest nginx, which talks to the application server) but this isn’t a big deal.

On Ubuntu-like hosts, you can just stick a file in /etc/nginx/sites-enabled/edx that looks like:

server {

    access_log on;

    location / {
      proxy_redirect     default;

      # These fix the headers for the guest's server. Without these, you'll get broken redirects and less useful logging.
      proxy_set_header   X-Real-IP  $remote_addr;
      proxy_set_header   X-Forwarded-For $remote_addr;
      proxy_set_header   Host $host;
      #proxy_set_header   X-Forwarded-Proto $scheme;

At this point, the guest will log the IP address of the LXC host instead of the actual IP that requested the page. You can fix this by modifying the nginx config on the guest. For the LMS, edit /edx/app/nginx/sites-available/lms. Where it says:

location @proxy_to_lms_app {
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-For $remote_addr;

modify the X-Forwarded-For line like so:

# forward the correct IP from our upstream nginx
proxy_set_header X-Forwarded-For $http_X_Real_IP;

Starting LXC containers automatically

You probably want your edX LXC containers to start automatically when you boot the machine.

There’s some conflicting information about how to do this. The way I’m doing it (on Ubuntu 14.04 LTS) is:

  1. Edit /var/lib/lxc/<container name>/config
  2. Add = 1 somewhere
  3. Verify that it has taken effect with lxc-ls -f

Also check that /etc/default/lxc has LXC_AUTO="true".

lxc-* commands very slow, or lxc-console takes minutes to return

Your guest network probably isn’t configured correctly.

Unsorted below this line. Beware!

You might also want to open up the SSH ports (depending on how paranoid you are about security). You can use iptables again to forward ports on the host to the LXC guests:

# TODO: this doesn't work yet
/sbin/iptables -t nat -A PREROUTING -p tcp --dport <new ssh port> -j DNAT --to-destination <guest ip>:22
# e.g. /sbin/iptables -t nat -I PREROUTING 1 -p tcp --dport 2222 -j DNAT --to-destination
/sbin/iptables -I INPUT 1 -p tcp -m state --state NEW -m tcp --dport <new ssh port> -j ACCEPT

Then, you’d access SSH on the guest with a command line like:

ssh -p <port number> <hostname>

comments powered by Disqus