Open edX: Using LXC Containers

This page is extremely rough. It’s just my notes with very little editing or checking. Be warned!

Instead of running Open edX directly on your VPS, consider running it within an LXC container.

Pros:

Run production and staging systems on the same hardware
- It’s usually cheaper to rent one big VPS instead of many small ones
Run other services on the same hardware. Open edX requires specific versions of everything which will conflict with your other services.
Test security patches or custom code on a production-like system

Cons:

Some increase in complexity
You have more machines to deploy security updates onto
- You could automatically deploy updates
- Better yet, you could use your staging container to test the updates before you deploy them onto the production container
A trivial decrease in performance
A very small increase in disk space usage
- The LXC host needs its own copy of the Ubuntu base packages
You need enough RAM to run both systems simultaneously
- A VM would use double the RAM, as you’d make a fixed allocation. With LXC, you’re just running two copies of the applications under the same kernel, so you’ll use the space much more efficiently.
- Therefore, swap is shared. If you have applications which aren’t use often – like for a staging site – they might get swapped out and thus not impact your production site too much.
Open edX uses AppArmor for a number of security features, and it’s not clear to me that that works within an LXC container
- LXC is probably not a good idea if you’re running untrusted code (e.g. for a programming MOOC).
- LXC containers, right now, do not provide strong protection of the host against malicious clients (such as your students).

LXC is containers for the Linux kernel. What that means is that you can run multiple Linux userspace instances under one kernel. They’re isolated but can still share resources efficiently.

As LXC isn’t full machine virtualisation (like VMware, Parallels or VirtualBox), you can run it underneath another VM instance, such as one rented from a VPS host.

LXC features are built into most modern kernels. The tools are available back to Ubuntu 12.04 LTS (and probably further). I’ve had significantly better results under Ubuntu 14.04 than 12.04.

If you’re running Ubuntu, just install the lxc package.

If you want to set up (say) an Ubuntu guest, you would run:

lxc-create -n <container name> -t ubuntu

<container name> can be anything; I use ‘edxstaging’ and ‘edxprod’.

The files for the guest will appear under /var/lib/lxc/<container name>/rootfs

The first time you set up a particular template/release combination, packages will probably be downloaded. Be prepared to wait a little. Subsequent creations of the same template/release should be very fast.

It’s useful to be able to specify exactly which release of Ubuntu will be installed. For edX, you probably want the 12.04 AMD64 release, so run:

lxc-create -n <container name> -t ubuntu -- -r precise -a amd64

(I don’t know what happens if you try to run amd64 on a 32-bit host; it probably won’t work.)

Start the guest with:

lxc-start -d -n <container name>

The -d starts the container in the background. If you leave this off, the container will run in your terminal like a program and it’ll die when you close the terminal (unless you were already in a tmux session).

You can then get a console on the container with:

lxc-console -n <container name>

For the ubuntu template, you can log in at the console with username ubuntu and password ubuntu. You can then set up Open edX as you would a normal VM, per the deployment checklist.

Later on, you might like to stop the guest with:

lxc-stop -n <container name>

or delete the guest with:

lxc-destroy -n <container name>

Depending on your release and configuration, the guest might not be able to access the network by default. Assuming that you’re running Ubuntu 12.04, you’ll need to set up an IP address on the guest and then use iptables to share the host’s network with the guest.

On the guest, as root, edit /etc/network/interfaces to look like:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address <guest IP address>
    netmask 255.255.255.0
    network 10.0.3.0
    gateway 10.0.3.1
    dns-nameservers 8.8.8.8 8.8.4.4

Change <guest IP address> to something unique for that guest. The default uses 10.0.3.1 for the host, so use something like 10.0.3.100 for the guest. Each guest must have its own IP, obviously.

On the guest, run /etc/init.d/networking restart to apply these changes. You should then be able to access the Internet through the guest.

I’m not sure why the default config uses a 10.x.x.x IP but only a /24 subnet; doesn’t hurt anything, though.

On the host, run:

/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
/sbin/iptables -I FORWARD 1 -i eth0 -o lxcbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -I FORWARD 1 -i lxcbr0 -o eth0 -j ACCEPT

Check that the updated iptables rules make sense with:

/sbin/iptables -L -v

To commit the rules for the next boot:

iptables-save > /etc/iptables.up-rules

You probably want to make the edX instances accessible to the Internet. If you want all of them (or other websites) accessible on port 80 but with different hostnames, use nginx on the host per http://nginx.org/en/docs/beginners_guide.html#proxy.

You’ll end up with stacked nginx proxies (the host nginx talks to the guest nginx, which talks to the application server) but this isn’t a big deal.

On Ubuntu-like hosts, you can just stick a file in /etc/nginx/sites-enabled/edx that looks like:

server {
    server_name edx.example.com;

    access_log on;

    location / {
      proxy_pass         http://10.0.3.101:80;
      proxy_redirect     default;

      # These fix the headers for the guest's server. Without these, you'll get broken redirects and less useful logging.
      proxy_set_header   X-Real-IP  $remote_addr;
      proxy_set_header   X-Forwarded-For $remote_addr;
      proxy_set_header   Host $host;
      #proxy_set_header   X-Forwarded-Proto $scheme;
    }
}

At this point, the guest will log the IP address of the LXC host instead of the actual IP that requested the page. You can fix this by modifying the nginx config on the guest. For the LMS, edit /edx/app/nginx/sites-available/lms. Where it says:

location @proxy_to_lms_app {
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-For $remote_addr;

modify the X-Forwarded-For line like so:

# forward the correct IP from our upstream nginx
proxy_set_header X-Forwarded-For $http_X_Real_IP;

You probably want your edX LXC containers to start automatically when you boot the machine.

There’s some conflicting information about how to do this. The way I’m doing it (on Ubuntu 14.04 LTS) is:

Edit /var/lib/lxc/<container name>/config
Add lxc.start.auto = 1 somewhere
Verify that it has taken effect with lxc-ls -f

Also check that /etc/default/lxc has LXC_AUTO="true".

`lxc-*` commands very slow, or `lxc-console` takes minutes to return

Your guest network probably isn’t configured correctly.

Unsorted below this line. Beware!

You might also want to open up the SSH ports (depending on how paranoid you are about security). You can use iptables again to forward ports on the host to the LXC guests:

# TODO: this doesn't work yet
/sbin/iptables -t nat -A PREROUTING -p tcp --dport <new ssh port> -j DNAT --to-destination <guest ip>:22
# e.g. /sbin/iptables -t nat -I PREROUTING 1 -p tcp --dport 2222 -j DNAT --to-destination 10.0.3.102:22
/sbin/iptables -I INPUT 1 -p tcp -m state --state NEW -m tcp --dport <new ssh port> -j ACCEPT

Then, you’d access SSH on the guest with a command line like:

ssh -p <port number> <hostname>

Ian Howson