Open edX: Deployment Checklist

25 Sep 2014

This page is extremely rough. It’s just my notes with very little editing or checking. Be warned!

Deploying a production service

You probably want to use the release branch of edx-platform. This is slightly more stable that the main branch; apparently it is what runs on edx.org.

Before updating your production server, it’s probably a good idea to run any updates against a staging server just to make sure things are sane. (A batch of unit testing wouldn’t hurt, either.)

Initial user and security setup

If you’re using a machine that is directly exposed to the Internet, the first thing to do is get basic account and network security in place. You can skip this if you’re in an LXC container on an isolated network.

1. Create a new user account, if the default install has you log in as root.
2. Fix /etc/sudoers so that the new account can sudo.
3. Copy your SSH public key to the new account’s ~/.ssh/authorized_keys.
4. Configure SSH: change the default port (Port <new port number>), only allow the new user account (AllowUser <username>) and disable password login (PasswordAuthentication no).
5. For some reason, service ssh restart or service ssh reload don’t actually do anything, so inside a tmux session I do /etc/init.d/ssh stop ; killall sshd ; /etc/init.d/ssh start. Obviously, this will kick you out of the SSH session, which is why we do it inside tmux. If you mistype this or have done something wrong in the config, you will be locked out. Be warned. (This is also why we do this config right at the start, so we can nuke from orbit if necessary.)
6. For paranoia’s sake, I set up the ufw firewall. Yes, it’s fiddly and annoying. But remember, this is a public-facing web service. Randoms will poke and prod it. You probably want (as root):

ufw allow <ssh port>/tcp  # permit SSH
ufw allow 80/tcp          # permit edX LMS
ufw allow 18010/tcp       # permit edX Studio
ufw default deny          # drop anything else

Again, setting these rules might lock you out of the system. Be careful.

Side note 1: I don’t believe that firewalls actually achieve much in reality, but it’s cheap insurance. Notably, if you forget to make a service internal-only and accidentally bind it to a public IP, the firewall will still protect you.

Side note 2: The edX codebase is huge and undoubtedly contains security problems. The firewall will not protect you against these. You will need to stay up-to-date with security alerts and patch your edX instance regularly.

Set up the host machine

There are a few tweaks that I like to make to all new Ubuntu machines.

• Install the following packages on all machines:

  • wget: downloads files through the command line. Needed for edX installation and not always installed by default.
  • aptitude: like apt-get, but better
  • tmux: detach and resume terminal sessions

• Install the following packages on anything that isn’t an LXC or OpenVZ guest:

  • swapspace: automatically scaling swap files
  • zram-config: automatically compresses memory (like swap)
  • You can see how your swap is allocated with cat /proc/swaps
  • Obviously, it’s best for performance if you don’t need swap at all, but running out of memory and invoking the OOM killer can be dangerous. You don’t control which process is killed (usually, it’s the largest one). This works for a while but eventually something important (like a database) will be killed and Bad Things will happen.
  • iotop: tells you which processes are hammering the disk

Setting up edX

Follow the instructions at [https://github.com/edx/configuration/wiki/edX-Ubuntu-12.04-64-bit-Installation]. If you’re in a rush, you can skip to ‘One step installation’, which I find works pretty well.

LXC: apparmor issues

While running vagrant.sh, you’ll get an error like:

stderr: apparmor_parser: Unable to replace "/edx/app/edxapp/venvs/edxapp-sandbox/bin/python".  Permission denied; attempted to load a profile while confined?

I spent a while trying to get this to work correctly but was not successful. It’s related to a Python sandbox, used for programming MOOCs (to ensure that students can’t run malicious code on the server). I’m not running a programming MOOC, so I disabled it.

Edit /var/tmp/configuration/playbooks/roles/edxapp/defaults/main.yml. Change:

EDXAPP_PYTHON_SANDBOX: true

to

EDXAPP_PYTHON_SANDBOX: false

Re-run the deployment script with

cd /var/tmp/configuration/playbooks && sudo ansible-playbook -c local ./edx_sandbox.yml -i "localhost,"

This is the same as the last line of vagrant.sh. Ideally, you would check that config change into a local branch of the edX Configuration repository.

The slightly nicer way to do this is to add the EDXAPP_PYTHON_SANDBOX line to your server-vars.yml, as described here.

LXC: rabbitmq issues

TASK: [rabbitmq | remove guest user]
stderr: Error: unable to connect to node rabbit@localhost: nodedown

I didn’t solve this completely, but a functional (if horrible) workaround is to edit /etc/hosts:

127.0.0.1 <hostname>
127.0.0.1 localhost

Installing your theme

Setting up user accounts

Setting configuration variables

In configuration repo, modify /playbooks/roles/edxapp/defaults/main.yml

###

Tasks to complete before live deployment

• Set up SSH access with public keys (preferably not on the default port 22)

• Disable the default accounts:

  https://github.com/edx/edx-platform/wiki/Frequently-Asked-Questions User: honor Password: edx User: audit Password: edx User: verified Password: edx User: staff Password:edx

• Verify that only your LMS, CMS and SSH ports are visible through the firewall. There are a lot of TCP-enabled services running; while they are probably configured to allow connections to localhost only, why take the chance? ** Run netstat -al to check

• Review the settings in /edx/app/edxapp/.json, especially things like cms.env.json which define contact details and titles for your instance. ** Or maybe you’re not supposed to touch those – https://groups.google.com/d/msg/edx-code/VjVFT4-Etjw/UrpzDbpazo0J says that they get overwritten during ansible update

• Add Google Analytics API key

• Set up your DNS to point to your instance.

  • Talk about how to use different DNS names to give Studio vs. LMS instead of different port numbers

• Think about backups and disaster recovery

• Set up authentication (Shibboleth, LDAP)

• Important URLs

• Configure the instance

• Adding users

• Creating a course

• Setting start and end dates

• Uploading SCORM zip files