Open edX devstack on KVM (Ficus edition)

The Open edX devstack expects to run on Vagrant with VirtualBox. With a few tweaks, you can use Vagrant with KVM instead.

The instructions here are specific to the Ficus release. If you need Eucalyptus, look here.

Rationale

At Scampersand we SSH into a shared development server that provides a stable environment and regular backups without relying on individual hardware or configuration. That server runs Fedora and hosts docker containers and KVM virtual machines.

KVM and VirtualBox are mutually exclusive on Linux because they each supply a kernel module that handles the CPU virtualization instructions, so only one can be loaded at a time. Since we have an existing infrastructure based on KVM, we’d rather run the devstack on it than need to bring up a separate host for Virtualbox.

Steps

If you’d like to run the Open edX devstack on KVM, these are the steps you’ll need. I’ll explain these in detail below.

Install Vagrant 1.8.7 or later, sorta

The Open edX Vagrantfile explicitly checks for Vagrant 1.8.7 or later. This is probably motivated by the bug that affected old Ubuntu boxes (even though it doesn’t affect Ficus), and might also be related to the bug with NFS exports.

You could get the latest version (1.9.1 at the time of writing) from vagrantup.com. However on Fedora this eventually leads to conflicts between Vagrant’s embedded ruby and the system-installed ruby.

Fedora 25 supplies Vagrant 1.8.5, but patched for the NFS bug. I recommend using the distro-supplied package, and planning to patch the Vagrantfile to allow it. (My modified Vagrantfile below does just that.)

$ sudo dnf install vagrant
$ vagrant -v
Vagrant 1.8.5
$ rpm -q --changelog vagrant | head -n2
* Tue Nov 15 2016 Vít Ondruch - 1.8.5-2
- Fix nfs_cleanup security race and permissions (rhbz#1395040).

Install the vagrant-libvirt plugin

“Wait, libvirt? I thought we were using KVM.”

Right, so let’s step back and review how this works. Vagrant is a tool for controlling a virtual machine. The configuration for the VM is described in Vagrantfile, and Vagrant depends on a virtualization provider to do the heavy lifting. Most of the time the provider is Virtualbox, but Vagrant can also use VMware, Hyper-V, Docker and others.

KVM is a kernel-level virtualization technology that’s managed by libvirtd in Linux distributions. In fact there used to be a KVM-specific provider plugin for Vagrant, but that’s been deprecated in favor of the libvirt provider plugin.

So in order to use Vagrant with KVM, you need the vagrant-libvirt plugin. You can install it like this:

$ sudo dnf install vagrant-libvirt

Enable user for vagrant with libvirt

The libvirt daemon will need to know that you’re authorized to manage virtual machines. The easiest way to do this is to add yourself to the libvirt group:

$ sudo usermod -aG libvirt $USER

There’s no need to log out and back in, since libvirtd checks group membership directly rather than via process inheritance.

It’s a good idea to make sure that virsh works properly before proceeding:

$ virsh --connect=qemu:///system sysinfo

If that command produces some XML output, you should be good to go. If it doesn’t, then you might need to investigate since vagrant-libvirt isn’t likely to work either.

Mutate the devstack box

The box image provided by edX is specific to to VirtualBox. Use the vagrant-mutate plugin to convert it for libvirt/KVM. This isn’t available from Fedora, so you’ll have to install it locally:

$ vagrant plugin install vagrant-mutate
Installing the 'vagrant-mutate' plugin. This can take a few minutes...
Installed the plugin 'vagrant-mutate (1.2.0)'!

$ vagrant box add --name ficus-devstack-2017-02-07 --provider virtualbox \
    http://files.edx.org/vagrant-images/ficus-devstack-2017-02-07.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'ficus-devstack-2017-02-07' (v0) for provider: virtualbox
    box: Downloading: http://files.edx.org/vagrant-images/ficus-devstack-2017-02-07.box
==> box: Successfully added box 'ficus-devstack-2017-02-07' (v0) for 'virtualbox'!

$ vagrant mutate ficus-devstack-2017-02-07 libvirt
Converting ficus-devstack-2017-02-07 from virtualbox to libvirt.
    (100.00/100%)
The box ficus-devstack-2017-02-07 (libvirt) is now ready to use.

You can inspect what this does on the filesystem in your .vagrant.d directory.

$ tree ~/.vagrant.d/boxes
/home/aron/.vagrant.d/boxes
└── ficus-devstack-2017-02-07
    └── 0
        ├── libvirt
        │   ├── Vagrantfile
        │   ├── box.img
        │   └── metadata.json
        └── virtualbox
            ├── Vagrantfile
            ├── box-disk1.vmdk
            ├── box.ovf
            ├── metadata.json
            └── vagrant_private_key

Notice that the vagrant_private_key wasn’t included in the migration. While it would be possible to copy this manually, the easier workaround is to set config.ssh.password, and this is the approach already taken in the custom Vagrantfile you’ll download next.

Repair the network configuration

Now you have ficus-devstack-2017-02-07 mutated to run under libvirt, but if you try to boot it up, it will hang at this line:

==> default: Waiting for domain to get an IP address...

The problem is that the network interface is named differently when running under libvirt, so the networking configuration in the guest doesn’t apply properly.

There’s a pretty easy fix for this, using a program called guestfish. This nifty tool lets you crack open the virtual machine image to make small edits. The two small edits you need to make are: (1) modify grub’s kernel parameters so the network interface is predictably named, and (2) add the network configuration.

$ sudo dnf install libguestfs-tools-c
$ guestfish -a ~/.vagrant.d/boxes/ficus-devstack-2017-02-07/0/libvirt/box.img

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

><fs> run
><fs> mount /dev/vagrant-vg/root /
><fs> mount /dev/sda1 /boot
><fs> edit /etc/default/grub

This will drop you into an editor. Change the value of GRUB_CMDLINE_LINUX_DEFAULT so it reads like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet net.ifnames=0 biosdevname=0"

Save and exit, which puts you back at the guestfish prompt. First, verify that your change was written properly:

><fs> grep quiet /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet net.ifnames=0 biosdevname=0"

Next, run update-grub in the context of the guest filesystem. This will rewrite the grub configuration at /boot/grub/grub.cfg with the new kernel parameters.

><fs> command /usr/sbin/update-grub
><fs> grep quiet /boot/grub/grub.cfg
linux /vmlinuz-4.4.0-31-generic root=/dev/mapper/vagrant--vg-root ro  quiet net.ifnames=0 biosdevname=0
linux /vmlinuz-4.4.0-31-generic root=/dev/mapper/vagrant--vg-root ro  quiet net.ifnames=0 biosdevname=0

Finally, add the network configuration for eth0:

><fs> write /etc/network/interfaces.d/eth0 "auto eth0\niface eth0 inet dhcp\n"
><fs> quit

If you tried to start Vagrant without making these changes first, you’ll need to remove the box image from libvirt storage. Next time you run Vagrant it will re-import from your modified box.

virsh vol-delete ficus-devstack-2017-02-07_vagrant_box_image_0.img --pool default

Download the modified devstack Vagrantfile

The edX-provided Vagrantfile assumes Virtualbox, so I’ve provided a modified Vagrantfile that supports libvirt/KVM. You can inspect the differences here.

You should download this into a new directory where the sources will also eventually be checked out.

$ mkdir devstack
$ cd devstack
$ curl -OL https://raw.githubusercontent.com/scampersand/configuration/scampersand/ficus.libvirt-kvm/vagrant/release/devstack/Vagrantfile

Fire it up!

At this point you can start devstack. Since the upstream Vagrantfile still defaults to Eucalyptus, you’ll need to override it by setting OPENEDX_RELEASE in the environment.

$ export OPENEDX_RELEASE=open-release/ficus

If you will have more than one user running on the same host, then you should start vagrant with VAGRANT_NO_PORTS=1 to avoid listening on a bunch of localhost ports that will collide between users. Instead you can access the VM by its own IP address as shown below.

$ VAGRANT_NO_PORTS=1 vagrant up --provider=libvirt
Bringing machine 'default' up with 'libvirt' provider...
==> default: Creating image (snapshot of base box volume).
==> default: Creating domain with the following settings...
==> default:  -- Name:              devstack_default_1478018111_5f69c2128844670d27e2
==> default:  -- Domain type:       kvm
==> default:  -- Cpus:              2
==> default:  -- Memory:            4096M
==> default:  -- Management MAC:
==> default:  -- Loader:
==> default:  -- Base box:          ficus-devstack-2017-02-07
==> default:  -- Storage pool:      default
==> default:  -- Image:             /var/lib/libvirt/images/devstack_default_1478018111_5f69c2128844670d27e2.img (80G)
==> default:  -- Volume Cache:      default
==> default:  -- Kernel:
==> default:  -- Initrd:
==> default:  -- Graphics Type:     vnc
==> default:  -- Graphics Port:     5900
==> default:  -- Graphics IP:       127.0.0.1
==> default:  -- Graphics Password: Not defined
==> default:  -- Video Type:        cirrus
==> default:  -- Video VRAM:        9216
==> default:  -- Keymap:            en-us
==> default:  -- TPM Path:
==> default:  -- INPUT:             type=mouse, bus=ps2
==> default:  -- Command line :
==> default: Creating shared folders metadata...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
    default:
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Configuring and enabling network interfaces...
==> default: Exporting NFS shared folders...
==> default: Preparing to edit /etc/exports. Administrator privileges will be required...
Redirecting to /bin/systemctl status  nfs-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
  Drop-In: /run/systemd/generator/nfs-server.service.d
           └─order-with-mounts.conf
   Active: active (exited) since Wed 2017-02-15 17:33:21 EST
 Main PID: 1470 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 512)
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/nfs-server.service

Feb 15 17:33:20 gargan.jupiter systemd[1]: Starting NFS server and services...
Feb 15 17:33:21 gargan.jupiter systemd[1]: Started NFS server and services.
==> default: Mounting NFS shared folders...
==> default: Running provisioner: shell...
    default: Running: inline script
==> default: stdin: is not a tty
==> default:
==> default: PLAY [all] ********************************************************************
...

When this finishes, you can start Studio or LMS just like the edX-provided instructions:

$ vagrant ssh
vagrant@vagrant:~$ sudo su edxapp
edxapp@vagrant:~/edx-platform$ paver devstack studio  # or lms

If you ran with VAGRANT_NO_PORTS=1 then you’ll need the IP address of the virtual machine:

$ vagrant ssh-config
Host default
  HostName 192.168.121.155
  User vagrant
  Port 22
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /home/aron/devstack/.vagrant/machines/default/libvirt/private_key
  IdentitiesOnly yes
  LogLevel FATAL

Use the IP address shown in the output construct the URL. For example, in the output above, the LMS will be on http://192.168.121.155:8000 and Studio is on http://192.168.121.155:8001

Note that this IP address is a private IPv4 address as defined by RFC 1918, so if you’re running on a server separate from your own workstation, you’ll need some additional network hookup. I’ll show you what I use in a following article.

comments powered by Disqus