The Open edX devstack expects to run on Vagrant with VirtualBox. With a few tweaks, you can use Vagrant with KVM instead.
The instructions here are specific to the Ficus release. If you need Eucalyptus, look here.
Rationale
At Scampersand we SSH into a shared development server that provides a stable environment and regular backups without relying on individual hardware or configuration. That server runs Fedora and hosts docker containers and KVM virtual machines.
KVM and VirtualBox are mutually exclusive on Linux because they each supply a kernel module that handles the CPU virtualization instructions, so only one can be loaded at a time. Since we have an existing infrastructure based on KVM, we’d rather run the devstack on it than need to bring up a separate host for Virtualbox.
Steps
If you’d like to run the Open edX devstack on KVM, these are the steps you’ll need. I’ll explain these in detail below.
- Install Vagrant 1.8.7 or later, sorta
- Install the vagrant-libvirt plugin
- Enable user for vagrant with libvirt
- Mutate the devstack box
- Repair the network configuration
- Download the modified devstack Vagrantfile
- Fire it up!
Install Vagrant 1.8.7 or later, sorta
The Open edX Vagrantfile explicitly checks for Vagrant 1.8.7 or later. This is probably motivated by the bug that affected old Ubuntu boxes (even though it doesn’t affect Ficus), and might also be related to the bug with NFS exports.
You could get the latest version (1.9.1 at the time of writing) from vagrantup.com. However on Fedora this eventually leads to conflicts between Vagrant’s embedded ruby and the system-installed ruby.
Fedora 25 supplies Vagrant 1.8.5, but patched for the NFS bug. I recommend using the distro-supplied package, and planning to patch the Vagrantfile to allow it. (My modified Vagrantfile below does just that.)
$ sudo dnf install vagrant
$ vagrant -v
Vagrant 1.8.5
$ rpm -q --changelog vagrant | head -n2
* Tue Nov 15 2016 Vít Ondruch - 1.8.5-2
- Fix nfs_cleanup security race and permissions (rhbz#1395040).
Install the vagrant-libvirt plugin
“Wait, libvirt? I thought we were using KVM.”
Right, so let’s step back and review how this works. Vagrant is a tool for
controlling a virtual machine. The configuration for the VM is described in
Vagrantfile, and Vagrant depends on a virtualization provider to do the heavy
lifting. Most of the time the provider is Virtualbox, but Vagrant can also use
VMware, Hyper-V, Docker and others.
KVM is a kernel-level virtualization technology that’s managed by libvirtd in Linux distributions. In fact there used to be a KVM-specific provider plugin for Vagrant, but that’s been deprecated in favor of the libvirt provider plugin.
So in order to use Vagrant with KVM, you need the vagrant-libvirt plugin. You can install it like this:
$ sudo dnf install vagrant-libvirt
Enable user for vagrant with libvirt
The libvirt daemon will need to know that you’re authorized to manage virtual machines. The easiest way to do this is to add yourself to the libvirt group:
$ sudo usermod -aG libvirt $USER
There’s no need to log out and back in, since libvirtd checks group membership directly rather than via process inheritance.
It’s a good idea to make sure that virsh works properly before proceeding:
$ virsh --connect=qemu:///system sysinfo
If that command produces some XML output, you should be good to go. If it doesn’t, then you might need to investigate since vagrant-libvirt isn’t likely to work either.
Mutate the devstack box
The box image provided by edX is specific to to VirtualBox. Use the vagrant-mutate plugin to convert it for libvirt/KVM. This isn’t available from Fedora, so you’ll have to install it locally:
$ vagrant plugin install vagrant-mutate
Installing the 'vagrant-mutate' plugin. This can take a few minutes...
Installed the plugin 'vagrant-mutate (1.2.0)'!
$ vagrant box add --name ficus-devstack-2017-02-07 --provider virtualbox \
http://files.edx.org/vagrant-images/ficus-devstack-2017-02-07.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'ficus-devstack-2017-02-07' (v0) for provider: virtualbox
box: Downloading: http://files.edx.org/vagrant-images/ficus-devstack-2017-02-07.box
==> box: Successfully added box 'ficus-devstack-2017-02-07' (v0) for 'virtualbox'!
$ vagrant mutate ficus-devstack-2017-02-07 libvirt
Converting ficus-devstack-2017-02-07 from virtualbox to libvirt.
(100.00/100%)
The box ficus-devstack-2017-02-07 (libvirt) is now ready to use.
You can inspect what this does on the filesystem in your .vagrant.d directory.
$ tree ~/.vagrant.d/boxes
/home/aron/.vagrant.d/boxes
└── ficus-devstack-2017-02-07
└── 0
├── libvirt
│ ├── Vagrantfile
│ ├── box.img
│ └── metadata.json
└── virtualbox
├── Vagrantfile
├── box-disk1.vmdk
├── box.ovf
├── metadata.json
└── vagrant_private_key
Notice that the vagrant_private_key wasn’t included in the migration. While it
would be possible to copy this manually, the easier workaround is to set
config.ssh.password, and this is the approach already taken in the custom
Vagrantfile you’ll download next.
Repair the network configuration
Now you have ficus-devstack-2017-02-07 mutated to run under libvirt, but if
you try to boot it up, it will hang at this line:
==> default: Waiting for domain to get an IP address...
The problem is that the network interface is named differently when running under libvirt, so the networking configuration in the guest doesn’t apply properly.
There’s a pretty easy fix for this, using a program called guestfish. This nifty tool lets you crack open the virtual machine image to make small edits. The two small edits you need to make are: (1) modify grub’s kernel parameters so the network interface is predictably named, and (2) add the network configuration.
$ sudo dnf install libguestfs-tools-c
$ guestfish -a ~/.vagrant.d/boxes/ficus-devstack-2017-02-07/0/libvirt/box.img
Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.
Type: 'help' for help on commands
'man' to read the manual
'quit' to quit the shell
><fs> run
><fs> mount /dev/vagrant-vg/root /
><fs> mount /dev/sda1 /boot
><fs> edit /etc/default/grub
This will drop you into an editor. Change the value of
GRUB_CMDLINE_LINUX_DEFAULT so it reads like this:
GRUB_CMDLINE_LINUX_DEFAULT="quiet net.ifnames=0 biosdevname=0"
Save and exit, which puts you back at the guestfish prompt. First, verify that your change was written properly:
><fs> grep quiet /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet net.ifnames=0 biosdevname=0"
Next, run update-grub in the context of the guest filesystem. This will
rewrite the grub configuration at /boot/grub/grub.cfg with the new kernel
parameters.
><fs> command /usr/sbin/update-grub
><fs> grep quiet /boot/grub/grub.cfg
linux /vmlinuz-4.4.0-31-generic root=/dev/mapper/vagrant--vg-root ro quiet net.ifnames=0 biosdevname=0
linux /vmlinuz-4.4.0-31-generic root=/dev/mapper/vagrant--vg-root ro quiet net.ifnames=0 biosdevname=0
Finally, add the network configuration for eth0:
><fs> write /etc/network/interfaces.d/eth0 "auto eth0\niface eth0 inet dhcp\n"
><fs> quit
If you tried to start Vagrant without making these changes first, you’ll need to remove the box image from libvirt storage. Next time you run Vagrant it will re-import from your modified box.
virsh vol-delete ficus-devstack-2017-02-07_vagrant_box_image_0.img --pool default
Download the modified devstack Vagrantfile
The edX-provided Vagrantfile assumes Virtualbox, so I’ve provided a modified Vagrantfile that supports libvirt/KVM. You can inspect the differences here.
You should download this into a new directory where the sources will also eventually be checked out.
$ mkdir devstack
$ cd devstack
$ curl -OL https://raw.githubusercontent.com/scampersand/configuration/scampersand/ficus.libvirt-kvm/vagrant/release/devstack/Vagrantfile
Fire it up!
At this point you can start devstack. Since the
upstream Vagrantfile still defaults to Eucalyptus,
you’ll need to override it by setting OPENEDX_RELEASE in the environment.
$ export OPENEDX_RELEASE=open-release/ficus
If you will have more than one user running on the same host, then you should
start vagrant with VAGRANT_NO_PORTS=1 to avoid listening on a bunch of
localhost ports that will collide between users. Instead you can access the VM
by its own IP address as shown below.
$ VAGRANT_NO_PORTS=1 vagrant up --provider=libvirt
Bringing machine 'default' up with 'libvirt' provider...
==> default: Creating image (snapshot of base box volume).
==> default: Creating domain with the following settings...
==> default: -- Name: devstack_default_1478018111_5f69c2128844670d27e2
==> default: -- Domain type: kvm
==> default: -- Cpus: 2
==> default: -- Memory: 4096M
==> default: -- Management MAC:
==> default: -- Loader:
==> default: -- Base box: ficus-devstack-2017-02-07
==> default: -- Storage pool: default
==> default: -- Image: /var/lib/libvirt/images/devstack_default_1478018111_5f69c2128844670d27e2.img (80G)
==> default: -- Volume Cache: default
==> default: -- Kernel:
==> default: -- Initrd:
==> default: -- Graphics Type: vnc
==> default: -- Graphics Port: 5900
==> default: -- Graphics IP: 127.0.0.1
==> default: -- Graphics Password: Not defined
==> default: -- Video Type: cirrus
==> default: -- Video VRAM: 9216
==> default: -- Keymap: en-us
==> default: -- TPM Path:
==> default: -- INPUT: type=mouse, bus=ps2
==> default: -- Command line :
==> default: Creating shared folders metadata...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
default:
default: Inserting generated public key within guest...
default: Removing insecure key from the guest if it's present...
default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Configuring and enabling network interfaces...
==> default: Exporting NFS shared folders...
==> default: Preparing to edit /etc/exports. Administrator privileges will be required...
Redirecting to /bin/systemctl status nfs-server.service
● nfs-server.service - NFS server and services
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
Drop-In: /run/systemd/generator/nfs-server.service.d
└─order-with-mounts.conf
Active: active (exited) since Wed 2017-02-15 17:33:21 EST
Main PID: 1470 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 512)
Memory: 0B
CPU: 0
CGroup: /system.slice/nfs-server.service
Feb 15 17:33:20 gargan.jupiter systemd[1]: Starting NFS server and services...
Feb 15 17:33:21 gargan.jupiter systemd[1]: Started NFS server and services.
==> default: Mounting NFS shared folders...
==> default: Running provisioner: shell...
default: Running: inline script
==> default: stdin: is not a tty
==> default:
==> default: PLAY [all] ********************************************************************
...
When this finishes, you can start Studio or LMS just like the edX-provided instructions:
$ vagrant ssh
vagrant@vagrant:~$ sudo su edxapp
edxapp@vagrant:~/edx-platform$ paver devstack studio # or lms
If you ran with VAGRANT_NO_PORTS=1 then you’ll need the IP address of the
virtual machine:
$ vagrant ssh-config
Host default
HostName 192.168.121.155
User vagrant
Port 22
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
PasswordAuthentication no
IdentityFile /home/aron/devstack/.vagrant/machines/default/libvirt/private_key
IdentitiesOnly yes
LogLevel FATAL
Use the IP address shown in the output construct the URL. For example, in the
output above, the LMS will be on http://192.168.121.155:8000 and Studio is on
http://192.168.121.155:8001
Note that this IP address is a private IPv4 address as defined by RFC 1918, so if you’re running on a server separate from your own workstation, you’ll need some additional network hookup. I’ll show you what I use in a following article.