History, KVM and Openstack

IaaS/PaaS/SaaS, Ganeti, Cloud Images and Packer

History of Virtualization


Adam Jollans - IBM - SCALE 13x


Software, firmware or hardware that creates and runs virtual machines.

Types of Hypervisors

Type 1 - native or bare-metal hypervisor
  • Hypervisors that directly run and control the hardware on the host system and manage the guest systems
  • Examples include: Xen, VMWare ESX Server and Microsoft Hyper-V
Type 2 - hosted hypervisor
  • Hypervisors run on a conventional operating system and abstract guest operating systems from the host operating system
  • Examples include: VMWare Workstation and VirtualBox

Which hypervisor type does KVM fall under?

Virtualization Framework Generalized


Adam Jollans - IBM - SCALE 13x

Virtualization Framework - KVM


Adam Jollans - IBM - SCALE 13x


Open Source hypervisor based on Linux

  • Kernel-Based Virtual Machine
  • Kernel module that turns Linux into a virtual machine monitor
  • Requires hardware CPU support
  • Merged into mainline Linux
  • Emulator used for I/O device virtualization
  • Runs as a user-space process
  • Avi Kivity began the development of KVM at Qumranet in the mid-2000s

Processors Supported

KVM Visualized


Adam Jollans - IBM - SCALE 13x

KVM Command Line

KVM has a lot of options

# Create a virtual disk file
$ qemu-img create -f qcow2 disk.img 10g

# Start a VM up and boot to an ISO
$ qemu-system-x86_64 -hda disk.img -cdrom \
  /path/to/CentOS-6.6-x86_64-minimal.iso -boot d -m 1024m

KVM on OpenStack

/usr/libexec/qemu-kvm -name instance-00000baa -S -M rhel6.6.0 -cpu
-enable-kvm -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid
96bea55c-6d58-4dcf-bc52-09aed81c0cee -smbios type=1,manufacturer=RDO
-nodefconfig -nodefaults -chardev
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
-no-kvm-pit-reinjection -no-shutdown -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
-netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=39 -device
-device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1
-device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0
-vnc -k en-us -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on


Micro-kernel hypervisor for Linux or BSD

Xen Hypervisor Architecture

Xen Project runs in a more privileged CPU state than any other software on the machine.

  • Dom0 -- Domain that has privileged access to hardware, or node node
  • DomU -- Unprivileged domains (VMs), or guests

Types of Xen virtualization

Paravirtualization (PV)
  • Simulates real hardware (PV Guests)
  • Guests are run in a modified OS and use a special hypercall ABI
Hardware Virtual Machine (HVM)
  • Uses hardware assisted virtualization via the CPU
  • HVM guests with PV drivers (PV-on-HVM or PVH) provide better performance than pure PV

Xen Visualized


KVM vs. Xen

Less mature More mature
Runs like a normal unix app Blackbox feel for how it works
Always been in mainline Had trouble getting into mainline Linux
Requires CPU support Doesn't require CPU support
Development is very active and growing Development has slowed a bit
Used by newer cloud providers (i.e. GCE, DigitalOcean) Used by Amazon EC2

Libvirt -- KVM management

Libvirt is an open source API daemon that standardizes management of various virtualization platforms.


Adam Jollans - IBM - SCALE 13x

Libvirt Features


<Name> as a Service

Software as a Service
Platform as a Service
Infrastructure as a Service

Infrastructure as a Service

Virtual computing platform that typically includes automated methods for deploying virtual machines on a set of physical machines


Platform as a Service

A platform that provides customers the ability to develop, run and manage web applications without the complexity of building and maintaining the underlying infrastructure

Typically layered on top of IaaS


Software as a Service

Software delivery model in which software is provided on a subscription basis and centrally hosted. Also referred to as "on-demand software".


IaaS Platforms

Private Cloud
  • OpenStack
  • Ganeti
  • oVirt
  • Apache CloudStack
Public Cloud
  • Amazon EC2
  • Linode
  • DigitalOcean
  • Rackspace


Open source cloud computing platform

OpenStack Visualized


Adam Jollans - IBM - SCALE 13x

Provision a VM on OpenStack


Adam Jollans - IBM - SCALE 13x


Key Features:
  • High-availability built-in
  • Relatively simple architecture compared to other platforms
  • Easy to expand and manage
  • No cloud-like features by default -- good for "pet" VMs
  • Designed to deal with hardware failures
  • Does not use libvirt (was created before libvirt existed)
  • Primary CLI driven
  • Easy to customize

Ganeti Cluster


Ganeti Architecture


Ganeti Components

Distributed Replicated Block Device (DRBD)

Distributed replicated storage system (think RAID1 over the network)


KVM Live Migration

A feature that allows a virtual machine to move from one host to another host while staying online.

KVM Live Migration

KVM Live Migration

KVM Live Migration

KVM Live Migration

KVM Live Migration

KVM Live Migration

Primary and Secondary Nodes


Cloud/System Image

A copy of an operating system including the entire state of the computer system stored in a non-volatile form such as a file.

Image Formats

  • Used by QEMU/KVM
  • Stands for "QEMU Copy On Write"
VHD (Virtual Hard Disk)
  • Format created by Connectix which was later acquired by Microsoft
  • Used primarily by Hyper-V

Image Formats

VMDK (Virtual Machine Disk)
  • Initially developed by VMWare
  • An open format and used by VirtualBox, QEMU and Parellels
AMI (Amazon Machine Image)
  • Disk image format used on EC2
  • Compress, encrypted, signed and split into a series of 10MB checks and uploaded on S3
  • Contains an XML manifest file
  • Does not contain kernel image

Image files vs Block Devices

Image Files Block Devices
Easy to move around and create Requires use of LVM or other block device tools
Can have a performance hit Typically has better performance
Offer more features such as compression You can't 'overcommit' space with LVM

Creating Images

Various tools exist to create images. Some are distribution specific while others aren't.

  • Python CLI app that uses KVM to install a virtual machine image
  • Typically used to create RHEL-based images, but has support for Debian and Windows.
  • Uses an XML file format
  • CLI tool that is typically used to create Debian or Ubuntu images

Creating Images

  • CLI tool that only works on Fedora but works on other RHEL systems
  • CLI tool to create Vagrant boxes, but can also create KVM images
  • CLI tool for creating machine images for multiple platforms
  • Tool that integrates with Oz to automate building, converting and uploading of images to different cloud providers.


Amazon EC2 Digital Ocean
Docker GCE
Openstack Parallels
QEMU (kvm) Virtual Box

What problem does Packer solve?


JSON files containing the build information
Platform specific building configuration
Tools that install software after the initial OS install
Actions to happen after the image has been built

Packer Build Steps

This varies depending on which builder you use. The following is an example for the QEMU builder

  1. Download ISO image
  2. Create virtual machine
  3. Boot virtual machine from the CD
  4. Using VNC, type in commands in the installer to start an automated install via kickstart/preseed/etc
  5. Packer automatically serves kickstart/preseed file with a built-in http server

Packer Build Steps

  1. Packer waits for ssh to become available
  2. OS installer runs and then reboots
  3. Packer connects via ssh to VM and runs provisioner (if set)
  4. Packer Shuts down VM and then runs the post processor (if set)
  5. PROFIT!

How it works

  "builders": [
      "boot_command": [
        "<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/centos-7.0/ks-openstack.cfg",
      "accelerator": "kvm",
      "boot_wait": "10s",
      "disk_size": 2048,
      "headless": true,
      "http_directory": "http",
      "iso_checksum": "df6dfdd25ebf443ca3375188d0b4b7f92f4153dc910b17bccc886bd54a7b7c86",
      "iso_checksum_type": "sha256",
      "iso_url": "{{user `mirror`}}/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-NetInstall.iso",
      "output_directory": "packer-centos-7.0-x86_64-openstack",
      "qemuargs": [ [ "-m", "1024m" ] ],
      "qemu_binary": "qemu-kvm",
      "shutdown_command": "echo 'centos'\|sudo -S /sbin/halt -h -p",
      "ssh_password": "centos",
      "ssh_port": 22,
      "ssh_username": "centos",
      "ssh_wait_timeout": "10000s",
      "type": "qemu",
      "vm_name": "packer-centos-7.0-x86_64"
  "provisioners": [
      "environment_vars": [
        "CHEF_VERSION={{user `chef_version`}}"
      "execute_command": "echo 'centos' | {{.Vars}} sudo -S -E bash '{{.Path}}'",
      "scripts": [
      "type": "shell"
  "variables": {
    "chef_version": "provisionerless",
    "mirror": ""

Building the Image

$ packer build centos-7.0-x86_64-openstack.json
qemu output will be in this color.

==> qemu: Downloading or copying ISO
    qemu: Downloading or copying:
==> qemu: Creating hard drive...
==> qemu: Starting HTTP server on port 8081
==> qemu: Found port for SSH: 3213.
==> qemu: Looking for available port between 5900 and 6000
==> qemu: Found available VNC port: 5947
==> qemu: Starting VM, booting from CD-ROM
    qemu: WARNING: The VM will be started in headless mode, as configured.
    qemu: In headless mode, errors during the boot sequence or OS setup
    qemu: won't be easily visible. Use at your own discretion.
==> qemu: Overriding defaults Qemu arguments with QemuArgs...
==> qemu: Waiting 10s for boot...
==> qemu: Connecting to VM via VNC
==> qemu: Typing the boot command over VNC...
==> qemu: Waiting for SSH to become available...


Shell Run either inline or shell scripts
File Uploads Upload files and use shell scripts to move files around as needed
Ansible Provision using playbook and role files
Chef Client Connect to a chef server and run chef
Chef Solo Run a Chef solo run by pointing to local cookbooks or uploading them
Puppet Masterless Run local manifests and modules
Puppet Server Connect to a puppet server and run puppet
Salt Using Salt states, deploy a vm using Salt