Pre-requisites¶

1. A supported host platforms: Red Hat Enterprise Linux (RHEL) or Fedora. OpenSUSE should also work, but currently autotest is still not packaged for it, which means you have to clone autotest and put its path in an env variable so virt tests can find the autotest libs. Debian/Ubuntu now have a new experimental package that allows one to run the virt tests in a failry straight forward way.
2. Install software packages (RHEL/Fedora)
3. Install software packages (Debian/Ubuntu)
4. A copy of the virt test source

For the impatient¶

1. Clone the virt test repo

git clone git://github.com/autotest/virt-test.git

2. Get into the base dir

cd virt-test

3. Run the bootstrap procedure. For example, if you want to run the qemu subtest, you will run:

./run -t qemu --bootstrap

This script will check if you have the minimum requirements for the test (required commands and includes), and download the JeOS image. You can omit running this script, since the code of this script also gets to run when you call the test runner, but it is discouraged. Explicitly running get_started.py first is iteractive, and gives you a better idea of what is going on.

4. For qemu and libvirt subtests, the default test set does not require root. However, other tests might fail due to lack of privileges.

$ ./run -t qemu

or

# ./run -t libvirt

If you ran get_started.py, the test runner should just run the test. If you didn’t, the runner will trigger the environment setup procedure:

1. Create the /var/tmp/libvirt_test dir to hold images and isos
2. Download the JeOS image (180 MB, takes about 3 minutes on a fast connection) and uncompress it (takes about a minute on an HDD laptop). p7ip has to be present.
3. Run a predefined set of tests.

Running different tests¶

You can list the available tests to run by using the flag –list-tests

$ ./run -t qemu --list-tests
(will print a numbered list of tests, with a paginator)

Then you can pass tests that interest you with –tests “list of tests”, for example:

1. qemu

$ ./run -t qemu --tests "migrate timedrift file_transfer"

2. Libvirt requires first importing the JeOS image. However, this cannot be done if the guest already exists. Therefore, it’s wise to also conclude a set with the remove_guest.without_disk test.

# ./run -t libvirt --tests "unattended_install.import.import boot reboot remove_guest.without_disk"

Checking the results¶

The test runner will produce a debug log, that will be useful to debug problems:

$ ./run -t qemu --tests usb
Running setup. Please wait...
SETUP: PASS (13.52 s)
DATA DIR: /home/lmr/virt_test
DEBUG LOG: /home/lmr/Code/virt-test.git/logs/run-2014-01-27-14.25.31/debug.log
TESTS: 203
(1/203) type_specific.io-github-autotest-qemu.usb.usb_boot.usb_kbd.without_usb_hub.uhci: PASS (25.47 s)
(2/203) type_specific.io-github-autotest-qemu.usb.usb_boot.usb_kbd.without_usb_hub.ehci: PASS (23.53 s)
(3/203) type_specific.io-github-autotest-qemu.usb.usb_boot.usb_kbd.without_usb_hub.xhci: PASS (24.34 s)
...

Here you can see that the debug log is in /home/lmr/Code/virt-test.git/logs/run-2014-01-27-14.25.31/debug.log. For convenience, the most recent log is pointed to by the logs/latest symlink.

Getting Help¶

The best way to get help from the command line options is the –help flag:

./run --help

General Flow¶

The test runner is nothing more than a very simple test harness, that replaces the autotest harness, and a set of options, that will trigger actions to create a test list and execute it. The way the tests work is:

1. Get a dict with test parameters
2. Based on these params, prepare the environment - create or destroy vm instances, create/check disk images, among others
3. Execute the test itself, that will use several of the params defined to carry on with its operations, that usually involve:
4. If a test did not raise an exception, it PASSed
5. If a test raised an exception it FAILed
6. Based on what happened during the test, perform cleanup actions, such as killing vms, and remove unused disk images.

The list of parameters is obtained by parsing a set of configuration files, present inside the SourceStructure. The command line options usually modify even further the parser file, so we can introduce new data in the config set.

Common Operations – Listing guests¶

If you want to see all guests defined, you can use

./run -t [test type] --list-guests

This will generate a list of possible guests that can be used for tests, provided that you have an image with them. The list will show which guests don’t have an image currently available. If you did perform the usual bootstrap procedure, only JeOS.17.64 will be available.

Now, let’s assume you have the image for another guest. Let’s say you’ve installed Fedora 17, 64 bits, and that –list-guests shows it as downloaded

./run -t qemu --list-guests
... snip...
16 Fedora.17.32 (missing f17-32.qcow2)
17 Fedora.17.64
18 Fedora.8.32 (missing f8-32.qcow2)

You can list all the available tests for Fedora.17.64 (you must use the exact string printed by the test, minus obviously the index number, that’s there only for informational purposes:

./run -t qemu -g Fedora.17.64 --list-tests
... snip ...
26 balloon_check.base
27 balloon_check.balloon-migrate
28 balloon_check.balloon-shutdown_enlarge
29 balloon_check.balloon-shutdown_evict
30 block_mirror
31 block_stream
... snip ...

Then you can execute one in particular. It’s the same idea, just copy the individual test you want and run it:

./run -t qemu -g Fedora.17.64 --tests balloon_check.balloon-migrate

And it’ll run that particular test.

Tip: By the rules of the cartesian config files, you can use:

./run -t qemu -g Fedora.17.64 --tests balloon_check

And it’ll run all tests from 26-29. Very useful for large sets, such as virtio_console and usb - You can just do a:

./run -t qemu --tests virtio_console
... 118 tests ...

./run -t qemu --tests usb
... 64 tests ...

Note that in the examples above, the fact I didn’t provide -g means that we’re using the default guest OS, that is, JeOS.

Tests that are not part of the default JeOS set¶

If you want to run guest install tests, you need to be able to create floppies and isos to hold kickstart files:

yum install mkisofs

For newer distros, such as Fedora, you’ll need:

yum install genisoimage

Both packages provide the same functionality, needed to create iso images that will be used during the guest installation process. You can also execute

Network tests¶

Last bug not least, now we depend on libvirt to provide us a stable, working bridge. * By default, the kvm test uses user networking, so this is not entirely necessary. However, non root and user space networking make a good deal of the hardcode networking tests to not work. If you might want to use bridges eventually:

yum install libvirt bridge-utils

Make sure libvirtd is started:

[lmr@freedom autotest.lmr]$ service libvirtd start

Make sure the libvirt bridge shows up on the output of brctl show:

[lmr@freedom autotest.lmr]$ brctl show
bridge name bridge id       STP enabled interfaces
virbr0      8000.525400678eec   yes     virbr0-nic

Tests that are not part of the default JeOS set¶

If you want to run guest install tests, you need to be able to create floppies and isos to hold kickstart files:

apt-get install genisoimage

Network tests¶

Last bug not least, now we depend on libvirt to provide us a stable, working bridge. * By default, the kvm test uses user networking, so this is not entirely necessary. However, non root and user space networking make a good deal of the hardcode networking tests to not work. If you might want to use bridges eventually:

apt-get install libvirt-bin python-libvirt bridge-utils

Make sure libvirtd is started:

[lmr@freedom autotest.lmr]$ service libvirtd start

Make sure the libvirt bridge shows up on the output of brctl show:

[lmr@freedom autotest.lmr]$ brctl show
bridge name bridge id       STP enabled interfaces
virbr0      8000.525400678eec   yes     virbr0-nic

Autotest¶

Virtualization Tests¶

Introduction¶

The virt-test suite helps exercise virtualization features with help from qemu, libvirt, and other related tools and facilities. However, due to it’s scope and complexity, this aspect of Autotest has been separated into the dedicated ‘virt-test’ suite. This suite includes multiple packages dedicated to specific aspects of virtualization testing.

Within each virt-test package, are a collection of independent sub-test modules. These may be addressed individually or as part of a sequence. In order to hide much of the complexity involved in virtualization testing and development, a dedicated test-runner is included with the virt-test suite (see section test_runner).

git clone git://github.com/autotest/virt-test.git

cd virt-test

./run -t qemu

./run -t libvirt

$ ./run -t qemu --list-tests
(will print a numbered list of tests, with a pagination)

[lmr@localhost virt-test.git]$ ./run -t qemu --tests boot_with_usb
SETUP: PASS (1.20 s)
DATA DIR: /path/to/virt_test
DEBUG LOG: /path/to/virt-test.git/logs/run-2012-12-12-01.39.34/debug.log
TESTS: 10
boot_with_usb.ehci: PASS (18.34 s)
boot_with_usb.keyboard.uhci: PASS (21.57 s)
boot_with_usb.keyboard.xhci: PASS (24.56 s)
boot_with_usb.mouse.uhci: PASS (21.59 s)
boot_with_usb.mouse.xhci: PASS (23.11 s)
boot_with_usb.usb_audio: PASS (20.99 s)
boot_with_usb.hub: PASS (22.12 s)
boot_with_usb.storage.uhci: PASS (21.61 s)
boot_with_usb.storage.ehci: PASS (23.27 s)
boot_with_usb.storage.xhci: PASS (25.03 s)

Detailed Test Execution¶

Tests are executed from a copy of the Autotest client code, typically on separate hardware from the Autotest server (if there is one). Executing tests directly from a clone of the git repositories or installed Autotest is possible. The tree is configured such that test results and local configuration changes are kept separate from test and Autotest code.

For virtualization tests, variant selection(s) and configuration(s) is required either manually through specification in tests.cfg (see section tests_cfg) or automatically by using the test-runner (see section run_different_tests). The test-runner is nearly trivial to use, but doesn’t offer the entire extent of test customization. See the virt_test_runner section for more information.

variants:
    - one:
        key1 = Hello
    - two:
        key2 = World
    - three:
variants:
    - four:
        key3 = foo
    - five:
        key3 = bar
    - six:
        key1 = foo
        key2 = bar

dict    1:  four.one
dict    2:  four.two
dict    3:  four.three
dict    4:  five.one
dict    5:  five.two
dict    6:  five.three
dict    7:  six.one
dict    8:  six.two
dict    9:  six.three

variants var1_name:
     - one:
         key1 = Hello
     - two:
         key2 = World
     - three:
variants var2_name:
     - one:
         key3 = Hello2
     - two:
         key4 = World2
     - three:

only (var2_name=one).(var1_name=two)

dict    1:  (var2_name=one).(var1_name=two)
      dep = []
      key2 = World         # variable key2 from variants var1_name and variant two.
      key3 = Hello2        # variable key3 from variants var2_name and variant one.
      name = (var2_name=one).(var1_name=two)
      shortname = (var2_name=one).(var1_name=two)
      var1_name = two      # variant name in same namespace as variables.
      var2_name = one      # variant name in same namespace as variables.

variants guest_os:
     - fedora:
     - ubuntu:
variants disk_interface:
     - virtio:
     - hda:

dict    1:  (disk_interface=virtio).(guest_os=fedora)
    dep = []
    disk_interface = virtio
    guest_os = fedora
    name = (disk_interface=virtio).(guest_os=fedora)
    shortname = (disk_interface=virtio).(guest_os=fedora)
dict    2:  (disk_interface=virtio).(guest_os=ubuntu)
    dep = []
    disk_interface = virtio
    guest_os = ubuntu
    name = (disk_interface=virtio).(guest_os=ubuntu)
    shortname = (disk_interface=virtio).(guest_os=ubuntu)
dict    3:  (disk_interface=hda).(guest_os=fedora)
    dep = []
    disk_interface = hda
    guest_os = fedora
    name = (disk_interface=hda).(guest_os=fedora)
    shortname = (disk_interface=hda).(guest_os=fedora)
dict    4:  (disk_interface=hda).(guest_os=ubuntu)
    dep = []
    disk_interface = hda
    guest_os = ubuntu
    name = (disk_interface=hda).(guest_os=ubuntu)
    shortname = (disk_interface=hda).(guest_os=ubuntu)

variants:
    - one:
        key1 = Hello
    - two: one
        key2 = World
    - three: one two

Linux..OpenSuse:
initrd = initrd

only Linux..Fedora..64

variants:
    - one:
        key1 = Hello
variants:
    - two:
        key2 = Complicated
    - three: one two
        key3 = World
variants:
    - default:
        only three
        key2 =

only default

name = default.three.one
key1 = Hello
key2 =
key3 = World

key1 = default value
key2 = default value

sub = "key1: ${key1}; key2: ${key2};"

variants:
    - one:
        key1 = Hello
        sub = "key1: ${key1}; key2: ${key2};"
    - two: one
        key2 = World
        sub = "key1: ${key1}; key2: ${key2};"
    - three: one two
        sub = "key1: ${key1}; key2: ${key2};"

dict    1:  one
    dep = []
    key1 = Hello
    key2 = default value
    name = one
    shortname = one
    sub = key1: Hello; key2: default value;
dict    2:  two
    dep = ['one']
    key1 = default value
    key2 = World
    name = two
    shortname = two
    sub = key1: default value; key2: World;
dict    3:  three
    dep = ['one', 'two']
    key1 = default value
    key2 = default value
    name = three
    shortname = three
    sub = key1: default value; key2: default value;

vms = vm1 second_vm another_vm
mem = 128
mem_vm1 = 512
mem_second_vm = 1024

.. means AND
. means IMMEDIATELY-FOLLOWED-BY

means match all dicts whose names have:
(qcow2 AND (Fedora IMMEDIATELY-FOLLOWED-BY 14)) OR
((RHEL IMMEDIATELY-FOLLOWED-BY 6) AND raw AND boot) OR
(smp2 AND qcow2 AND migrate AND ide)

'qcow2..Fedora.14' is not equivalent to 'qcow2..14.Fedora'.
'ide, scsi' is equivalent to 'scsi, ide'.

Dependencies¶

If you write an application that is supposed to be run on the test-target, place it in the directory ../deps/<name>/ relative to where your test case is placed. The easiest way to obtain the full path to this directory is by calling data_dir.get_deps_dir(“<name>”). Don’t forget to add from virttest import data_dir to your test case.

Besides the source file, create a Makefile that will be used to build your test application. The below example shows a Makefile for the application for the timedrift test cases. The remote_build module requires that a Makefile is included with all test applications.

CFLAGS+=-Wall
LDLIBS+=-lrt

.PHONY: clean

all: clktest get_tsc

clktest: clktest.o

get_tsc: get_tsc.o

clean:
        rm -f clktest get_tsc

remote_build¶

To simplfy the building of applications on target, and to simplify avoiding the building of applications on target when they are installed pre-built, use the remote_build module. This module handles both the transfer of files, and running make on target.

A simple example:

address = vm.get_address(0)
source_dir = data_dir.get_deps_dir("<testapp>")
builder = remote_build.Builder(params, address, source_dir)
full_build_path = builder.build()

In this case, we utilize the .build() method, which execute the neccessary methods in builder to copy all files to target and run make (if needed). When done, .build() will return the full path on target to the application that was just built. Be sure to use this path when running your test application, as the path is changed if the parameters of the build is changed. For example:

session.cmd_status(%s --test" % os.path.join(full_build_path, "testapp"))

The remote_build.Builder class can give you fine-grained control over your build process as well. Another way to write the above .build() invocation above is:

builder = remote_build.Builder(params, address, source_dir)
if builder.sync_directories():
    builder.make()
full_build_path = builder.full_build_path

This pattern can be useful if you e.g. would like to add an additonal command to run before builder.make(), perhaps to install some extra dependencies.

A bit of context on how autotest works¶

The default upstream configuration file instructs autotest to perform the following tasks:

1. Install a Linux guest, on this case, Fedora 15, due to the fact it is publicly available, so everybody can try it out. The hardware configuration for the VM:
   • one qcow2 image on an ide bus
   • one network card, model rtl8139
   • two cpus
   • no pci hotplug devices will be attached to this vm
   • Also, the VM is not going to use hugepage memory explicitly
2. Run a boot test.
3. Run a shutdown test.

# Runs qemu-kvm, f15 64 bit guest OS, install, boot, shutdown
- @qemu_kvm_f15_quick:
    # We want qemu-kvm for this run
    qemu_binary = /usr/bin/qemu-kvm
    qemu_img_binary = /usr/bin/qemu-img
    only qcow2
    only rtl8139
    only ide
    only smp2
    only no_pci_assignable
    only smallpages
    only Fedora.15.64
    only unattended_install.cdrom, boot, shutdown

This is defined in such a way that the kvm test config system will generate only 3 tests. Let’s see at the tests generated. On the kvm test dir $AUTOTEST_ROOT/client/tests/kvm, you can call the configuration parser:

[lmr@freedom kvm]$ ../../common_lib/cartesian_config.py tests.cfg
dict    1:  smp2.Fedora.15.64.unattended_install.cdrom
dict    2:  smp2.Fedora.15.64.boot
dict    3:  smp2.Fedora.15.64.shutdown

You can see on top of the file tests.cfg some includes that point us from where all the test information comes from:

# Copy this file to tests.cfg and edit it.
#
# This file contains the test set definitions. Define your test sets here.
include tests_base.cfg
include cdkeys.cfg
include virtio-win.cfg

tests_base.cfg is a pretty large file, that contains a lot of variants, that are blocks defining tests, vm hardware and pre/post processing directives that control autotest infrastructure behavior. You can check out the definition of each of the variants restricting the test sets (qcow2, rtl8139, smp2, no_pci_assignable, smallpages, Fedora 15.64, unattended_install.cdrom, boot, shutdown) on tests_base.cfg.

About guest install¶

It is no mystery that for a good deal of the virtualization tests we are going to execute, a guest with an operating system on its disk image is needed. To get this OS there, we have some methods defined:

1. Install the VM with the OS CDROM through an engine that interacts with the VM using VNC, simulating a human being, called step engine. This engine works surprisingly well, frequently yielding successful installations. However, each step is a point of failure of the whole process, so we moved towards handing the dirty install control to the guest OS itself, as many of them have this capacity.
2. Install the VM using the automated install mechanisms provided by the guest OS itself. In windows, we have a mechanism called answer files, for Fedora and RHEL we have kickstarts, and for OpenSUSE we have autoyast. Of course, other OS, such as debian, also have their own mechanism, however they are not currently implemented in autotest (hint, hint).

And then an even simpler alternative:

1. Just copy a known good guest OS image, that was already installed, and use it. This tends to be faster and less error prone, since the install is already done, so we don’t need to work on failures on this step. The inevitable question that arises:

• Q. Hey, why don’t you just go with that on the first place?
• A. Because installing a guest exercises several aspects of the VM, such as disk, network, hardware probing, so on and so forth, so it’s a good functional test by itself. Also, installing manually a guest may work well for a single developer working on his/her patches, but certainly does not scale if you need fully automated test done on a regular basis, as there is the need of someone going there and making the install, which seriously, is a waste of human resources. KVM autotest is also a tool for doing such a massively automated test on a regular basis.

Also, this method assumes the least possible for the person running the tests, as they won’t need to have preinstalled guests, and because we always get the same vm, with the same capabilities and same configuration. Now that we made this point clear, let’s explain how to use your preinstalled guest.

Needed setup for a typical linux guest¶

virt-test relies heavily on cartesian config files. Those files use a flexible file format, defined on the file format documentation If you are curious about the defaults assumed for Linux or Windows guests, you can always check the file base.cfg.sample, which contains all our guest definitions (look at the Linux or Windows variant). Without diving too much into it, it’s sufficient to say that you need a guest to have a root password of 123456 and an enabled ssh daemon which will allow you to log in as root. The password can be also configured through the config files.

Before you start¶

1. Make sure you have the appropriate packages installed. You can read the install prerequesite packages (client section) for more information. For this how to our focus is not to build kvm from git repos, so we are assuming you are going to use the default qemu installed in the system. However, if you are interested in doing so, you might want to recap our docs on building qemu-kvm and running unittests.

Step by step procedure¶

1. Git clone autotest to a convenient location, say $HOME/Code/autotest. See the download source documentation. Please do use git and clone the repo to the location mentioned.

2. Execute the ./run -t qemu --bootstrap command (see the get started documentation <GetStarted>. Since we are going to boot our own guests, you can safely skip each and every iso download possible.

3. Edit the file tests.cfg. You can see we have a session overriding Custom Guest definitions present on tests_base.cfg. If you want to use a raw block device (see image_raw_device), you can uncomment the lines mentioned on the comments. When image_raw_device = yes, virt-test will not append a ‘.qcow2’ extension to the image name. Important: If you opt for a raw device, you must comment out the line that appends a base path to image names (one that looks like image_name(_.*)? ?<= /tmp/kvm_autotest_root/images/)

   CustomGuestLinux:
       # Here you can override the default login credentials for your custom guest
       username = root
       password = 123456
       image_name = custom_image_linux
       image_size = 10G
       # If you want to use a block device as the vm disk, uncomment the 2 lines
       # below, pointing the image name for the device you want
       #image_name = /dev/mapper/vg_linux_guest
       #image_raw_device = yes
   

4. Some lines below, you will also find this config snippet. This is for the case where you want to specify new base directories for kvm autotest to look images, cdroms and floppies.

   # Modify/comment the following lines if you wish to modify the paths of the
   # image files, ISO files or qemu binaries.
   #
   # As for the defaults:
   # * qemu and qemu-img are expected to be found under /usr/bin/qemu-kvm and
   #   /usr/bin/qemu-img respectively.
   # * All image files are expected under /tmp/kvm_autotest_root/images/
   # * All install iso files are expected under /tmp/kvm_autotest_root/isos/
   # * The parameters cdrom_unattended, floppy, kernel and initrd are generated
   #   by virt-test, so remember to put them under a writable location
   #   (for example, the cdrom share can be read only)
   image_name(_.*)? ?<= /tmp/kvm_autotest_root/images/
   cdrom(_.*)? ?<= /tmp/kvm_autotest_root/
   floppy ?<= /tmp/kvm_autotest_root/
   

5. Change the fields image_name, image_size to your liking. Now, the example test set that uses custom guest configuration can be found some lines below:

   # Runs your own guest image (qcow2, can be adjusted), all migration tests
   # (on a core2 duo laptop with HD and 4GB RAM, F15 host took 3 hours to run)
   # Be warned, disk stress + migration can corrupt your image, so make sure
   # you have proper backups
   - @qemu_kvm_custom_migrate:
       # We want qemu-kvm for this run
       qemu_binary = /usr/bin/qemu-kvm
       qemu_img_binary = /usr/bin/qemu-img
       only qcow2
       only rtl8139
       only ide
       only smp2
       only no_pci_assignable
       only smallpages
       only CustomGuestLinux
       only migrate
   

6. Since we want to execute this custom migrate test set, we need to look at the last couple of lines of the configuration file:

   # Choose your test list from the testsets defined
   only qemu_kvm_f15_quick
   

7. This line needs to become

   # Choose your test list from the testsets defined
   only qemu_kvm_custom_migrate
   

8. Now, if you haven’t changed any of the settings of the previous blocks, now our configuration system will run tests with the following expectations:

• qemu-kvm and qemu are under /usr/bin/qemu-kvm and /usr/bin/qemu-kvm, respectively. Please remember RHEL installs qemu-kvm under ``/usr/libexec``.
• Our guest image is under /tmp/kvm_autotest_root/images/custom_image_linux.qcow2, since the test set specifies only qcow2.
• All current combinations for our migrate tests variant will be executed with your custom image. It is never enough to remember that some of the tests can corrupt your qcow2 (or raw) image.

1. If you want to verify all tests that the config system will generate, you can run the parser to tell you that. This set took 3 hours to run on my development laptop setup.

   [lmr@freedom kvm]$ ../../common_lib/cartesian_config.py tests.cfg
   dict    1:  smp2.CustomGuestLinux.migrate.tcp
   dict    2:  smp2.CustomGuestLinux.migrate.unix
   dict    3:  smp2.CustomGuestLinux.migrate.exec
   dict    4:  smp2.CustomGuestLinux.migrate.mig_cancel
   dict    5:  smp2.CustomGuestLinux.migrate.with_set_speed.tcp
   dict    6:  smp2.CustomGuestLinux.migrate.with_set_speed.unix
   dict    7:  smp2.CustomGuestLinux.migrate.with_set_speed.exec
   dict    8:  smp2.CustomGuestLinux.migrate.with_set_speed.mig_cancel
   dict    9:  smp2.CustomGuestLinux.migrate.with_reboot.tcp
   dict   10:  smp2.CustomGuestLinux.migrate.with_reboot.unix
   dict   11:  smp2.CustomGuestLinux.migrate.with_reboot.exec
   dict   12:  smp2.CustomGuestLinux.migrate.with_reboot.mig_cancel
   dict   13:  smp2.CustomGuestLinux.migrate.with_file_transfer.tcp
   dict   14:  smp2.CustomGuestLinux.migrate.with_file_transfer.unix
   dict   15:  smp2.CustomGuestLinux.migrate.with_file_transfer.exec
   dict   16:  smp2.CustomGuestLinux.migrate.with_file_transfer.mig_cancel
   dict   17:  smp2.CustomGuestLinux.migrate.with_autotest.dbench.tcp
   dict   18:  smp2.CustomGuestLinux.migrate.with_autotest.dbench.unix
   dict   19:  smp2.CustomGuestLinux.migrate.with_autotest.dbench.exec
   dict   20:  smp2.CustomGuestLinux.migrate.with_autotest.dbench.mig_cancel
   dict   21:  smp2.CustomGuestLinux.migrate.with_autotest.stress.tcp
   dict   22:  smp2.CustomGuestLinux.migrate.with_autotest.stress.unix
   dict   23:  smp2.CustomGuestLinux.migrate.with_autotest.stress.exec
   dict   24:  smp2.CustomGuestLinux.migrate.with_autotest.stress.mig_cancel
   dict   25:  smp2.CustomGuestLinux.migrate.with_autotest.monotonic_time.tcp
   dict   26:  smp2.CustomGuestLinux.migrate.with_autotest.monotonic_time.unix
   dict   27:  smp2.CustomGuestLinux.migrate.with_autotest.monotonic_time.exec
   dict   28:  smp2.CustomGuestLinux.migrate.with_autotest.monotonic_time.mig_cancel
   

2. If you want to make sure virt-test is assigning images to the right places, you can tell the config system to print the params contents for each test.

   [lmr@freedom kvm]$ ../../common_lib/cartesian_config.py -c tests.cfg | less
   ... lots of output ...
   

3. In any of the dicts you should be able to see an image_name key that has something like the below. virt-test will only append ‘image_format’ to this path and then use it, so in the case mentioned, ‘/tmp/kvm_autotest_root/images/custom_image_linux.qcow2’

   image_name = /tmp/kvm_autotest_root/images/custom_image_linux
   

4. After you have verified things, you can run autotest using the command line get_started.py has informed you:

   $AUTOTEST_ROOT/client/bin/autotest $AUTOTEST_ROOT/client/tests/kvm/control
   

5. Profit!

Common questions¶

• Q: How do I restrict the test set so it takes less time to run?

• A: You can look at the output of the cartesian config parser and check out the test combinations. If you look at the output above, and say you want to run only migration + file transfer tests, your test set would look like the below snippet. Make sure you validate your changes calling the parser again.

  # Runs your own guest image (qcow2, can be adjusted), all migration tests
  # (on a core2 duo laptop with HD and 4GB RAM, F15 host took 3 hours to run)
  # Be warned, disk stress + migration can corrupt your image, so make sure
  # you have proper backups
  - @qemu_kvm_custom_migrate:
      # We want qemu-kvm for this run
      qemu_binary = /usr/bin/qemu-kvm
      qemu_img_binary = /usr/bin/qemu-img
      only qcow2
      only rtl8139
      only ide
      only smp2
      only no_pci_assignable
      only smallpages
      only CustomGuestLinux
      only migrate.with_file_transfer
  

  [lmr@freedom kvm]$ ../../common_lib/cartesian_config.py tests.cfg
  dict    1:  smp2.CustomGuestLinux.migrate.with_file_transfer.tcp
  dict    2:  smp2.CustomGuestLinux.migrate.with_file_transfer.unix
  dict    3:  smp2.CustomGuestLinux.migrate.with_file_transfer.exec
  dict    4:  smp2.CustomGuestLinux.migrate.with_file_transfer.mig_cancel
  

What is profiling¶

Profiling, by its definition (see this wikipedia article for a non formal introduction), is to run an analysis tool to inspect the behavior of a certain property of the system (be it memory, CPU consumption or any other).

How autotest can help with profiling?¶

Autotest provides support for running profilers during the execution of tests, so we know more about a given system resource. For the kvm test, our first idea of profiling usage was to run the kvm_stat program, that usually ships with kvm, to provide data useful for debugging. kvm_stat provides the number of relevant kvm events every time it is called, so by the end of a virt-test test we end up with a long list of information like this one:

kvm_ack_i  kvm_age_p   kvm_apic  kvm_apic_  kvm_apic_  kvm_async  kvm_async  kvm_async  kvm_async  kvm_cpuid     kvm_cr  kvm_emula  kvm_entry   kvm_exit  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(  kvm_exit(    kvm_fpu  kvm_hv_hy  kvm_hyper  kvm_inj_e  kvm_inj_v  kvm_invlp  kvm_ioapi   kvm_mmio  kvm_msi_s    kvm_msr  kvm_neste  kvm_neste  kvm_neste  kvm_neste  kvm_neste  kvm_page_  kvm_pic_s    kvm_pio  kvm_set_i  kvm_skini  kvm_try_a  kvm_users
        1         54         11          5          0          0          0          0          0          0          3         15         28         28         11          0          3          0          0          0          1          2          5          0          0          5          0          0          0          0          0          0          0          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          4          0          0          0          5          0          2         11          0          0          0          0          0          0          0          0          2          5          2          0          0          5

How to control the execution of profilers ?¶

Profiling in virt-test is controlled through configuration files. You can set the profilers that are going to run by setting the variable profilers. On tests_base.cfg.sample, the section of the file that sets the profilers that run by default looks like this:

# Profilers. You can add more autotest profilers (see list on client/profilers)
# to the line below. You can also choose to remove all profilers so no profiling
# will be done at all.
profilers = kvm_stat

How to add a profiler?¶

So, say you want to run the perf profiler in addition to kvm_stat. You can just edit that place and put ‘perf’ right next to it:

# Profilers. You can add more autotest profilers (see list on client/profilers)
# to the line below. You can also choose to remove all profilers so no profiling
# will be done at all.
profilers = kvm_stat perf

How to remove all profilers (including kvm_stat)?¶

If you want no profiling at all for your tests, profilers can be changed to be an empty string:

# Profilers. You can add more autotest profilers (see list on client/profilers)
# to the line below. You can also choose to remove all profilers so no profiling
# will be done at all.
profilers =

Of course, the config system makes it easy to override the value of any param for your test variable, so you can have fine grained control of things. Say you don’t want to run profilers on your new ‘crazy_test’ variant, which you have developed. Easy:

- crazy_test:
    type = crazy_test
    profilers =

So this will turn of profilers just for this particular test of yours.

Configuration¶

echo "-I FORWARD -m physdev --physdev-is-bridged -j ACCEPT" > /etc/sysconfig/iptables-forward-bridged
lokkit --custom-rules=ipv4:filter:/etc/sysconfig/iptables-forward-bridged
service libvirtd reload

ip_nic1 = 192.168.100.119
nic_mac_nic1 = 11:22:33:44:55:67
bridge = vbr

TestCases¶

mkdir -p /home/kermit/Downloads
cd /home/kermit/Downloads

wget http://people.redhat.com/mrodrigu/kvm/winutils.iso
mkdir original
sudo mount -o loop winutils.iso original
mkdir winutils

cp -r original/* winutils/

mkdir -p winutils/NTttcp

cd winutils/NTttcp
wget http://download.microsoft.com/download/f/1/e/f1e1ac7f-e632-48ea-83ac-56b016318735/NT%20Testing%20TCP%20Tool.msi -O "winutils/NTttcp/NT Testing TCP Tool.msi"
cp /usr/local/autotest/client/virt/scripts/ntttcp.au3 ./
sudo umount original

sudo mv winutils.iso winutils.iso.bak
mkisofs -o winutils.iso -max-iso9660-filenames -relaxed-filenames -D --input-charset iso8859-1 winutils

Performance subtests¶

Environment setup¶

Autotest already supports prepare environment for performance testing, guest/host need to be reboot for some configuration. setup script

Autotest supports to numa pining. Assign “numanode=-1” in tests.cfg, then vcpu threads/vhost_net threads/VM memory will be pined to last numa node. If you want to pin other processes to numa node, you can use numctl and taskset.

memory: numactl -m $n $cmdline
cpu: taskset $node_mask $thread_id

The following content is manual guide.

1.First level pinning would be to use numa pinning when starting the guest.
e.g  numactl -c 1 -m 1 qemu-kvm  -smp 2 -m 4G <> (pinning guest memory and cpus to numa-node 1)

2.For a single instance test, it would suggest trying a one to one mapping of vcpu to pyhsical core.
e.g
get guest vcpu threads id
#taskset -p 40 $vcpus1  (pinning vcpu1 thread to pyshical cpu #6 )
#taskset -p 80 $vcpus2  (pinning vcpu2 thread to physical cpu #7 )

3.To pin vhost on host. get vhost PID and then use taskset to pin it on the same soket.
e.g
taskset -p 20 $vhost (pinning vcpu2 thread to physical cpu #5 )

4.In guest,pin the IRQ to one core and the netperf to another.
1) make sure irqbalance is off - `service irqbalance stop`
2) find the interrupts - `cat /proc/interrupts`
3) find the affinity mask for the interrupt(s) - `cat /proc/irq/<irq#>/smp_affinity`
4) change the value to match the proper core.make sure the vlaue is cpu mask.
e.g pin the IRQ to first core.
   echo 01>/proc/irq/$virti0-input/smp_affinity
   echo 01>/proc/irq/$virti0-output/smp_affinity
5)pin the netserver to another core.
e.g
taskset -p 02 netserver

5.For host to guest scenario. to get maximum performance. make sure to run netperf on different cores on the same numa node as the guest.
e.g
numactl  -m 1 netperf -T 4 (pinning netperf to physical cpu #4)

Execute testing¶

• Submit jobs in Autotest server, only execute netperf.guset_exhost for three times.

tests.cfg:

only netperf.guest_exhost
variants:
    - repeat1:
    - repeat2:
    - repeat3:
# vbr0 has a static ip: 192.168.100.16
bridge=vbr0
# virbr0 is created by libvirtd, guest nic2 get ip by dhcp
bridge_nic2 = virbr0
# guest nic1 static ip
ip_nic1 = 192.168.100.21
# external host static ip:
client = 192.168.100.15

Result files:

# cd /usr/local/autotest/results/8-debug_user/192.168.122.1/
# find .|grep RHS
kvm.repeat1.r61.virtio_blk.smp2.virtio_net.RHEL.6.1.x86_64.netperf.exhost_guest/results/netperf-result.RHS
kvm.repeat2.r61.virtio_blk.smp2.virtio_net.RHEL.6.1.x86_64.netperf.exhost_guest/results/netperf-result.RHS
kvm.repeat3.r61.virtio_blk.smp2.virtio_net.RHEL.6.1.x86_64.netperf.exhost_guest/results/netperf-result.RHS

• Submit same job in another env (different packages) with same configuration

Result files:

# cd /usr/local/autotest/results/9-debug_user/192.168.122.1/
# find .|grep RHS
kvm.repeat1.r61.virtio_blk.smp2.virtio_net.RHEL.6.1.x86_64.netperf.exhost_guest/results/netperf-result.RHS
kvm.repeat2.r61.virtio_blk.smp2.virtio_net.RHEL.6.1.x86_64.netperf.exhost_guest/results/netperf-result.RHS
kvm.repeat3.r61.virtio_blk.smp2.virtio_net.RHEL.6.1.x86_64.netperf.exhost_guest/results/netperf-result.RHS

Analysis result¶

• Config file: perf.conf

[ntttcp]
result_file_pattern = .*.RHS
ignore_col = 1
avg_update =

[netperf]
result_file_pattern = .*.RHS
ignore_col = 2
avg_update = 4,2,3|14,5,12|15,6,13

[iozone]
result_file_pattern =

• Execute regression.py to compare two results:

login autotest server
# cd /usr/local/autotest/client/tools
# python regression.py netperf /usr/local/autotest/results/8-debug_user/192.168.122.1/ /usr/local/autotest/results/9-debug_user/192.168.122.1/

• T-test:

scipy: http://www.scipy.org/ t-test: http://en.wikipedia.org/wiki/Student’s_t-test Two python modules (scipy and numpy) are needed. Script to install numpy/scipy on rhel6 automatically: https://github.com/kongove/misc/blob/master/scripts/install-numpy-scipy.sh Unpaired T-test is used to compare two samples, user can check p-value to know if regression bug exists. If the difference of two samples is considered to be not statistically significant(p <= 0.05), it will add a ‘+’ or ‘-‘ before p-value. (‘+’: avg_sample1 < avg_sample2, ‘-‘: avg_sample1 > avg_sample2) “- only over 95% confidence results will be added “+/-” in “Significance” part. “+” for cpu-usage means regression, “+” for throughput means improvement.”

Regression results

netperf.exhost_guest.html fio.html - Every Avg line represents the average value based on $n repetitions of the same test, and the following SD line represents the Standard Deviation between the $n repetitions. - The Standard deviation is displayed as a percentage of the average. - The significance of the differences between the two averages is calculated using unpaired T-test that takes into account the SD of the averages. - The paired t-test is computed for the averages of same category. - only over 95% confidence results will be added “+/-” in “Significance” part. “+” for cpu-usage means regression, “+” for throughput means improvement.

Highlight HTML result o green/red –> good/bad o Significance is larger than 0.95 –> green dark green/red –> important (eg: cpu) light green/red –> other o test time o version (only when diff) o other: repeat time, title o user light green/red to highlight small (< %5) DIFF o highlight Significance with same color in one raw o add doc link to result file, and describe color in doc

netperf.avg.html - Raw data that the averages are based on.

Problem:¶

For multi-host tests multiple physical systems are often required. It is possible to use two virtual guests for most of autotest tests except for virt-tests (kvm, libvirt, ...).

However, It is possible to use Nested Virtualization to serve as first level (L0) guests and run nested guests inside them.

This page explains, how to setup (Fedora/RHEL) and use single computer with nested virtualization for such cases. Be careful that nested virtualization works usually right, but there are cases where it might lead to hidden problems. Do not use nested virtualization for production testing.

Nested Virtualization:¶

1. Emulated:

   • qemu very slow

2. hardware accelerated:

   • Hardware for the accelerated nested virtualization

     AMD Phenom and never core extension (smv, NPT) Intel Nehalem and never core extension (vmx, EPT)

   • Software which supports the accelerated nested virtualization

     kvm, xen, vmware, .... almost the same speed like native guest (1.0-0.1 of native quest performance). Performance depends on the type of load. IO load could be quite slow. Without vt-d or AMD-Vi and network device pass through.

Configuration for multi-host virt tests:¶

### Config of host system

• Intel CPU

  options kvm_intel nested=1 to the end of some modules config file /etc/modprobe.d/modules.conf

• AMD CPU

  options kvm_amd nested=1 to the end of some modules config file /etc/modprobe.d/modules.conf

### Config of Guest L0

• Intel CPU

  • Virtual manager

    Procesor config->CPU Features->vmx set to require

  • Native qemu-kvm

    qemu-kvm -cpu core2duo,+vmx -enable-kvm .....

• AMD CPU

  • Virtual manager

    Procesor config->CPU Features->svm set to require

  • Native qemu-kvm

    qemu-kvm -cpu qemu64,+svm -enable-kvm .....

### Config of Guest L0 System Connect to host bridge with guest L0 bridge without DHCP (dhcp collision with host system dhcp).

1. Destroy libvirt bridge which contain dhcp.
2. Enable network service systemctl enable network.service
3. Add new bridge virbr0 manually and insert to them L0 network adapter eth0 which is connected to host bridge

#interface connected to host system bridge
vi /etc/sysconfig/network-scripts/ifcfg-eth0
     NM_CONTROLLED="no"
     DEVICE="eth0"
     ONBOOT="yes"
     BRIDGE=virbr0

#Bridge has name virbr0 for compatibility with standard autotest settings.
vi /etc/sysconfig/network-scripts/ifcfg-virbr0
    DHCP_HOSTNAME="atest-guest"
    NM_CONTROLLED="no"
    BOOTPROTO="dhcp"
    ONBOOT="yes"
    IPV6INIT="no"
    DEVICE=virbr0
    TYPE=Bridge
    DELAY=0

and for sure disable NetworkManager systemctl disable NetworkManager.service

### Check Guest L0 System

modprobe kvm-intel or modprobe kvm-amd should work

Start for multi-host virt tests:¶

### Manually from host machine

cd autotest/client/tests/virt/qemu/ sudo rm -rf results.*; sudo ../../../../server/autoserv -m guestL0_1,guestL0_2 multi_host.srv

### From autotest gui

1. Start autotest frontend RPC or WEB interface https://github.com/autotest/autotest/wiki/SysAdmin
2. Select multi_host test from pull-request https://github.com/autotest/autotest-server-tests/pull/1

More details:¶

Set up/configuration, the root directory (of this git repo) also has simple scripts to create L1 and L2 guests. And reference of L1, L2 libvirt files are also added – https://github.com/kashyapc/nvmx-haswell/blob/master/SETUP-nVMX.rst

Running Multi Host Migration Tests¶

virt-test is our test suite, but for simplicity purposes it can only run on a single host. For multi host tests, you’ll need the full autotest + virt-test package, and the procedure is more complex. We’ll try to keep this procedure as objective as possible.

Prerequesites¶

This guide assumes that:

1. You have at least 2 virt capable machines that have shared storage setup in [insert specific path]. Let’s call them host1.foo.com and host2.foo.com.
2. You can ssh into both of those machines without a password (which means there is an SSH key setup with the account you’re going to use to run the tests) as root.
3. The machines should be able to communicate freely, so beware of the potential firewall complications. On each of those machines you need a specific NFS mount setup:

• /var/lib/virt_test/isos
• /var/lib/virt_test/steps_data
• /var/lib/virt_test/gpg

They all need to be backed by an NFS share read only. Why read only? Because it is safer, we exclude the chance to delete this important data by accident. Besides the data above is only needed in a read only fashion. fstab example:

myserver.foo.com:/virt-test/iso /var/lib/virt_test/isos nfs ro,nosuid,nodev,noatime,intr,hard,tcp 0 0
myserver.foo.com:/virt-test/steps_data  /var/lib/virt_test/steps_data nfs rw,nosuid,nodev,noatime,intr,hard,tcp 0 0
myserver.foo.com:/virt-test/gpg  /var/lib/virt_test/gpg nfs rw,nosuid,nodev,noatime,intr,hard,tcp 0 0

• /var/lib/virt_test/images
• /var/lib/virt_test/images_archive

Those all need to be backed by an NFS share read write (or any other shared storage you might have). This is necessary because both hosts need to see the same coherent storage. fstab example:

myserver.foo.com:/virt-test/images_archive  /var/lib/virt_test/images_archive nfs rw,nosuid,nodev,noatime,intr,hard,tcp 0 0
myserver.foo.com:/virt-test/images /var/lib/virt_test/images  nfs rw,nosuid,nodev,noatime,intr,hard,tcp 0 0

The images dir must be populated with the installed guests you want to run your tests on. They must match the file names used by guest OS in virt-test. For example, for RHEL 6.4, the image name virt-test uses is:

rhel64-64.qcow2

double check your files are there:

$ ls /var/lib/virt_test/images
$ rhel64-64.qcow2

Setup step by step¶

First, clone the autotest repo recursively. It’s a repo with lots of submodules, so you’ll see a lot of output:

$ git clone --recursive https://github.com/autotest/autotest.git
... lots of output ...

Then, edit the global_config.ini file, and change the key:

serve_packages_from_autoserv: True

to:

serve_packages_from_autoserv: False

Then you need to update virt-test’s config files and sub tests (that live in separate repositories that are not git submodules). You don’t need to download the JeOS file in this step, so simply answer ‘n’ to the quest

Note: The bootstrap procedure described below will be performed automatically upon running the autoserv command that triggers the test. The problem is that then you will not be able to see the config files and modify filters prior to actually running the test. Therefore this documentation will instruct you to run the steps below manually.

$ export AUTOTEST_PATH=.;client/tests/virt/run -t qemu --bootstrap --update-providers
16:11:14 INFO | qemu test config helper
16:11:14 INFO |
16:11:14 INFO | 1 - Updating all test providers
16:11:14 INFO | Fetching git [REP 'git://github.com/autotest/tp-qemu.git' BRANCH 'master'] -> /var/tmp/autotest/client/tests/virt/test-providers.d/downloads/io-github-autotest-qemu
16:11:17 INFO | git commit ID is 6046958afa1ccab7f22bb1a1a73347d9c6ed3211 (no tag found)
16:11:17 INFO | Fetching git [REP 'git://github.com/autotest/tp-libvirt.git' BRANCH 'master'] -> /var/tmp/autotest/client/tests/virt/test-providers.d/downloads/io-github-autotest-libvirt
16:11:19 INFO | git commit ID is edc07c0c4346f9029930b062c573ff6f5433bc53 (no tag found)
16:11:20 INFO |
16:11:20 INFO | 2 - Checking the mandatory programs and headers
16:11:20 INFO | /usr/bin/7za
16:11:20 INFO | /usr/sbin/tcpdump
16:11:20 INFO | /usr/bin/nc
16:11:20 INFO | /sbin/ip
16:11:20 INFO | /sbin/arping
16:11:20 INFO | /usr/bin/gcc
16:11:20 INFO | /usr/include/bits/unistd.h
16:11:20 INFO | /usr/include/bits/socket.h
16:11:20 INFO | /usr/include/bits/types.h
16:11:20 INFO | /usr/include/python2.6/Python.h
16:11:20 INFO |
16:11:20 INFO | 3 - Checking the recommended programs
16:11:20 INFO | Recommended command missing. You may want to install it if not building it from source. Aliases searched: ('qemu-kvm', 'kvm')
16:11:20 INFO | Recommended command qemu-img missing. You may want to install it if not building from source.
16:11:20 INFO | Recommended command qemu-io missing. You may want to install it if not building from source.
16:11:20 INFO |
16:11:20 INFO | 4 - Verifying directories
16:11:20 INFO |
16:11:20 INFO | 5 - Generating config set
16:11:20 INFO |
16:11:20 INFO | 6 - Verifying (and possibly downloading) guest image
16:11:20 INFO | File JeOS 19 x86_64 not present. Do you want to download it? (y/n) n
16:11:30 INFO |
16:11:30 INFO | 7 - Checking for modules kvm, kvm-amd
16:11:30 WARNI| Module kvm is not loaded. You might want to load it
16:11:30 WARNI| Module kvm-amd is not loaded. You might want to load it
16:11:30 INFO |
16:11:30 INFO | 8 - If you wish, take a look at the online docs for more info
16:11:30 INFO |
16:11:30 INFO | https://github.com/autotest/virt-test/wiki/GetStarted

Then you need to copy the multihost config file to the appropriate place:

cp client/tests/virt/test-providers.d/downloads/io-github-autotest-qemu/qemu/cfg/multi-host-tests.cfg client/tests/virt/backends/qemu/cfg/

Now, edit the file:

server/tests/multihost_migration/control.srv

In there, you have to change the EXTRA_PARAMS to restrict the number of guests you want to run the tests on. On this example, we’re going to restrict our tests to RHEL 6.4. The particular section of the control file should look like:

EXTRA_PARAMS = """
only RHEL.6.4.x86_64
"""

It is important to stress that the guests must be installed for this to work smoothly. Then the last step would be to run the tests. Using the same convention for the machine hostnames, here’s the command you should use:

server/autotest-remote -m host1.foo.com,host2.foo.com server/tests/multihost_migration/control.srv

Now, you’ll see a boatload of output from the autotest remote output. This is normal, and you should be patient until all the tests are done.

Writing Multi Host Migration tests¶

Scheme:¶

Source file for the diagram above (LibreOffice file)

Example:¶

class TestMultihostMigration(virt_utils.MultihostMigration):
    def __init__(self, test, params, env):
        super(testMultihostMigration, self).__init__(test, params, env)

    def migration_scenario(self):
        srchost = self.params.get("hosts")[0]
        dsthost = self.params.get("hosts")[1]

        def worker(mig_data):
            vm = env.get_vm("vm1")
            session = vm.wait_for_login(timeout=self.login_timeout)
            session.sendline("nohup dd if=/dev/zero of=/dev/null &")
            session.cmd("killall -0 dd")

        def check_worker(mig_data):
            vm = env.get_vm("vm1")
            session = vm.wait_for_login(timeout=self.login_timeout)
            session.cmd("killall -9 dd")

        # Almost synchronized migration, waiting to end it.
        # Work is started only on first VM.

        self.migrate_wait(["vm1", "vm2"], srchost, dsthost,
                          worker, check_worker)

        # Migration started in different threads.
        # It allows to start multiple migrations simultaneously.

        # Starts one migration without synchronization with work.
        mig1 = self.migrate(["vm1"], srchost, dsthost,
                            worker, check_worker)

        time.sleep(20)

        # Starts another test simultaneously.
        mig2 = self.migrate(["vm2"], srchost, dsthost)
        # Wait for mig2 finish.
        mig2.join()
        mig1.join()

mig = TestMultihostMigration(test, params, env)
# Start test.
mig.run()

When you call:

mig = TestMultihostMigration(test, params, env):

What happens is

1. VM’s disks will be prepared.
2. The synchronization server will be started.
3. All hosts will be synchronized after VM create disks.

When you call the method:

migrate():

What happens in a diagram is:

source	destination
It prepare VM if machine is not started.
Start work on VM.
mig.migrate_vms_src()	mig.migrate_vms_dest()
	Check work on VM after migration.
Wait for finish migration on all hosts.

It’s important to note that the migrations are made using the tcp protocol, since the others don’t support multi host migration.

def migrate_vms_src(self, mig_data):
    vm = mig_data.vms[0]
    logging.info("Start migrating now...")
    vm.migrate(mig_data.dst, mig_data.vm_ports)

This example migrates only the first machine defined in migration. Better example is in virt_utils.MultihostMigration.migrate_vms_src. This function migrates all machines defined for migration.

How to use in virt-test¶

You can use virt-test to test GlusterFS support with following steps.

1. Edit qemu/cfg/tests.cfg with following changes,

only glusterfs_support
remove ‘only no_glusterfs_support’ line from the file

2) Optionally, edit shared/cfg/guest-hw.cfg for the gluster volume name and brick path, default is going to be,

gluster_volume_name = test-vol
gluster_brick = /tmp/gluster

How to use manually¶

The following is just an example to show how we create gluster volume and run a guest on that volume manually.

Starting Gluster daemon¶

service glusterd start

Gluster volume creation¶

gluster volume create [volume-name]  [hostname/host_ip]:/[brick_path]

E:g: gluster volume create test-vol satheesh.ibm.com://home/satheesh/images_gluster

Qemu Img creation¶

qemu-img create gluster://[hostname]:0/[volume-name]/[image-name] [size]

E:g: qemu-img create gluster://satheesh.ibm.com:0/test-vol/test_gluster.img 10G

Example of qemu cmd Line¶

qemu-system-x86_64 --enable-kvm -smp 4 -m 2048 -drive file=gluster://satheesh.ibm.com/test-vol/test_gluster.img,if=virtio -net nic,macaddr=52:54:00:09:0a:0b -net tap,script=/path/to/qemu-ifupVirsh

Some conventions¶

We are assuming you will install autotest to its default upstream location

/usr/local/autotest

Therefore a lot of paths referred here will have this as the base dir.

CLI vs Web UI¶

During this text, we’ll use frequently the terms CLI and Web UI.

By CLI we mean specifically the program:

/usr/local/autotest/cli/autotest-rpc-client

That is located in the autotest code checkout.

By Web UI, we mean the web interface of autotest, that can be accessed through

http://your-autotest-server.com/afe

Step 1 - Install an autotest server¶

Provided that you have internet on your test lab, this should be the easiest step. Pick up either a VM accessible in your lab, or a bare metal machine (it really doesn’t make a difference, we use a VM here). We’ll refer it from now on as the “Server” box.

The hard drive of the Server should hold enough room for test results. We found out that at least 250 GB holds data for more than 6 months, provided that QEMU doesn’t crash a lot.

You’ll follow the procedure described on

https://github.com/autotest/autotest/wiki/AutotestServerInstallRedHat

for Red Hat derivatives (such as Fedora and RHEL), and

https://github.com/autotest/autotest/wiki/AutotestServerInstall

for Debian derivatives (Debian, Ubuntu).

Note that using the install script referred right in the beginning of the documentation is the preferred method, and should work pretty well if you have internet on your lab. In case you don’t have internet there, you’d need to follow the instructions after the ‘installing with the script’ instructions. Let us know if you have any problems.

Step 2 - Add test machines¶

It should go without saying, but the machines you have to add have to be virtualization capable (support KVM).

You can add machines either by using the CLI or the Web UI, following the documentation:

https://github.com/autotest/autotest/wiki/ConfiguringHosts

If you don’t want to read that, I’ll try to write a quick howto.

Say you have two x86_64 hosts, one AMD and the other, Intel. Their hostnames are:

foo-amd.bazcorp.com foo-intel.bazcorp.com

I would create 2 labels, amd64 and intel64, I would also create a label to indicate the machines can be provisioned by cobbler. This is because you can tell autotest to run a job in any machine that matches a given label.

Logged as the autotest user:

$ /usr/local/autotest/cli/autotest-rpc-client label create -t amd64
Created label:
    'amd64'
$ /usr/local/autotest/cli/autotest-rpc-client label create -t intel64
Created label:
    'intel64'
$ /usr/local/autotest/cli/autotest-rpc-client label create hostprovisioning
Created label:
    'hostprovisioning'

Then I’d create each machine with the appropriate labels

$ /usr/local/autotest/cli/autotest-rpc-client host create -t amd64 -b hostprovisioning foo-amd.bazcorp.com
Added host:
    foo-amd.bazcorp.com

$ /usr/local/autotest/cli/autotest-rpc-client host create -t amd64 -b hostprovisioning foo-intel.bazcorp.com
Added host:
    foo-amd.bazcorp.com

Step 3 - Prepare the test jobs¶

Now you have to copy the plugin we have developed to extend the CLI to parse additional information for the virt jobs:

cp /usr/local/autotest/contrib/virt/site_job.py /usr/local/autotest/cli/

This should be enough to enable all the extra functionality.

You also need to copy the site-config.cfg file that we published as a reference, to the qemu config module:

cp /usr/local/autotest/contrib/virt/site-config.cfg /usr/local/autotest/client/tests/virt/qemu/cfg

Be aware that you need to read this file well, and later, configure it to your testing needs. We specially stress that you might want to create private git mirrors of the git repos you want to test, so you tax the upstream mirrors less, and have increased reliability.

Right now it is able to run regression testing on Fedora 18, and upstream kvm, provided that you have a cobbler instance functional, with a profile called f18-autotest-kvm that can be properly installed on your machines. Having that properly set up may open another can of worms.

One simple way to schedule the jobs, that we does use at our server, is to use cron to schedule daily testing jobs of the things you want to test. Here is an example that should work ‘out of the box’. Provided that you have an internal mailing list that you created with the purpose of receiving email notifications, called autotest-virt-jobs@foocorp.com, you can stick that on the crontab of the user autotest in the Server:

07 00 * * 1-7 /usr/local/autotest/cli/autotest-rpc-client job create -B never -a never -s -e autotest-virt-jobs@foocorp.com -f "/usr/local/autotest/contrib/virt/control.template" -T --timestamp -m '1*hostprovisioning' -x 'only qemu-git..sanity' "Upstream qemu.git sanity"
15 00 * * 1-7 /usr/local/autotest/cli/autotest-rpc-client job create -B never -a never -s -e autotest-virt-jobs@foocorp.com -f "/usr/local/autotest/contrib/virt/control.template" -T --timestamp -m '1*hostprovisioning' -x 'only f18..sanity' "Fedora 18 koji sanity"
07 01 * * 1-7 /usr/local/autotest/cli/autotest-rpc-client job create -B never -a never -s -e autotest-virt-jobs@foocorp.com -f "/usr/local/autotest/contrib/virt/control.template" -T --timestamp -m '1*hostprovisioning' -x 'only qemu-git..stable' "Upstream qemu.git stable"
15 01 * * 1-7 /usr/local/autotest/cli/autotest-rpc-client job create -B never -a never -s -e autotest-virt-jobs@foocorp.com -f "/usr/local/autotest/contrib/virt/control.template" -T --timestamp -m '1*hostprovisioning' -x 'only f18..stable' "Fedora 18 koji stable"

That should be enough to have one sanity and stable job for:

• Fedora 18.
• qemu.git userspace and kvm.git kernel.

What does these ‘stable’ and ‘sanity’ jobs do? In short:

• Host OS (Fedora 18) installation through cobbler
• Latest kernel for the Host OS installation (either the last kernel update build for fedora, or check out, compile and install kvm.git).

sanity job¶

• Install latest Fedora 18 qemu-kvm, or compiles the latest qemu.git
• Installs a VM with Fedora 18, boots, reboots, does simple, single host migration with all supported protocols
• Takes about two hours. In fact, internally we test more guests, but they are not widely available (RHEL 6 and Windows 7), so we just replaced them with Fedora 18.

stable job¶

• Same as above, but many more networking, timedrift and other tests

Setup cobbler to install hosts¶

Cobbler is an installation server, that control DHCP and/or PXE boot for your x86_64 bare metal virtualization hosts. You can learn how to set it up in the following resource:

https://github.com/cobbler/cobbler/wiki/Start%20Here

You will set it up for simple installations, and you probably just need to import a Fedora 18 DVD into it, so it can be used to install your hosts. Following the import procedure, you’ll have a ‘profile’ created, which is a label that describes an OS that can be installed on your virtualization host. The label we chose, as already mentioned is f18-autotest-kvm. If you want to change that name, you’ll have to change site-config.cfg accordingly.

Also, you will have to add your test machines to your cobbler server, and will have to set up remote control (power on/off) for them.

The following is important:

The hostname of your machine in the autotest server has to be the name of your system in cobbler.

So, for the hypothetical example you’ll have to have set up systems with names foo-amd.bazcorp.com foo-intel.bazcorp.com in cobbler. That’s right, the ‘name’ of the system has to be the ‘hostname’. Otherwise, autotest will ask cobbler and cobbler will not know which machine autotest is taking about.

Other assumptions we have here:

1) We have a (read only, to avoid people deleting isos by mistake) NFS share that has the Fedora 18 DVD and other ISOS. The structure for the base dir could look something like:

.
|-- linux
|   `-- Fedora-18-x86_64-DVD.iso
`-- windows
    |-- en_windows_7_ultimate_x64_dvd_x15-65922.iso
    |-- virtio-win.iso
    `-- winutils.iso

This is just in case you are legally entitled to download and use Windows 7, for example.

2) We have another NFS share with space for backups of qcow2 images that got corrupted during testing, and you want people to analyze them. The structure would be:

.
|-- foo-amd
`-- bar-amd

That is, one directory for each host machine you have on your grid. Make sure they end up being properly configured in the kickstart.

Now here is one excerpt of kickstart with some of the gotchas we learned with experience. Some notes:

• This is not a fully formed, functional kickstart, just in case you didn’t notice.
• This is provided in the hopes you read it, understand it and adapt things to your needs. If you paste this into your kickstart and tell me it doesn’t work, I WILL silently ignore your email, and if you insist, your emails will be filtered out and go to the trash can.

install

text
reboot
lang en_US
keyboard us
rootpw --iscrypted [your-password]
firewall --disabled
selinux --disabled
timezone --utc America/New_York
firstboot --disable
services --enabled network --disabled NetworkManager
bootloader --location=mbr
ignoredisk --only-use=sda
zerombr
clearpart --all --drives=sda --initlabel
autopart
network --bootproto=dhcp --device=eth0 --onboot=on

%packages
@core
@development-libs
@development-tools
@virtualization
wget
dnsmasq
genisoimage
python-imaging
qemu-kvm-tools
gdb
iasl
libvirt
python-devel
ntpdate
gstreamer-plugins-good
gstreamer-python
dmidecode
popt-devel
libblkid-devel
pixman-devel
mtools
koji
tcpdump
bridge-utils
dosfstools
%end

%post

echo "[nfs-server-that-holds-iso-images]:[nfs-server-that-holds-iso-images]/base_path/iso /var/lib/virt_test/isos nfs ro,defaults 0 0" >> /etc/fstab
echo "[nfs-server-that-holds-iso-images]:[nfs-server-that-holds-iso-images]/base_path/steps_data  /var/lib/virt_test/steps_data nfs rw,nosuid,nodev,noatime,intr,hard,tcp 0 0" >> /etc/fstab
echo "[nfs-server-that-has-lots-of-space-for-backups]:/base_path/[dir-that-holds-this-hostname-backups] /var/lib/virt_test/images_archive nfs rw,nosuid,nodev,noatime,intr,hard,tcp 0 0" >> /etc/fstab
mkdir -p /var/lib/virt_test/isos
mkdir -p /var/lib/virt_test/steps_data
mkdir -p /var/lib/virt_test/images
mkdir -p /var/lib/virt_test/images_archive

mkdir --mode=700 /root/.ssh
echo 'ssh-dss [the-ssh-key-of-the-Server-autotest-user]' >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys

ntpdate [your-ntp-server]
hwclock --systohc

systemctl mask tmp.mount
%end

Painful trial and error process to adjust details¶

After all that, you can start running your test jobs and see what things will need to be fixed. You can run your jobs easily by logging into your Server, with the autotest user, and use the command:

/usr/local/autotest/cli/autotest-rpc-client job create -B never -a never -s -e autotest-virt-jobs@foocorp.com -f "/usr/local/autotest/contrib/virt/control.template" -T --timestamp -m '1*hostprovisioning' -x 'only f18..sanity' "Fedora 18 koji sanity"

As you might have guessed, this will schedule a Fedora 18 sanity job. So go through it and fix things step by step. If anything, you can take a look at this:

https://github.com/autotest/autotest/wiki/DiagnosingFailures

And see if it helps. You can also ask on the mailing list, but please, pretty please do your homework before you ask us to guide you through all the process step by step. This is already a step by step procedure.

All right, good luck, and happy testing!

A bit of context on windows virtio drivers install¶

This method of install so far covers the storage (viostor) and network (NetKVM) drivers. virt-test uses a boot floppy with a Windows answer file in order to perform unattended install of windows guests. For winXP and win2003, the unattended files are simple .ini files, while for win2008 and later, the unattended files are XML files.

In order to install the virtio drivers during guest install, KVM autotest has to inform the windows install programs *where* to find the drivers. So, we work from the following assumptions:

1. You already have an iso file that contains windows virtio drivers (inf files) for both netkvm and viostor. If you are unsure how to generate that iso, there’s an example script under contrib, inside the kvm test directory. Here is an example of how the files inside this cd would be organized, assuming the iso image is mounted under /tmp/virtio-win (the actual cd has more files, but we took only the parts that concern to the example, win7 64 bits).

/tmp/virtio-win/
/tmp/virtio-win/vista
/tmp/virtio-win/vista/amd64
/tmp/virtio-win/vista/amd64/netkvm.cat
/tmp/virtio-win/vista/amd64/netkvm.inf
/tmp/virtio-win/vista/amd64/netkvm.pdb
/tmp/virtio-win/vista/amd64/netkvm.sys
/tmp/virtio-win/vista/amd64/netkvmco.dll
/tmp/virtio-win/vista/amd64/readme.doc
/tmp/virtio-win/win7
/tmp/virtio-win/win7/amd64
/tmp/virtio-win/win7/amd64/balloon.cat
/tmp/virtio-win/win7/amd64/balloon.inf
/tmp/virtio-win/win7/amd64/balloon.pdb
/tmp/virtio-win/win7/amd64/balloon.sys
/tmp/virtio-win/win7/amd64/blnsvr.exe
/tmp/virtio-win/win7/amd64/blnsvr.pdb
/tmp/virtio-win/win7/amd64/vioser.cat
/tmp/virtio-win/win7/amd64/vioser.inf
/tmp/virtio-win/win7/amd64/vioser.pdb
/tmp/virtio-win/win7/amd64/vioser.sys
/tmp/virtio-win/win7/amd64/vioser-test.exe
/tmp/virtio-win/win7/amd64/vioser-test.pdb
/tmp/virtio-win/win7/amd64/viostor.cat
/tmp/virtio-win/win7/amd64/viostor.inf
/tmp/virtio-win/win7/amd64/viostor.pdb
/tmp/virtio-win/win7/amd64/viostor.sys
/tmp/virtio-win/win7/amd64/wdfcoinstaller01009.dll
...

If you are planning on installing WinXP or Win2003, you should also have a pre-made floppy disk image with the virtio drivers *and* a configuration file that the installer program will read to fetch the right drivers from it. Unfortunately, I don’t have much info on how to build that file, you probably would have the image already assembled if you are willing to test those guest OS.

So you have to map the paths of your cd containing the drivers on the config variables. We hope to improve this in cooperation with the virtio drivers team.

Step by step procedure¶

We are assuming you already have the virtio cd properly assembled with you, as well as windows iso files that *do match the ones provided in our test_base.cfg.sample*. Don’t worry though, we try as much as possible to use files from MSDN, to standardize.

We will use win7 64 bits (non sp1) as the example, so the CD you’d need is:

cdrom_cd1 = isos/windows/en_windows_7_ultimate_x86_dvd_x15-65921.iso
sha1sum_cd1 = 5395dc4b38f7bdb1e005ff414deedfdb16dbf610

This file can be downloaded from the MSDN site, so you can verify the SHA1 sum of it matches.

1. Git clone autotest to a convenient location, say $HOME/Code/autotest. See the download source documentation Please do use git and clone the repo to the location mentioned.

2. Execute the get_started.py script (see the get started documentation <../basic/GetStarted>`. It will create the directories where we expect the cd files to be available. You don’t need to download the Fedora 14 DVD, but you do need to download the winutils.iso cd (on the example below, I have skipped the download because I do have the file, so I can copy it to the expected location, which is in this case /tmp/kvm_autotest_root/isos/windows). Please, do read the documentation mentioned on the script to avoid missing packages installed and other misconfiguration.

3. Create a windows dir under /tmp/kvm_autotest_root/isos

4. Copy your windows 7 iso to /tmp/kvm_autotest_root/isos/windows

5. Edit the file cdkeys.cfg and put the windows 7 64 bit key on that file

6. Edit the file win-virtio.cfg and verify if the paths are correct. You can see that by looking this session:

   64:
       unattended_install.cdrom, whql.support_vm_install:
           # Look at your cd structure and see where the drivers are
           # actually located (viostor and netkvm)
           virtio_storage_path = 'F:\win7\amd64'
           virtio_network_path = 'F:\vista\amd64'
   
           # Uncomment if you have a nw driver installer on the iso
           #virtio_network_installer_path = 'F:\RHEV-Network64.msi'
   

7. If you are using the cd with the layout mentioned on the beginning of this article, the paths are already correct. However, if they’re different (more likely), you have to adjust paths. Don’t forget to read and do all the config on win-virtio.cfg file as instructed by the comments.

8. On tests.cfg, you have to enable virtio install of windows 7. On the block below, you have to change only rtl8139 to only virtio_net and only ide to only virtio-blk. You are informing autotest that you only want a vm with virtio hard disk and network device installed.

   # Runs qemu-kvm, Windows Vista 64 bit guest OS, install, boot, shutdown
   - @qemu_kvm_windows_quick:
       # We want qemu-kvm for this run
       qemu_binary = /usr/bin/qemu-kvm
       qemu_img_binary = /usr/bin/qemu-img
       # Only qcow2 file format
       only qcow2
       # Only rtl8139 for nw card (default on qemu-kvm)
       only rtl8139
       # Only ide hard drives
       only ide
       # qemu-kvm will start only with -smp 2 (2 processors)
       only smp2
       # No PCI assignable devices
       only no_pci_assignable
       # No large memory pages
       only smallpages
       # Operating system choice
       only Win7.64
       # Subtest choice. You can modify that line to add more subtests
       only unattended_install.cdrom, boot, shutdown
   

9. You have to change the bottom of tests.cfg to look like the below, Which means you are informing autotest to only run the test set mentioned above, rather than the default, that installs Fedora 15.

   only qemu_kvm_windows_quick
   

10. As informed on the output of get_started.py, the command you can execute to run autotest is (please run this AS ROOT or sudo)

    $HOME/Code/autotest/client/bin/autotest $HOME/Code/autotest/client/tests/kvm/control
    

11. Profit! You automated install of Windows 7 with the virtio drivers will be carried out.

If you want to install other guests, as you might imagine, you can change only Win7.64 with other guests, say only Win2008.64.sp2. Now, during the first time you perform your installs, it’s good to watch the installation to see if there aren’t problems such as a wrong cd key preventing your install from happening. virt-test prints the qemu command line used, so you can see which vnc display you can connect to to watch your vm being installed.

Please give us feedback on whether this procedure was helpful - email me at lmr AT redhat DOT com.

A bit of context on virt-test build tests¶

People usually don’t know that virt-test has support to build and install QEMU/KVM for testing purposes, from many different software sources. You can:

1. Build qemu-kvm from a git repo (most common choice for developers hacking on code)
2. Install qemu-kvm from an rpm file (people testing a newly built rpm package)
3. Install qemu-kvm straight from the Red Hat build systems (Koji is the instance of the build system for Fedora, Brew is the same, but for RHEL. With this we can perform quality control on both Fedora and RHEL packages, trying to anticipate breakages before the packages hit users)

For this article, we are going to focus on git based builds. Also, we are focusing on Fedora and RHEL. We’ll try to write the article in a pretty generic fashion, you are welcome to improve this with details on how to do the same on your favorite linux distribution.

Before you start¶

You need to verify that you can actually build qemu-kvm from source, as well as the unittest suite.

1. Make sure you have the appropriate packages installed. You can read the install prerequesite packages (client) section for more information.

Step by step procedure¶

1. Git clone autotest to a convenient location, say $HOME/Code/autotest. See the download source documentation Please do use git and clone the repo to the location mentioned.

2. Execute the get_started.py script (see the get started documentation. If you just want to run unittests, you can safely skip each and every iso download possible, as qemu-kvm will straight boot small kernel images (the unittests) rather than full blown OS installs.

3. As running unittests is something that’s fairly independent of other virt-test testing you can do, and it’s something people are interested in, we prepared a special control file and a special configuration file for it. On the kvm directory, you can see the files unittests.cfg control.unittests. You only need to edit unittests.cfg.

4. The file unittests.cfg is a stand alone configuration for running unittests. It is comprised by a build variant and a unittests variant. Edit the file, it’ll look like:

   ... bunch of params needed for the virt-test preprocessor
   # Tests
   variants:
       - build:
           type = build
           vms = ''
           start_vm = no
           # Load modules built/installed by the build test?
           load_modules = no
           # Save the results of this build on test.resultsdir?
           save_results = no
           variants:
               - git:
                   mode = git
                   user_git_repo = git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
                   user_branch = next
                   user_lbranch = next
                   test_git_repo = git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
   
       - unittest:
           type = unittest
           vms = ''
           start_vm = no
           unittest_timeout = 600
           testdev = yes
           extra_params += " -S"
           # In case you want to execute only a subset of the tests defined on the
           # unittests.cfg file on qemu-kvm, uncomment and edit test_list
           #test_list = idt_test hypercall vmexit realmode
   
   only build.git unittest
   

5. As you can see above, you have places to specify both the userspace git repo and the unittest git repo. You are then free to replace user_git_repo with your own git repo. It can be a remote git location, or it can simply be the path to a cloned tree inside your development machine.

6. As of Fedora 15, that ships with gcc 4.6.0, the compilation is more strict, so things such as an unused variable in the code *will* lead to a build failure. You can disable that level of strictness by providing extra configure script options to your qemu-kvm userspace build. Right below the user_git_repo line, you can set the variable extra_configure_options to include --disable-werror. Let’s say you also want virt-test to fetch from my local tree, /home/lmr/Code/qemu-kvm, master branch, same for the kvm-unit-tests repo. If you make those changes, your build variant will look like:

   - git:
       mode = git
       user_git_repo = /home/lmr/Code/qemu-kvm
       extra_configure_options = --disable-werror
       user_branch = master
       user_lbranch = master
       test_git_repo = /home/lmr/Code/kvm-unit-tests
   

7. Now you can just run virt-test as usual, you just have to change the main control file (called control with the unittest one control.unittests

   $HOME/Code/autotest/client/bin/autotest $HOME/Code/autotest/client/tests/kvm/control.unittests
   

8. The output of a typical unittest execution looks like. Notice that autotest informs you where the logs of each individual unittests are located, so you can check that out as well.

   07/14 18:49:44 INFO |  unittest:0052| Running apic
   07/14 18:49:44 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/apic.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S -cpu qemu64,+x2apic
   07/14 18:49:46 INFO |  unittest:0096| Waiting for unittest apic to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 18:59:46 ERROR|  unittest:0108| Exception happened during apic: Timeout elapsed (600s)
   07/14 18:59:46 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/apic.log
   07/14 18:59:46 INFO |  unittest:0052| Running smptest
   07/14 19:00:15 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:00:16 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/smptest.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:00:17 INFO |  unittest:0096| Waiting for unittest smptest to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:00:17 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:00:18 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/smptest.log
   07/14 19:00:18 INFO |  unittest:0052| Running smptest3
   07/14 19:00:18 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 3 -kernel '/usr/local/autotest/tests/kvm/unittests/smptest.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:00:19 INFO |  unittest:0096| Waiting for unittest smptest3 to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:00:19 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:00:20 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/smptest3.log
   07/14 19:00:20 INFO |  unittest:0052| Running vmexit
   07/14 19:00:20 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/vmexit.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:00:21 INFO |  unittest:0096| Waiting for unittest vmexit to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:00:31 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:00:31 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/vmexit.log
   07/14 19:00:31 INFO |  unittest:0052| Running access
   07/14 19:00:31 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/access.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:00:32 INFO |  unittest:0096| Waiting for unittest access to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:01:02 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:01:03 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/access.log
   07/14 19:01:03 INFO |  unittest:0052| Running emulator
   07/14 19:01:03 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/emulator.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:01:05 INFO |  unittest:0096| Waiting for unittest emulator to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:01:06 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:01:07 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/emulator.log
   07/14 19:01:07 INFO |  unittest:0052| Running hypercall
   07/14 19:01:07 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/hypercall.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:01:08 INFO |  unittest:0096| Waiting for unittest hypercall to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:01:08 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:01:09 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/hypercall.log
   07/14 19:01:09 INFO |  unittest:0052| Running idt_test
   07/14 19:01:09 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/idt_test.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:01:10 INFO |  unittest:0096| Waiting for unittest idt_test to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:01:10 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:01:11 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/idt_test.log
   07/14 19:01:11 INFO |  unittest:0052| Running msr
   07/14 19:01:11 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/msr.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:01:12 INFO |  unittest:0096| Waiting for unittest msr to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:01:13 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:01:13 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/msr.log
   07/14 19:01:13 INFO |  unittest:0052| Running port80
   07/14 19:01:13 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/port80.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:01:14 INFO |  unittest:0096| Waiting for unittest port80 to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:01:31 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:01:32 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/port80.log
   07/14 19:01:32 INFO |  unittest:0052| Running realmode
   07/14 19:01:32 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/realmode.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:01:33 INFO |  unittest:0096| Waiting for unittest realmode to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:01:33 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:01:34 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/realmode.log
   07/14 19:01:34 INFO |  unittest:0052| Running sieve
   07/14 19:01:34 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/sieve.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:01:35 INFO |  unittest:0096| Waiting for unittest sieve to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:02:05 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:02:05 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/sieve.log
   07/14 19:02:05 INFO |  unittest:0052| Running tsc
   07/14 19:02:05 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/tsc.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:02:06 INFO |  unittest:0096| Waiting for unittest tsc to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:02:06 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:02:07 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/tsc.log
   07/14 19:02:07 INFO |  unittest:0052| Running xsave
   07/14 19:02:07 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/xsave.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:02:08 INFO |  unittest:0096| Waiting for unittest xsave to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:02:09 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:02:09 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/xsave.log
   07/14 19:02:09 INFO |  unittest:0052| Running rmap_chain
   07/14 19:02:09 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/rmap_chain.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S
   07/14 19:02:11 INFO |  unittest:0096| Waiting for unittest rmap_chain to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:02:12 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:02:13 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/rmap_chain.log
   07/14 19:02:13 INFO |  unittest:0052| Running svm
   07/14 19:02:13 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/svm.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S -enable-nesting -cpu qemu64,+svm
   07/14 19:02:13 INFO |   aexpect:0783| (qemu) qemu: -enable-nesting: invalid option
   07/14 19:02:13 INFO |   aexpect:0783| (qemu) (Process terminated with status 1)
   07/14 19:02:13 ERROR|  unittest:0108| Exception happened during svm: VM creation command failed:    "/usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/svm.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S -enable-nesting -cpu qemu64,+svm"    (status: 1,    output: 'qemu: -enable-nesting: invalid option\n')
   07/14 19:02:13 ERROR|  unittest:0115| Not possible to collect logs
   07/14 19:02:13 INFO |  unittest:0052| Running svm-disabled
   07/14 19:02:13 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/svm.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S -cpu qemu64,-svm
   07/14 19:02:14 INFO |  unittest:0096| Waiting for unittest svm-disabled to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:02:15 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:02:16 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/svm-disabled.log
   07/14 19:02:16 INFO |  unittest:0052| Running kvmclock_test
   07/14 19:02:16 INFO |    kvm_vm:0782| Running qemu command:
   /usr/local/autotest/tests/kvm/qemu -name 'vm1' -nodefaults -vga std -monitor unix:'/tmp/monitor-humanmonitor1-20110714-184944-6ms0',server,nowait -qmp unix:'/tmp/monitor-qmpmonitor1-20110714-184944-6ms0',server,nowait -serial unix:'/tmp/serial-20110714-184944-6ms0',server,nowait -m 512 -smp 2 -kernel '/usr/local/autotest/tests/kvm/unittests/kvmclock_test.flat' -vnc :0 -chardev file,id=testlog,path=/tmp/testlog-20110714-184944-6ms0 -device testdev,chardev=testlog  -S --append "10000000 `date +%s`"
   07/14 19:02:17 INFO |  unittest:0096| Waiting for unittest kvmclock_test to complete, timeout 600, output in /tmp/testlog-20110714-184944-6ms0
   07/14 19:02:33 INFO |   aexpect:0783| (qemu) (Process terminated with status 0)
   07/14 19:02:34 INFO |  unittest:0113| Unit test log collected and available under /usr/local/autotest/results/default/kvm.qemu-kvm-git.unittests/debug/kvmclock_test.log
   07/14 19:02:34 ERROR|       kvm:0094| Test failed: TestFail: Unit tests failed: apic svm
   

You might take a look at the unittests.cfg config file options to do some tweaking you might like, such as making the timeout to consider a unittest as failed smaller and other things.

Please give us feedback on whether this procedure was helpful - email me at lmr AT redhat DOT com.

Subpackages¶