Integration testing patches
Tests added:
- Armbian 20.08 on Orange Pi PC (Philippe)
- MPC8544ds machine (Thomas)
- Virtex-ml507 ppc machine (Thomas)
- Re-enable the microblaze test (Thomas)
Various fixes and documentation improvements from Cleber.
# gpg: Signature made Mon 08 Feb 2021 20:19:12 GMT
# gpg: using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
# Primary key fingerprint: FAAB E75E 1291 7221 DCFD 6BB2 E3E3 2C2C DEAD C0DE
* remotes/philmd-gitlab/tags/integration-testing-20210208:
Acceptance Tests: remove unnecessary tag from documentation example
Acceptance tests: clarify ssh connection failure reason
tests/acceptance/virtiofs_submounts: required space between IP and port
tests/acceptance/virtiofs_submounts: standardize port as integer
tests/acceptance/virtiofs_submounts: use a virtio-net device instead
tests/acceptance/virtiofs_submounts: do not ask for ssh key password
tests/acceptance/virtiofs_submounts: use workdir property
tests/acceptance/boot_linux: rename misleading cloudinit method
tests/acceptance/boot_linux: fix typo on cloudinit error message
tests/acceptance: Re-enable the microblaze test
tests/acceptance: Add a test for the virtex-ml507 ppc machine
tests/acceptance: Test the mpc8544ds machine
tests/acceptance: Move the pseries test to a separate file
tests/acceptance: Test U-Boot/Linux from Armbian 20.08 on Orange Pi PC
tests/acceptance: Extract do_test_arm_orangepi_armbian_uboot() method
tests/acceptance: Introduce tesseract_ocr() helper
tests/acceptance: Extract tesseract_available() helper in new namespace
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
If the connection to the ssh server fails, it may indeed be a "sshd"
issue, but it may also not be that. Let's state what we know: the
establishment of the connection from the client side was not possible.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210203172357.1422425-13-crosa@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
AFAICT, there should not be a situation where IP and port do not have
at least one whitespace character separating them.
This may be true for other '\s*' patterns in the same regex too.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210203172357.1422425-10-crosa@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Instead of having to cast it whenever it's going to be used, let's
standardize it as an integer, which is the data type that will be
used most often.
Given that the regex will only match digits, it's safe that we'll
end up getting a integer, but, it could as well be a zero.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Message-Id: <20210203172357.1422425-9-crosa@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tests are supposed to be non-interactive, and ssh-keygen is asking for
a passphrase when creating a key. Let's set an empty passphrase to
avoid the prompt.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210203172357.1422425-6-crosa@redhat.com>
[PMD: Reword description per Alex Bennée comment]
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
There's no downloading happening on that method, so let's call it
"prepare" instead. While at it, and because of it, the current
"prepare_boot" and "prepare_cloudinit" are also renamed.
The reasoning here is that "prepare_" methods will just work on the
images, while "set_up_" will make them effective to the VM that will
be launched. Inspiration comes from the "virtiofs_submounts.py"
tests, which this expects to converge more into.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20210203172357.1422425-3-crosa@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
The microblaze kernel sometimes gets stuck during boot (ca. 1 out of 200
times), so we disabled the corresponding acceptance tests some months
ago. However, it's likely better to check that the kernel is still
starting than to not testing it at all anymore. Move the test to
a separate file, enable it again and check for an earlier console
message that should always appear.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210128152815.585478-1-thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
There was a race condition in the first test where there was already the
"crw" output in the dmesg, but the "0.0.4711" entry has not been created
in the /sys fs yet. Fix it by waiting until it is there.
The second test has even more problems on gitlab-CI. Even after adding some
more synchronization points (that wait for some messages in the "dmesg"
output to make sure that the modules got loaded correctly), there are still
occasionally some hangs in this test when it is running in the gitlab-CI.
So far I was unable to reproduce these hangs locally on my computer, so
this issue might take a while to debug. Thus disable the 2nd test in the
gitlab-CI until the problems are better understood and fixed.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Tested-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210108185645.86351-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
This initrd contains a virtio-net and a virtio-gpu kernel module,
so we can check that we can set a MAC address for the network device
and whether we can hot-plug and -unplug a virtio-crypto device.
But the most interesting part is maybe that we can also successfully
write some stuff into the emulated framebuffer of the virtio-gpu
device and make sure that we can read back that data from a screenshot.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20201221143423.23607-1-thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Cornelia Huck <cohuck@redhat.com>
The kernel/initrd combination does not provide the virtio-net
driver; therefore, simply check whether the presented device type
is indeed virtio-net for the two virtio-net-{ccw,pci} devices.
Signed-off-by: Cornelia Huck <cohuck@redhat.com>
[re-formatted overlong lines]
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20201130180216.15366-3-cohuck@redhat.com>
The max_revision prop of virtio-ccw devices can be used to force
an older revision for compatibility handling. The easiest way to
check this is to force a device to revision 0, which turns off
virtio-1.
Signed-off-by: Cornelia Huck <cohuck@redhat.com>
[re-formatted overlong lines]
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20201130180216.15366-2-cohuck@redhat.com>
This adds a very basic test for checking that we present devices
in a way that Linux can consume: boot with both virtio-net-ccw and
virtio-net-pci attached and then verify that Linux is able to see
and detect these devices.
Signed-off-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20201126130158.1471985-1-cohuck@redhat.com>
Previously we were leaving temporary directories behind. While the
QEMUMachine does make efforts to clean up after itself the directory
belongs to the calling function. We use TemporaryDirectory to wrap
this although we explicitly clear the reference in tearDown() as it
doesn't get cleaned up otherwise.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20201117173635.29101-4-alex.bennee@linaro.org>
Fix Lesser GPL license versions (should be "2.1" and not "2")
# gpg: Signature made Sun 15 Nov 2020 16:20:10 GMT
# gpg: using RSA key 27B88847EEE0250118F3EAB92ED9D774FE702DB5
# gpg: issuer "thuth@redhat.com"
# gpg: Good signature from "Thomas Huth <th.huth@gmx.de>" [full]
# gpg: aka "Thomas Huth <thuth@redhat.com>" [full]
# gpg: aka "Thomas Huth <huth@tuxfamily.org>" [full]
# gpg: aka "Thomas Huth <th.huth@posteo.de>" [unknown]
# Primary key fingerprint: 27B8 8847 EEE0 2501 18F3 EAB9 2ED9 D774 FE70 2DB5
* remotes/huth-gitlab/tags/pull-request-2020-11-15: (26 commits)
nomaintainer: Fix Lesser GPL version number
test: Fix LGPL information in the file headers
tests/acceptance: Fix LGPL information in the file headers
tests/migration: Fix LGPL information in the file headers
sparc tcg cpus: Fix Lesser GPL version number
e1000e: Fix Lesser GPL version number
x86 hvf cpus: Fix Lesser GPL version number
nvdimm: Fix Lesser GPL version number
w32: Fix Lesser GPL version number
tpm: Fix Lesser GPL version number
overall/alpha tcg cpus|hppa: Fix Lesser GPL version number
overall usermode...: Fix Lesser GPL version number
migration: Fix Lesser GPL version number
parallel nor flash: Fix Lesser GPL version number
arm tcg cpus: Fix Lesser GPL version number
x86 tcg cpus: Fix Lesser GPL version number
linux user: Fix Lesser GPL version number
usb: Fix Lesser GPL version number
tricore tcg cpus: Fix Lesser GPL version number
xtensa tcg cpus: Fix Lesser GPL version number
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
This test is regularly failing on CI:
(05/34) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_microblaze_s3adsp1800:
Linux version 4.11.3 (thuth@thuth.remote.csb) (gcc version 6.4.0 (Buildroot 2018.05.2) ) #5 Tue Dec 11 11:56:23 CET 2018
...
Freeing unused kernel memory: 1444K
This architecture does not have kernel memory protection.
[nothing happens here]
Runner error occurred: Timeout reached (90.91 s)
This is a regression. Until someone figure out the problem,
disable the test to keep CI pipeline useful.
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20201109091719.2449141-1-f4bug@amsat.org>
Message-Id: <20201110192316.26397-9-alex.bennee@linaro.org>
There never was a "Lesser GPL version 2.0", It is either "GPL version 2.0"
or "Lesser GPL version 2.1". This patch replaces all "Lesser GPL version 2.0"
with "Lesser GPL version 2.1" in the tests/acceptance folder.
Signed-off-by: Gan Qixin <ganqixin@huawei.com>
Message-Id: <20201110184223.549499-3-ganqixin@huawei.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
This test invokes several shell scripts to create a random directory
tree full of submounts, and then check in the VM whether every submount
has its own ID and the structure looks as expected.
(Note that the test scripts must be non-executable, so Avocado will not
try to execute them as if they were tests on their own, too.)
Because at this commit's date it is unlikely that the Linux kernel on
the image provided by boot_linux.py supports submounts in virtio-fs, the
test will be cancelled if no custom Linux binary is provided through the
vmlinuz parameter. (The on-image kernel can be used by providing an
empty string via vmlinuz=.)
So, invoking the test can be done as follows:
$ avocado run \
tests/acceptance/virtiofs_submounts.py \
-p vmlinuz=/path/to/linux/build/arch/x86/boot/bzImage
This test requires root privileges (through passwordless sudo -n),
because at this point, virtiofsd requires them. (If you have a
timestamp_timeout period for sudoers (e.g. the default of 5 min), you
can provide this by executing something like "sudo true" before invoking
Avocado.)
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20201102161859.156603-8-mreitz@redhat.com>
Tested-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Testing and gitdm updates
- add some more individual contributors
- include SDL2 in centos images
- skip checkpatch check when no commits found
- use random port for gdb reverse debugging
- make gitlab use it's own mirrors to clone
- fix detection of make -nqp
# gpg: Signature made Tue 27 Oct 2020 09:55:55 GMT
# gpg: using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8 DF35 FBD0 DB09 5A9E 2A44
* remotes/stsquad/tags/pull-testing-and-misc-271020-1:
makefile: handle -n / -k / -q correctly
gitlab-ci: Clone from GitLab itself
tests/acceptance: pick a random gdb port for reverse debugging
scripts: fix error from checkpatch.pl when no commits are found
gitlab: skip checkpatch.pl checks if no commit delta on branch
tests/docker/dockerfiles/centos: Use SDL2 instead of SDL1
contrib/gitdm: Add more individual contributors
Adding ani's email as an individual contributor
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Acceptance testing patches
- More ARM tests
- Documentation update
CI jobs results:
. https://cirrus-ci.com/build/5754555684093952
. https://gitlab.com/philmd/qemu/-/pipelines/207807648
# gpg: Signature made Mon 26 Oct 2020 19:06:52 GMT
# gpg: using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
# Primary key fingerprint: FAAB E75E 1291 7221 DCFD 6BB2 E3E3 2C2C DEAD C0DE
* remotes/philmd-gitlab/tags/acceptance-testing-20201026:
docs/devel/testing.rst: Update outdated Avocado URLs
tests/acceptance: Allow running Orange Pi test using cached artifacts
tests/acceptance: Add a 'virt_kvm' test using the GICv3
tests/boot_linux_console: Boot Trusted Firmware-A on the Raspberry Pi 3
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Currently the test randomly fails if you are using a shared machine
due to contention on the well known port 1234. We can ameliorate this
a bit by picking a random non-ephemeral port although it doesn't
totally avoid the problem. While we could use a totally unique socket
address for debugging it is fiddly to probe for gdb support. While gdb
socket debugging is not yet ubiquitous this a sub-optimal but workable
option.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Pavel Dovgalyuk <pavel.dovgalyuk@ispras.ru>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20201021163136.27324-7-alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
This test invokes several shell scripts to create a random directory
tree full of submounts, and then check in the VM whether every submount
has its own ID and the structure looks as expected.
(Note that the test scripts must be non-executable, so Avocado will not
try to execute them as if they were tests on their own, too.)
Because at this commit's date it is unlikely that the Linux kernel on
the image provided by boot_linux.py supports submounts in virtio-fs, the
test will be cancelled if no custom Linux binary is provided through the
vmlinuz parameter. (The on-image kernel can be used by providing an
empty string via vmlinuz=.)
So, invoking the test can be done as follows:
$ avocado run \
tests/acceptance/virtiofs_submounts.py \
-p vmlinuz=/path/to/linux/build/arch/x86/boot/bzImage
This test requires root privileges (through passwordless sudo -n),
because at this point, virtiofsd requires them. (If you have a
timestamp_timeout period for sudoers (e.g. the default of 5 min), you
can provide this by executing something like "sudo true" before invoking
Avocado.)
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200909184028.262297-9-mreitz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
The current 'virt_kvm' test is restricted to GICv2, but can also
work with a GICv3. Duplicate it but add a GICv3 test which can be
tested on some hardware.
Noticed while running:
$ avocado --show=app run -t machine:virt tests/acceptance/
...
(2/6) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm: ERROR: Unexpected empty reply from server (1.82 s)
The job.log content is:
L0351 DEBUG| Output: 'qemu-system-aarch64: host does not support in-kernel GICv2 emulation\n'
With this patch:
$ avocado --show=app run -t device:gicv3 tests/acceptance/
(1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm_gicv3: PASS (55.10 s)
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20200929224857.1225107-1-philmd@redhat.com>
This test runs Trusted Firmware-A on the Raspberry Pi 3.
We deliberately stop the boot process when the EDK2 UEFI version
is displayed.
The binary is build on AppVeyor CI using Pete Batard repository [1].
ATF v2.1 binary are used (see [2]).
It is very simple and fast:
$ avocado --show=app,console run -t atf tests/acceptance
JOB ID : 1e748d7c9e9011cf0af3250ddc8ebf2389d6204e
JOB LOG : avocado/job-results/job-2020-02-16T18.08-1e748d7/job.log
(1/1) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf:
console: NOTICE: Booting Trusted Firmware
console: NOTICE: BL1: v2.1(release):v2.1
console: NOTICE: BL1: Built : 15:26:06, May 13 2019
console: NOTICE: rpi3: Detected: Raspberry Pi 3 Model B (1GB, Sony, UK) [0x00a02082]
console: NOTICE: BL1: Booting BL2
console: ERROR: rpi3_sdhost: timeout status 0x40
console: NOTICE: BL2: v2.1(release):v2.1
console: NOTICE: BL2: Built : 15:26:01, May 13 2019
console: NOTICE: BL1: Booting BL31
console: NOTICE: BL31: v2.1(release):v2.1
console: NOTICE: BL31: Built : 15:26:04, May 13 2019
console: =UEFI firmware (version UEFI Firmware v1.15 built at 11:58:44 on Feb 14 2020)
PASS (1.54 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 1.88 s
[1] https://github.com/pbatard/RPi3#summary
[2] https://github.com/ARM-software/arm-trusted-firmware/blob/v2.1/docs/plat/rpi3.rst
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Luc Michel <luc.michel@greensocs.com>
Message-Id: <20200217103442.30318-1-f4bug@amsat.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
