🌮 🦏 👨🏽‍🍳 Automatización de pruebas de sistemas basada en QEMU (Parte 2/2) 🐌 🤗 🎅🏻

Esta es la segunda parte de un artículo sobre la automatización de pruebas de sistemas basada en VM. La primera parte se puede encontrar aquí .

En esta parte del artículo, utilizaremos las habilidades aprendidas en la primera parte para automatizar realmente las pruebas del sistema. Al final del artículo, recibiremos un script que cualquiera puede ejecutar en su computadora y completamente desde cero obtener un soporte desplegado de tres máquinas, instaladas por la aplicación bajo prueba, así como también pasar las pruebas reales del sistema (escribiremos tres pruebas).

Descargo de responsabilidad

, , , . " " , . , .

Entonces, en la última parte, nos abastecimos de un impresionante arsenal de habilidades para trabajar con máquinas virtuales desde la línea de comandos: aprendimos cómo instalar máquinas virtuales, implementar un sistema operativo en ellas (por ejemplo, Ubuntu Server 18.04), conectar una máquina virtual a un host a través de una red e incluso organizar un canal de control a través de SSH. Todo esto nos será útil en este artículo, pero antes de pasar a la práctica, hay varios temas que discutir.

¿Qué queremos conseguir?

La pregunta más importante que debe responderse es "¿Qué resultado queremos obtener?" Sí, la última vez hablamos mucho sobre automatizar la instalación, despliegue y configuración de máquinas virtuales, pero aparte del objetivo final, todo esto no tiene mucho sentido.

Para mí, personalmente, las pruebas del sistema todo en uno se ven así: descargo varios archivos pequeños de VCS (el script en sí con el inicio, más, posiblemente, varios artefactos auxiliares), pongo el programa a prueba donde lo necesito (en forma de un instalador o un paquete, por ejemplo), Aprieto un botón y voy a tomar café. Cuando regrese, quiero ver que todas las pruebas pasaron, o que tales o cuales pruebas se han roto. No quiero participar en ninguna configuración del stand, no quiero implementar máquinas virtuales o configurar algo allí. Quiero descargar un script y usarlo.

, : , . , ( ).

, .

. -, Data Plane Development Kit (DPDK). DPDK — , , , DPDK. DPDK , , end-to-end .

DPDK

DPDK (Data Plane Development Kit) — , C. , . , . , . DPDK . ? . , , Linux, , , , . , Linux — , . , DPDK .

;
;
DROP — ;
ACCEPT — ;

, , :

;
, ;

, , , :

(client, middlebox, server), Ubuntu Server 18.04 ();
: client middlebox ( net_1) middlebox server ( net_2);
;
, ;
- middlebox;
.

. :

, SSH- , , , ;
SSH net_1 net_2, ( net_for_ssh). :

.. , , - ;
, - (, ), .

!

- :

, , :

run_tests.sh

#!/bin/bash
set -euo pipefail

# =======================================
#   net_for_ssh
# =======================================
virsh net-define net_for_ssh.xml
virsh net-start net_for_ssh

# =======================================
#   net_1
# =======================================
virsh net-define net_1.xml
virsh net-start net_1

# =======================================
#   net_2
# =======================================
virsh net-define net_2.xml
virsh net-start net_2

# =======================================
#   client
# =======================================
virt-builder ubuntu-18.04 \
    --format qcow2 \
    --output client.qcow2 \
    --install wget \
    --root-password password:1111 \
    --run-command "ssh-keygen -A" \
    --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
    --copy-in netcfg_client.yaml:/etc/netplan/

virt-install \
    --import \
    --name client \
    --ram 1024 \
    --disk client.qcow2 \
    --network network=net_for_ssh \
    --network network=net_1,mac=52:54:56:11:00:00 \
    --noautoconsole

# =======================================
#   middlebox
# =======================================
virt-builder ubuntu-18.04 \
    --format qcow2 \
    --output middlebox.qcow2 \
    --install python,daemon,libnuma1 \
    --root-password password:1111 \
    --run-command "ssh-keygen -A" \
    --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
    --copy-in netcfg_middlebox.yaml:/etc/netplan/

virt-install \
    --import \
    --name middlebox \
    --vcpus=2,sockets=1,cores=2,threads=1 \
    --cpu host \
    --ram 2048 \
    --disk middlebox.qcow2 \
    --network network=net_for_ssh \
    --network network=net_1,model=e1000 \
    --network network=net_2,model=e1000 \
    --noautoconsole

# =======================================
#   server
# =======================================
virt-builder ubuntu-18.04 \
    --format qcow2 \
    --output server.qcow2 \
    --install nginx \
    --root-password password:1111 \
    --run-command "ssh-keygen -A" \
    --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
    --copy-in netcfg_server.yaml:/etc/netplan/

virt-install \
    --import \
    --name server \
    --ram 1024 \
    --disk server.qcow2 \
    --network network=net_for_ssh \
    --network network=net_2,mac=52:54:56:00:00:00 \
    --noautoconsole

# =======================================
# ,    
#     
# =======================================

SSH_CMD="sshpass -p 1111 ssh -o StrictHostKeyChecking=no"

while ! SSH_CMD root@192.168.100.2 "echo Hello world from client!" echo
do
    echo "Waiting for client VM ..."
    sleep 1
done

while ! SSH_CMD root@192.168.100.3 "echo Hello world from middlebox!" echo
do
    echo "Waiting for middlebox VM ..."
    sleep 1
done

while ! SSH_CMD root@192.168.100.4 "echo Hello world from server!" echo
do
    echo "Waiting for server VM ..."
    sleep 1
done

net_for_ssh.xml

<network>
    <name>net_for_ssh</name>
    <bridge name='net_for_ssh'/>
    <ip address='192.168.100.1' netmask='255.255.255.0'/>
</network>

net_1.xml

<network>
    <name>net_1</name>
    <bridge name='net_1'/>
    <ip address='192.168.101.1' netmask='255.255.255.0'/>
</network>

net_2.xml

<network>
    <name>net_2</name>
    <bridge name='net_2'/>
    <ip address='192.168.102.1' netmask='255.255.255.0'/>
</network>

netcfg_client.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.100.2/24
    ens4:
      addresses:
        - 192.168.101.2/24
      gateway4: 192.168.101.3

netcfg_middlebox.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.100.3/24

netcfg_server.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.100.4/24
    ens4:
      addresses:
        - 192.168.102.2/24
      gateway4: 192.168.102.3

, :

--install virt-builder, . --run-command "apt install ...". , , — virt-builder . client wget, server — nginx ( http- ). middlebox , DPDK-;
client server , - , . ;
middlebox ( --vcpus): CPU c hyperthreading. — , DPDK . --cpu host, , , . , - QEMU , SSE3 . , , DPDK .
middlebox , : e1000. , - DPDK.

run_tests.sh ( , ). , :

;
.

, run_tests.sh, , . :

run_clean.sh

#!/bin/bash
set -euo pipefail

# =======================================
#   client
# =======================================
if virsh list --all | grep -q " client "; then
    if virsh domstate client | grep -q "running"; then
        virsh destroy client
    fi
    virsh undefine client --snapshots-metadata --remove-all-storage
fi

# =======================================
#   middlebox
# =======================================
if virsh list --all | grep -q " middlebox "; then
    if virsh domstate middlebox | grep -q "running"; then
        virsh destroy middlebox
    fi
    virsh undefine middlebox --snapshots-metadata --remove-all-storage
fi

# =======================================
#   server
# =======================================
if virsh list --all | grep -q " server "; then
    if virsh domstate server | grep -q "running"; then
        virsh destroy server
    fi
    virsh undefine server --snapshots-metadata --remove-all-storage
fi

# =======================================
#   net_for_ssh
# =======================================
if virsh net-list --all | grep -q " net_for_ssh "; then
    if virsh net-list --all | grep " net_for_ssh " | grep -q " active "; then
        virsh net-destroy net_for_ssh
    fi
    virsh net-undefine net_for_ssh
fi

# =======================================
#   net_1
# =======================================
if virsh net-list --all | grep -q " net_1 "; then
    if virsh net-list --all | grep " net_1 " | grep -q " active "; then
        virsh net-destroy net_1
    fi
    virsh net-undefine net_1
fi

# =======================================
#   net_2
# =======================================
if virsh net-list --all | grep -q " net_2 "; then
    if virsh net-list --all | grep " net_2 " | grep -q " active "; then
        virsh net-destroy net_2
    fi
    virsh net-undefine net_2
fi

, run_tests.sh, . :

( virsh list --all, — virsh net-list -all);
/, , / , ;
( --snapshots-metadata) ( --remove-all-storage).

run_tests.sh run_clean.sh. run_tests.sh , run_clean.sh.

run_clean.sh . . , , , .

. SSH- — SSH ~/.ssh/known_hosts. , SSH — , , - . SSH_CMD:

SSH_CMD="sshpass -p 1111 ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR"

-o UserKnownHostsFile=/dev/null, .

, — . Ubuntu Server ( GUI) bash-, .

bash-, . , .

-, , :

EXEC_CLIENT="$SSH_CMD root@192.168.100.2"
EXEC_MIDDLEBOX="$SSH_CMD root@192.168.100.3"
EXEC_SERVER="$SSH_CMD root@192.168.100.4"

. . :

$EXEC_CLIENT echo hello from client
$EXEC_SERVER echo hello from server

-, ? bash- Heredoc. Heredoc :

$EXEC_CLIENT << EOF
    echo hello from client
    ls
    pwd
EOF

$EXEC_MIDDLEBOX << EOF
    echo hello from middlebox
    some_another_command
EOF

EOF — , . EOF , , .

, set -xeuo pipefail. :

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    command1
    command2 | command3
EOF

, :

-x bash- ;
-e , ;
-u , ;
-o pipeline , - .

, - — .

, . bash- , - . :

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    command1
    ! command2
EOF

: command1 0, command2 , .

$SCP_CMD l3fwd-acl-1.0.0.deb root@192.168.100.3:~

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    dpkg -i l3fwd-acl-1.0.0.deb

    echo 256 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
    mkdir -p /mnt/huge
    mount -t hugetlbfs nodev /mnt/huge

    modprobe uio_pci_generic
    dpdk-devbind --bind=uio_pci_generic ens4 ens5

    echo "R0.0.0.0/0 192.168.102.0/24 0 : 65535 0 : 65535 0x0/0x0 1" > /etc/rule_ipv4.db
    echo "R0.0.0.0/0 192.168.101.0/24 0 : 65535 0 : 65535 0x0/0x0 0" >> /etc/rule_ipv4.db

    echo "R0:0:0:0:0:0:0:0/0 0:0:0:0:0:0:0:0/0 0 : 65535 0 : 65535 0x0/0x0 0" > /etc/rule_ipv6.db

    daemon --name l3fwd --unsafe --output /var/log/l3fwd -- l3fwd-acl \
        -l 1 \
        -n 4 \
        -- \
        -p 0x3 \
        -P \
        --config="(0,0,1),(1,0,1)" \
        --rule_ipv4="/etc/rule_ipv4.db" \
        --rule_ipv6="/etc/rule_ipv6.db"
EOF

, ( ):

deb-;
256 2 (DPDK- - );
poll-mode uio_pci_generic ( Ubuntu Server). , DPDK ;
ens4 ( ) ens5 ( ) uio_pci_generic;
rule_ipv4.db IPv4 : 192.168.102.0/24 1 ( , ), 192.168.101.0/24 0 ( );
rule_ipv6.db, " 0". IPv6 , DPDK ;
l3fwd daemon. , l3fwd: https://doc.dpdk.org/guides/sample_app_ug/l3_forward.html

, DPDK :

.deb ;
;
;
;
, , uio_pci_generic;
.

: " ", - . , . .

: , DPDK- ( unit-, ), , : - :

$EXEC_CLIENT arp -s 192.168.101.3 52:54:56:00:00:00
$EXEC_SERVER arp -s 192.168.102.3 52:54:56:11:00:00

$EXEC_CLIENT << EOF
    set -xeuo pipefail

    ping -c 5 192.168.102.2
    wget --timeout=5 --tries=1 http://192.168.102.2
EOF

ARP-

, , ARP-.

? , l3fwd,

DPDK, , ARP.

l3fwd ,

rule_ipv4.db rule_ipv6.db,

: , / , --.

:

, .

, middlebox

, MAC-

Ethernet- ( client server ).

:

destination MAC-.

ARP-.

# =======================================
#  ,  tcp 
# =======================================

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    daemon --name l3fwd --stop

    #     
    echo "@0.0.0.0/0 0.0.0.0/0 0 : 65535 0 : 65535 0x06/0xff" > /etc/rule_ipv4.db

    echo "R0.0.0.0/0 192.168.102.0/24 0 : 65535 0 : 65535 0x0/0x0 1" >> /etc/rule_ipv4.db
    echo "R0.0.0.0/0 192.168.101.0/24 0 : 65535 0 : 65535 0x0/0x0 0" >> /etc/rule_ipv4.db

    daemon --name l3fwd --unsafe --output /var/log/l3fwd -- l3fwd-acl \
        -l 1 \
        -n 4 \
        -- \
        -p 0x3 \
        -P \
        --config="(0,0,1),(1,0,1)" \
        --rule_ipv4="/etc/rule_ipv4.db" \
        --rule_ipv6="/etc/rule_ipv6.db"
EOF

# =======================================
# ,  ping  ,
#  http  - 
# =======================================

$EXEC_CLIENT << EOF
    set -xeuo pipefail

    ping -c 5 192.168.102.2
    ! wget --timeout=5 --tries=1 http://192.168.102.2
EOF

, wget : , , wget .

`run_tests.sh`

, , run_clean.sh.

: : DPDK- middlebox . ? , ( ) DPDK- . , , . ?

. , ? , . : , init, .

# =======================================
#   client
# =======================================
if ! virsh list --all | grep -q " client "
then
    virt-builder ubuntu-18.04 \
        --format qcow2 \
        --output client.qcow2 \
        --hostname client \
        --install wget,net-tools \
        --root-password password:1111 \
        --run-command "ssh-keygen -A" \
        --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
        --copy-in netcfg_client.yaml:/etc/netplan/

    virt-install \
        --import \
        --name client \
        --ram 1024 \
        --disk client.qcow2 \
        --network network=net_for_ssh \
        --network network=net_1,mac=52:54:56:11:00:00 \
        --noautoconsole

    virsh snapshot-create-as client --name init
else
    virsh snapshot-revert client --snapshotname init
fi

, init.

, , : , .deb- DPDK, . .

, , :

# =======================================
#   net_1
# =======================================
if ! virsh net-list --all | grep -q " net_1 "
then
    virsh net-define net_1.xml
    virsh net-start net_1
fi

, , , . : !

, , , , . : , , , (end-to-end) .

, , : DPDK- ( Ubuntu Server 18.04) ( ping wget). , .deb . , , ( , ). .

: (run_tests.sh run_clean.sh), xml- yaml-. VCS. .

, , " — ". , . , " ", .

"", "" "" , , , , . Proof Of Concept . - … . , , , , . ...

→

Automatización de pruebas de sistemas basada en QEMU (Parte 2/2)

¿Qué queremos conseguir?

!

run_tests.sh

More articles:

`run_tests.sh`