QEMU Intro and Network Configuration

To analyse firmwares or apps from different architectures on your Linux PC, you have to use emulator to run them.
Here I introduce QEMU to you.

QEMU

QEMU is a generic and open source machine emulator and virtualizer.
QEMU has two operating modes:

• Full system emulation
  In this mode, QEMU emulates a full system (for example a PC), including a processor and various peripherals. It can be used to launch different Operating Systems without rebooting the PC or to debug system code.

• User mode emulation (Linux host only)
  In this mode, QEMU can launch Linux processes compiled for one CPU on another CPU. For example, it can be used to launch Wine or to ease cross-compilation and cross-debugging.

Installation

Full system emulation:

$ apt install qemu-system # install many qemu's platform emulators 

User mode emulation:

$ apt install qemu-user # to run apps with dynamic linked libs
$ apt install qemu-user-binfmt # installed along with qemu-user
$ apt install binfmt-support
$ apt install qemu-user-static # to run apps with static linked libs

Create Virtual Disk

It prepares for installation of virtual machine.

$ qemu-img create -f qcow2 OS.img 20G
# 'create' means to create vm disk
# '-f qcow2' assigns its format. vmdk and qcow are ok.
# 'OS.img' names the vm disk
# '20G' means the size

Install/Run Virtual Machine

To install virtual machine

$ qemu-system-x86_64 -boot d -cdrom ubuntu-16.04-desktop-amd64.iso -hda OS.img -m 2048 -bios OVMF.fd
# 'qemu-system-xxx' depends on what platform you want to use
# '-boot d' means boot from cdrom
# '-cdrom ubuntu-xxx' assigns the image to install
# '-hda OS.img' assigns the vm disk to use
# '-m 2048' means using 2048MB memory
# '-bios OVMF.fd' assigns bios (default bios is seabios)

To accelerate (use kvm)

$ apt install qemu-kvm 

$ kvm -boot d -cdrom ubuntu-16.04-desktop-amd64.iso -hda OS.img -m 2048 -bios OVMF.fd

To run virtual machine

$ qemu-system-mips -M malta -nographic -no-reboot -kernel "zImage-mips" -hda "image-mips.ext2" -append "root=/dev/hda rw init=/usr/sbin/init.sh panic=1 PATH=/usr/bin console=ttyS0" -net nic -net tap -drive file=/tmp/share.img
# '-M malta' assigns developing board/platform
# '-kernel "zImage-mips"' assigns kernel
# '-append "xxx"' gives boot arguments of kernel
# '-net nic -net tap' gives network configuration
# '-drive file=/tmp/share.img' defines a new drive. 
# '-L' assigns dynamic lib location

Other examples:

$ qemu-system-mipsel -kernel openwrt-malta-le-vmlinux-initramfs.elf -M malta  -drive file=/tmp/share.img -net nic -net tap -nographic
 
$ qemu-system-mips -M malta -kernel vmlinux-2.6.32-5-4kc-malta -hda debian_squeeze_mips_standard.qcow2 -append "root=/dev/sda1 console=tty0" -net nic -net tap -nographic

# run dynamic linked program, not whole system
$ qemu-mipsel -L Path/to/dynamic/linked/libs hello

Note: images and kernel modules can be found Here, in which the README.txt provides QEMU using scripts.

Debug Programs

Since QEMU implements gdbserver, dynamic analysis can be performed by adding -g and gdb attach to it.

First install necessary libs. Take ARM as example.

$ apt install -y gcc-arm-linux-gnueabi        #32-bit
$ apt install -y gcc-aarch64-linux-gnu g++-aarch64-linux-gnu      #64-bit

Here we have:
64-bit:
gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
32-bit:
gcc-arm-linux-gnueabi gcc-arm-linux-gnueabihf
g++-arm-linux-gnueabi g++-arm-linux-gnueabihf

• Let’s distinguish the two:
  gcc-arm-linux-gnueabi – The GNU C compiler for armel architecture
  gcc-arm-linux-gnueabihf – The GNU C compiler for armhf architecture
  The only difference is, they means differnet value of ‘-mfloat-abi’ option of gcc

  There are 3 values to choose for ‘-mfloat-abi’:

  1. soft: do float caculation without fpu(floating-point unit), but using software method.
  2. softfp: default method that armel arch uses (default value gcc-arm-linux-gnueabi uses), which means using fpu to calculate, but save arguments in general registors.
  3. hard: default method that armhf arch uses (default value gcc-arm-linux-gnueabihf uses), which means using fpu to calculate, and save arguments in fpu registors.

• Let’s distinguish gcc and g++
  GCC: GNU Compiler Collection, Referrers to all the different languages that are supported by the GNU compiler.
  gcc: GNU C Compiler
  g++: GNU C++ Compiler

  The main differences:

  1. gcc will compile: .c/.cpp files as C and C++ respectively.
  2. g++ will compile: .c/.cpp files but they will all be treated as C++ files.
     Also if you use g++ to link the object files it automatically links in the std C++ libraries (gcc does not do this).
  3. gcc compiling C files has less predefined macros.
  4. gcc compiling .cpp and g++ compiling *.c/.cpp files has a few extra macros.

Run apps with dynamic libs

$ qemu-arm -L /usr/arm-linux-gnueabi ./filename       #32-bit
$ qemu-aarch64 -L /usr/aarch64-linux-gnu ./filename   #64-bit

Install gdb

$ apt install git gdb gdb-multiarch

Debug 32-bit apps

# term window 1
$ qemu-arm -g 1234 -L /usr/arm-linux-gnueabi ./file
# term window 2
$ gdb-multiarch ./file
pwndbg> target remote :1234
pwndbg> b *0x8bb0

Debug 64-bit apps

# term window 1
$ qemu-aarch64 -g 1234 -L /usr/aarch64-linux-gnu ./file
# term window 2
$ gdb-multiarch ./file
pwndbg> target remote :1234
pwndbg> b *0x8bb0

Network Configuration

QEMU provides 4 network communication methods, which are:

1. User mode stack: to implement a protocol stack in the QEMU process. Work like NAT server.
2. Socket: Create a socket for a VLAN and connect multiple VLANs together.
3. TAP / bridge: The most important communication method, we need to use this method if we want to apply QEMU virtual machine’s external communication.
4. VDE: also used for connecting VLAN

TAP mode works like VMWare’s network bridge.
Essentially implements network card virtually (not physically)

Software Preparations

$ apt-get install bridge-utils        # virtual bridge utils
$ apt-get install uml-utilities       # UML（User-mode linux） utils

Build a network bridge

At first your network is:

Build the bridge

$ ifconfig your-net-int down           # stop network interface
$ brctl addbr br0                     # add a bridge br0
$ brctl addif br0 your-net-int         # add an interface in br0
$ brctl stp br0 off                   # spanning-tree-protocol can be closed
$ brctl setfd br0 1                   # set br0 forwarding delay
$ brctl sethello br0 1                # set br0 hello time
$ ifconfig br0 0.0.0.0 promisc up     # start br0 interface
$ ifconfig your-net-int 0.0.0.0 promisc up    # start network interface
$ dhclient br0                        # get br0 ip from dhcp server
$ brctl show br0                      # see br0 list
$ brctl showstp br0                   # see br0 interfaces

Type ifconfig to see result:

The brief picture of network (bridge and internet):

Create a TAP device

Create a TAP device to be an interface on the bridge, which QEMU can use

$ tunctl -t tap0 -u root              # create a tap0 interface, only accessed by root
$ brctl addif br0 tap0                # add tap0 to br0
$ ifconfig tap0 0.0.0.0 promisc up    # start tap0 interface
$ brctl showstp br0                   # show br0 interfaces

Type ifconfig to see result:

Now the picture of network (bridge and internet)

Well, if other device or interface is needed…
Existence of more than 2 interfaces is permitted in the bridge:

Connect QEMU to tap0

Now we just need to connect QEMU vm to tap0

$ sudo qemu-system-mipsel -M malta -kernel vmlinux-3.2.0-4-4kc-malta -hda debian_squeeze_mipsel_standard.qcow2 -append "root=/dev/sda1 console=tty0" -nographic -net nic -net tap,ifname=tap0,script=no,downscript=no

Finally we get vm connected to internet

But the problems are:

1. sometimes, under special circumstances, QEMU cannot connect to internet. type ifconfig to check network configuration, and use dhclient br0 to get dhcp ip again.
2. All commands above used by root, so remember to add sudo.
3. Once you reboot, all configs disappear. Write a script please.

Another Method

Edit /etc/network/interfaces

auto lo 
iface lo inet loopback 

auto eth0 
iface eth0  inet manual 
up ifconfig eth0  0.0.0.0 up 

auto br0
iface br0 inet dhcp 
bridge_ports eth0 
bridge_stp off 
bridge_maxwait 1

Edit /etc/qemu-ifup

#!/bin/sh 
echo "Executing /etc/qemu-ifup" 
echo "Bringing $1 for bridged mode..." 
sudo /sbin/ifconfig $1 0.0.0.0 promisc up 
echo "Adding $1 to br0..." 
sudo /sbin/brctl addif br0 $1 
sleep 3

Then

$ chmod a+x /etc/qemu-ifup

Restart network

$ /etc/init.d/networking restart

In this way, there is no need to configure next time you boot up.

Method for WSL users

If you use WSL(Windows Subsystem for Linux), the above network modifying is not recommended, because

# on WSL 1 Ubuntu we failed to add devices
$ tunctl
Failed to open '/dev/net/tun' : No such file or directory
$ brctl addbr br0
add bridge failed: Invalid argument
$ ifconfig  -a  # no new devices
eth1: ...
eth2: ...
...
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 1500
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0xfe<compat,link,site,host>
        loop  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


# but normally we get (like on WSL 2 Ubuntu):
$ tunctl
Set 'tap0' persistent and owned by uid 0
$ brctl addbr br0
$ ifconfig  -a  # added br0 and tap0
...
br0: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether 12:93:04:50:e6:db  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
...
tap0: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether 2a:a7:6d:67:b2:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Well, it doesn’t mean this method works on WSL 2. If continue, we have to disable ethx and enable br0 to get ip byDHCP.

This step is dangerous since br0 cannot receive a valid ip via DHCP and your ethx may not be fine as well.

But We can use available bridge device that docker provides us with – docker0.

On WSL 2 we can use docker (WSL 1 not supporting), though systemctl not

$ apt install docker.io
$ systemctl start docker    # systemctl still invalid
System has not been booted with systemd as init system (PID 1). Cant operate.
Failed to connect to bus: Host is down
$ service docker start
Starting Docker: docker.

And the bridge docker0 is assigned a ip: 172.17.10.1

$ ifconfig  -a
...
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.10.1  netmask 255.255.255.0  broadcast 172.17.10.255
        ether 02:42:69:86:df:1f  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Then we add a TAP device tap0 and connect it with docker0

$ tunctl -t tap0
Set 'tap0' persistent and owned by uid 0
$ ifconfig tap0 up
$ brctl addif docker0 tap0
$ brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.000000000000       no
docker0         8000.02426986df1f       no              tap0

When starting QEMU we add -net nic -net tap,ifname=tap0,script=no,downscript=no at the tail.

On QEMU guest we configure ip and gateway and dns

$ ifconfig eth0 172.17.10.xxx/24
$ route add default gw 172.17.10.1
$ echo "nameserver 8.8.8.8" > /etc/resolv.conf

Then the network configuration is finished.

But when using docker virtual machines, conflicts may occur, so you can add "bip": "x.x.x.x/24" to /etc/docker/daemon.json to assign new ip for docker0.