Thin client router part 2: virtual insanity

 

So in part 1, I set-up the hardware and the base OS install. Now I'd like to try out OpenWRT with the Dell Wyse 3040 thin client and the Comfast CF-953AX USB3 WiFi6 1800Mbps adapter. Unfortunately the thin client has an Atom x5-Z8350 CPU, which none of the official OpenWRT binaries will run on it directly (more on that in part 3, I guess). Nevertheless, I'd still like to test-out the WiFi adapter with OpenWRT, so I'll do the next best thing: run it in a virtual machine.

How/why virtualise?

OK, so really the sensible thing to do would be solve the Atom build issue now (remember: part 3!), but I was impressed that the CPU supports VT-x virtualisation and I just wanted to fiddle-around with direct attaching USB hardware to a VM. It also means I can quickly try out and roll-back changes if there's much messing around needed.

So here's the full lscpu info for the thin client, where you can see all its VT-x majesty:

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         36 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               GenuineIntel
  BIOS Vendor ID:        Intel
  Model name:            Intel(R) Atom(TM) x5-Z8350  CPU @ 1.44GHz
    BIOS Model name:     Intel(R) Atom(TM) x5-Z8350 CPU @ 1.44GHz Fill By OEM CPU @ 1.4GHz
    BIOS CPU family:     43
    CPU family:          6
    Model:               76
    Thread(s) per core:  1
    Core(s) per socket:  4
    Socket(s):           1
    Stepping:            4
    CPU(s) scaling MHz:  47%
    CPU max MHz:         1920.0000
    CPU min MHz:         480.0000
    BogoMIPS:            2880.00
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse
                         2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology tsc_reliab
                         le nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16
                          xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch epb pti ibrs ib
                         pb stibp tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm ida arat md_clear
Virtualization features:
  Virtualization:        VT-x
Caches (sum of all):
  L1d:                   96 KiB (4 instances)
  L1i:                   128 KiB (4 instances)
  L2:                    2 MiB (2 instances)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-3
Vulnerabilities:
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Mitigation; Clear CPU buffers; SMT disabled
  Meltdown:              Mitigation; PTI
  Mmio stale data:       Unknown: No mitigations
  Retbleed:              Not affected
  Spec store bypass:     Not affected
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

The network quandary

The 3040 has a single gigabit network interface and a single USB3 port. In the long run I'd like to have the WAN network on the gigabit port and the CF-953AX in the USB3 port and deliver the LAN network. The problem is that OpenWRT's Luci web interface is necessarily only available on the LAN network and until I can configure the WiFi interface, there won't be an external interface to contact it over. While I could use the gigabit interface for the LAN side, I'd have no WAN connectivity that I'd need to be able to install updates and packages for the WiFi adapter. Another possibility is doing all the configuration through the VM console, but I'd rather use the Luci web UI to do this. So the way I'll go is to set-up a virtual network for the LAN side and connect the WAN interface to the external gigabit port. The last little wrinkle is that the WAN network needs to be attached to the host virtual network "default" (The default NAT network with the network address of "192.168.122.0/24"). Using the "default" network results in double-NATting the VM's LAN traffic, but it should be fine for this test setup.

The only extra detail for the locallan virtual network is that I don't want it to run its own DHCP service, but I do want an address in the 192.168.1.0/24 range that OpenWRT uses by default. While it's possible (with considerable fiddling around) to run a DHCP client on the host's virtual network, it'd just be easier to just set a static address for the host connection and hope the VM's DHCP server doesn't try to allocate this address. So I'll choose 192.168.1.254 just to be as far from the router address as possible (I can't imagine a way that OpenWRT will allocate this address before the host gets into its ARP table anyway).

Rather than trying to add a large number of difficult to parse command line parameters, LibVirt prefers that complicated configurations are expressed in XML files which are then passed to the virsh command line tool to process. Here's the XML to create the new virtual network specified above (and calling it "locallan"):

<network>
  <name>locallan</name>
  <ip address='192.168.1.254' netmask='255.255.255.0'/>
</network>

Once created, the network needs to be started and set to "autostart" when the host OS boots up:

sudo virsh net-create locallan.xml
sudo virsh net-start locallan
sudo virsh net-autostart locallan

Virtual NIC attachment

So the VM will have 2 NICs attached to it, but which one needs to be attached to what entities outside the VM? By default OpenWRT uses its first NIC as a member of a bridge for the LAN network (Later we'll attach the WiFi interface to this same network). The second NIC is used as the a direct attachment to the WAN network, where it uses a DHCP client setup to determine its address, gateway and DNS settings.

Passing-in the USB WiFi adapter

I want to attach the Comfast WiFi adapter to the VM directly by presenting the host USB device to the OpenWRT guest VM. In LibVirt USB devices can be identified by either their attachment address (i.e. "Bus 002 Device 003"), or vendor:device ID (In the case of the CF-953AX adapter, this is "0e8d:7961"). Unfortunately USB attachment addresses can change a lot (Especially if there's any hotplugging or changes between boots) and I'm not expecting to deal with more than one WiFi dongle (for the moment?), so I can just use the vendor:device ID to attach the WiFi adapter to the VM.

The disk image

I'll download the latest (22.03.5) OpenWRT 64-bit EFI "combined" image. Remembering to decompress it before I use it (I sometimes didn't and kept pulling my hair out wondering why a freshly downloaded image stopped the VM from booting). Libvirt disk images typically live under "/var/lib/libvirt/images", so I'll put it there. I'll keep the original image as a spare/backup, and make a working copy just called "openwrt.img". The image is only 128M, so I can have several copies around without any problem in the 8G MMC flash disk.

The VM definition

As mentioned in the network definition secrion above, LibVirt wants an XML specification for the virtual machine. Most of the "important" unique details for the VM are the virtio data devices:

• The boot disk - "/var/lib/libvirt/images/openwrt.img"
  
• Virtual NICs - 2
  
• NIC network attachments - "locallan" and "default", in that order
  

There are a handful of necessary but "dull" settings all VMs have like:

• Name - "openwrt"
• VCPU count - might as well use all 4 threads
  
• Ram - 1G or half the physical ram is more than enough and the host OS won't miss it.
  

There are a couple more VM and device settings that are nice or necessary too:

• A domain type of "kvm" - As the OS we're running can run on the host CPU using virtualisation features rather than being emulated.
  
• An os type of "hvm" - Same reasons as above
• The "acpi" feature - So the VM domain will "power off" when it's shutdown (Otherwise it just halts but stays "running" as far as libvirt is concerned)
• Pass-through the host CPU rather than use a QEMU virtual CPU.
  
• A "pty" type console device - Allows interacting with GRUB boot loader and the guest OS console from the host OS
• A "qemu-xhci" model USB controller device - So I can attach USB3 devices to the VM.

The whole XML file

<domain type='kvm'>
  <name>openwrt</name>
  <memory>1048576</memory>
  <vcpu>4</vcpu>
  <os>
    <type>hvm</type>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu mode='host-passthrough'/>
  <devices>
    <console type='pty'/>
    <controller type='usb' model='qemu-xhci'/>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/openwrt.img'/>
      <target dev='vda'/>
    </disk>
    <interface type='network'>
      <source network='locallan'/>
      <model type='virtio'/>
    </interface>
    <interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
    </interface>
    <hostdev mode='subsystem' type='usb'>
      <source>
        <vendor id='0x0e8d'/>
        <product id='0x7961'/>
      </source>
    </hostdev>
  </devices>
</domain>

Running the virtual machine

I want to define the VM, have it automatically start when the host boots, start it right now and track its boot progress by connecting to the running VM's console. These are the fairly self-explanatory commands to do that:

virsh define openwrt.xml
virsh autostart openwrt
virsh start openwrt
virsh console openwrt

I pretty soon saw the GRUB boot prompt:

Then the usual wall of Linux kernel boot messages and eventually a prompt "Please press Enter to activate this console." (This might be a few lines before the end of the boot messages, though). I pressed Enter and saw the friendly and familiar OpenWRT login message:

Connect to the Luci UI and package setup

So now OpenWRT's running, I want to connect to the Luci web UI and finish the system setup. It's definitely possible to do all this from the commandline, but I'd rather use the comfort of a GUI.

As detailed in the network config above, the OpenWRT LAN interface is where the Luci web UI presents itself and this network is not accessible outside the host OS. So to get to it, I'm going to make an SSH tunnel to the web port and use my desktop browser to get to Luci. Here's the command line I'm using:

ssh -L 12345:192.168.1.1:80 ${USER_NAME}@${HOST_IP}

Once in place, I pointed a browser at http://localhost:12345 (Note the explicit http protocol as many modern browsers now assume https as the default) and was eventually be led to the Luci login page (and there is no root password yet, so just pressed Enter):

Once I logged-in, I saw the system status page (Warning me again that no root password has been set):

Initial config and package setup

I set the root password (used for both logging into the Luci web UI and through SSH) at System → Administration. Next stop was System → Software, where I firstly installed any available updates, then selected the "Available" tab to install the suplimentary packages I needed. The Comfast CF-953AX USB WiFi dongle has the Mediatek 7921au chip in it, so I put "7921" into the search field. Ultimately I installed the "kmod-mt7921u" package for the dongle (It includes quite a lot of common dependancies from the Mediatek family).

Because I don't want an open access point, I also need some sort of WiFi security package. For a WiFi client, this would be "wpa_supplicant", however I want to set-up an access point, so I put "hostapd" into the search field so I can select one of the alternatives.

I chose "hostapd-mini" as I only want pre-shared key (PSK) authentication (No need for RADIUS, etc.). Once it's installed, I need to reboot OpenWRT to pick-up the newly installed device driver(s):

I now have a "Network" -> "Wireless" option:

And the WiFi adapter shows up in the Wireless Overview:

I can now go ahead and set-up a new WiFi access point with the (disabled) Master device. This isn't anything particular to the setup I have, so I won't detail it here.

Performance?

Well, it works, but it's not super great. I have a VDSL2 uplink with a line rate of 100M. Right now I get about 70M over the main WiFi connection measured using both the Netflix fast.com service and also speedtest.net:

If I connect over the OpenWRT virtual machine's WiFi access point, I only get about 10M initially (both up and down). However there's an error on the OpenWRT console stating "sched: RT throttling activated". No doubt this would not be an issue for a bare metal (Not virtualised) installation.

I can mitigate this a bit by disabling the realtime throttling by entering "echo -1 >/proc/sys/kernel/sched_rt_runtime_us". While this technically leaves the system open to lock-up if a realtime process goes awry, but it's fine for this testing environment. Ideally I'd need some startup task to disble RT throttling at boot too. Testing now gets me about 30M up and down - Still about half what I should be getting, but about 3 times better than before.

There was one other weird transient error where there would be USB3/XHCI comms errors semi-regularly, spewing lots of errors on the OpenWRT console and resulting in a device reset (and a few seconds of WiFi pause while it did so). This might have been an electrical connectivity error (unplug it and plug it back in?), but is probably just another spooky side-effect of running in a virtualised environment. I've not been able to reproduce this since and I'm not really worried about it.

So, while I might be able to hunt down some more tweaks to make the VM access point perform a bit better, it's really "diminishing returns" at this point. I really just wanted to see if and how I could get the Comfast USB WiFi dongle working with OpenWRT and that has been achieved.

So that's the end of part 2. Part 3 will hopefully be me building/running an Atom specific version of OpenWRT (And dare I hope much better performance?)

UPDATE:

I've added the third part:

• Part 3: Recompiling OpenWrt to run natively on the 3040