nofreebsd_vm

Deploying a non-FreeBSD VM

Deploying a VM with FreeBSD as guest OS with Overlord is easy but the best thing is that it is very flexible: you can create the VM using FreeBSD components or using an AppJail image, customize the environment and much more. Overlord can also deploy a non-FreeBSD VM, say Linux or Windows, but in a less flexible way: running an ISO as a virtual CD to install the OS of your choice and then booting the virtual disk with the OS installed. Of course, this means that we can not only use Linux or Windows, but we can even use FreeBSD or OpenBSD, or any other operating system.

We need to figure out how to get the ISO we want to install. Remember that Overlord relies on the fact that the Makejail creates the isolated environment with vm-bhyve, but assumes that all things are ready to use. So we have several options:

1. Download the ISO file from the Makejail.
2. Mount an external directory in the jail.

I think the first method is good if you use a small OS (for example Alpine Linux), but the problem is when you want to create many VMs with the same OS, then the ISO needs to be downloaded each time unless you use a HTTP client that avoids this, but even if this is the case, you need to mount an external directory and share it between the jails to not download the ISO over and over again. Even taking all this into account, the ISO can easily get corrupted due to external factors when it is downloading, so you need to add more logic to your Makejail to verify the checksums, but if the ISO gets corrupted will you try again?

With all the above I think the best method is the second one. You use an external directory with the ISO already downloaded. The downloading part can easily be done manually as it is not common to download an ISO over and over again. Of course, this directory must be shared between all the jails, so we have many options. We can use nullfs, but since Overlord is a distributed system, it is better to have options that are... distributed, so NFS or SSHFS are good options.

Another thing to keep in mind is the VM template. The vm-bhyve wiki has some good examples for deploying the most common operating systems used by FreeBSD users, but there may be times when you want to use an OS less known to the FreeBSD community, so you are on your own here.... well you are not, you can open a thread on the FreeBSD forums asking if anyone has ever tried to install the OS you want using vm-bhyve.

Having said all that, here is our deployment file for deploying Kicksecure:

kicksecure.yml:

kind: vmJail
datacenters:
  main:
    entrypoint: 'http://127.0.0.1:8888'
    access_token: '<access token>'
deployIn:
  labels:
    - desktop
vmName: 'spo'
makejail: 'gh+DtxdF/vm-makejail'
options:
  - fstab: '"c2c.lan:/var/appjail-volumes/filebrowser/www/Operating Systems" /vm/.iso nfs ro,nfsv4'
template:
  loader: 'uefi'
  cpu: '4'
  memory: '2G'
  graphics: 'yes'
  graphics_listen: '0.0.0.0'
  graphics_res: '1280x720'
  xhci_mouse: 'yes'
  network0_type: 'virtio-net'
  network0_switch: 'public'
  wired_memory: 'YES'
  uefi_vars: 'YES'
diskLayout:
  driver: 'nvme'
  size: '40G'
  from:
    type: 'iso'
    isoFile: 'Kicksecure-Xfce-17.2.8.5.Intel_AMD64.iso'

I have chosen NFSv4 for simplicity. If you choose SSHFS, you need to configure a few more things because the SSH key is a new variable. However, for best performance, you should use a switch or router as your LAN. You should not distribute the ISO over the Internet unless you have a very fast Internet connection with near-zero latency and no collisions. (And you plan to use NFSv4 over the Internet without authentication?)

Deploy:

$ overlord apply -f kicksecure.yml
$ overlord get-info -f kicksecure.yml -t projects --filter-per-project
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: None
  labels:
    - all
    - desktop
    - vm-only
  projects:
    spo:
      state: UNFINISHED
      last_log: 2025-03-14_19h56m40s
      locked: True
      services:
        - {'name': 'vm', 'status': 0, 'jail': 'spo'}
      up:
        operation: RUNNING
        last_update: 38.73 seconds
        job_id: 9

After a while, the VM should be cooked.

$ overlord get-info -f kicksecure.yml -t vm --filter-per-project
datacenter: http://127.0.0.1:8888
  entrypoint: main
  chain: None
  labels:
    - all
    - desktop
    - vm-only
  projects:
    spo:
      virtual-machines:
          operation: COMPLETED
          output: |
            Starting spo
              * found guest in /vm/spo
              * booting...
          last_update: 1 minute and 17.65 seconds
          job_id: 9

But we are not done, we need to use a VNC client and connect to the VM. That is why we need to specify loader: uefi in our template, so that VNC works smoothly.

Connecting...

Boot menu

Configuring virtual network interface (1)

It is not mandatory to install Kicksecure, but I show you the network parameters that I used and that depend on the Makejail (e.g. router).

Configuring virtual network interface (2)

Configuring virtual network interface (3)

Configuring virtual network interface (4)

Testing network connection

Testing DNS

Installing Kicksecure (1)

Installing Kicksecure (2)

Installing Kicksecure (3)

Instead of restarting the virtual machine, shut it down because we need to set a new parameter in our deployment file.

Installing Kicksecure (4)

Installing Kicksecure (5)

After installing the operating system, we need to make a small change to our deployment file:

$ cat kicksecure.yml
...
  from:
    type: 'iso'
    isoFile: 'Kicksecure-Xfce-17.2.8.5.Intel_AMD64.iso'
    installed: true
$ overlord apply -f kicksecure.yml
$ overlord get-info -f kicksecure.yml -t vm --filter-per-project
datacenter: http://127.0.0.1:8888
  entrypoint: main   
  chain: None
  labels:
    - all
    - desktop
    - vm-only
  projects:
    spo:
      virtual-machines:
          operation: COMPLETED
          output: |
            vm_list:  -> spo
            Starting spo
              * found guest in /vm/spo
              * booting...
          last_update: 6.49 seconds
          job_id: 10

With the above configuration we have specified to vm-bhyve not to run the ISO again and instead use the virtual disk.

Done.