-
Notifications
You must be signed in to change notification settings - Fork 2
ephemeral_vm
Jesús Daniel Colmenares Oviedo edited this page Apr 11, 2026
·
5 revisions
ephemeral_vm.mp4
"The Ephemeral Concept" of AppJail is easy to implement in jails due to how easily they are managed, but virtual machines are just another way to achieve this, albeit a more complex one. In this article, we will implement "The Ephemeral Concept" on FreeBSD virtual machines using nbdkit as our server and nbd-client-kmod as our client, so that in the end, everything we store on the NBD device will persist even after recreating the virtual machine, and everything we do not store within this device will be destroyed.
| Document | Implement "The Ephemeral Concept"? | Where? | - |
|---|---|---|---|
| Deploying FreeBSD VMs | No | - | * |
| Deploying virtual machines with ephemeral jails | Yes | Jails | ** |
| Deploying Ephemeral VMs | Yes | Jails, VMs | *** |
As can be seen in the comparison table, with Overlord (and its underlying subsystems), virtual machines can be deployed in three different ways, depending on where "The Ephemeral Concept" is applied:
-
*- When “The Ephemeral Concept” is not implemented, it's much easier to deploy virtual machines, but at the cost of the destruction of the jail (and therefore the virtual machine) being permanent. The other disadvantage is that it's necessary to maintain the jail, updating vm-bhyve(-devel) and any other dependencies, and in the case of updating the jail itself when a new version of FreeBSD is created, this simply complicates things greatly.Choose this option when your workflow is ephemeral and should not persist. Or choose this option when you can recreate your data in some way, such as by restoring it from a backup. However, once you are familiar with all the other options, you will see that this option is disadvantageous.
-
**- When "The Ephemeral Concept" is implemented only in jails, this offers you the advantage that you can recreate your jail and it will be created based on the latest version of your base directory (seeappjail-fetch(1)), plus you don't need to maintain vm-bhyve(-devel) and its dependencies. In this case, the virtual machine image and related data will remain outside the jail.Choose this option when your virtual machine cannot be ephemeral and you do not want to maintain your jail.
-
***- When "The Ephemeral Concept" is implemented on both sides, in jails and virtual machines, and when the jail is destroyed (for example, when recreating it), the virtual machine is also destroyed. However, in this case, the data is stored on a special device where data remains even after the virtual machine is destroyed. The special device can be an NBD device such as the one described in this document, but in theory, 9pfs, SSHFS, NFS, etc. can be used. The downside of this approach is that it increases latency, but this depends entirely on where you place your NBD server, and in the case of this document, the NBD server is close by, as it's deployed on the same host as the virtual machine.Choose this option when you're accustumed to "The Ephemeral Concept" in jails.
The reason for choosing NBD over other options such as SSHFS, 9pfs, NFS, etc., is that with NBD we have a device that we can partition and format like any other non-special device, so we can format it with UFS or ZFS. This also means that we will not have any limitations and, in certain workflows, performance is not seriously affected.
mkdir -p nbdkit
mkdir -p pubVM
mkdir -p /var/cache/pkgcache/FreeBSD:15:amd64
mkdir -p /var/cache/pkgcache/FreeBSD:16:amd64.env:
ENTRYPOINT=<root-chain URL>
TOKEN=<access token>
SSH_KEY=<public SSH key>
nbdkit/metadata.yml:
kind: metadata
datacenters:
main:
entrypoint: !ENV '${ENTRYPOINT}'
access_token: !ENV '${TOKEN}'
deployIn:
labels:
- vm-only
metadataPrefix: nbdkit
metadata:
appConfig: |
<%page args="nbdkit_serial='0',nbdkit_size='10G',nbdkit_fadvise='sequential'"/>
kind: directorProject
projectName: ${appName}
projectFile: |
services:
nbd-server:
<%text filter="n">makejail: !ENV '${OVERLORD_METADATA}/nbdkit.makejail</%text>'
serial: ${nbdkit_serial}
arguments:
- nbdkit_volume: ${appName}
- nbdkit_address: ${nbdkit_address}
- nbdkit_size: ${nbdkit_size}
- nbdkit_fadvise: ${nbdkit_fadvise}
options:
- label: 'appjail.dns.alt-name:${appName}'
volumes:
- data: nbdkit-data
default_volume_type: '<volumefs>'
volumes:
data:
device: /var/appjail-nbd/${appName}
makejail: |
ARG nbdkit_volume
ARG nbdkit_address
ARG nbdkit_size=10G
ARG nbdkit_fadvise=sequential
OPTION start
OPTION overwrite=force
OPTION volume=nbdkit-data mountpoint:/data owner:0 group:0
OPTION virtualnet=:<random> address:${nbdkit_address} default
OPTION nat
INCLUDE gh+DtxdF/efficient-makejail
CMD mkdir -p /usr/local/etc/pkg/repos
COPY ${OVERLORD_METADATA}/nbdkit.pkg.conf /usr/local/etc/pkg/repos/Latest.conf
PKG nbdkit
RAW if appjail cmd jexec "${APPJAIL_JAILNAME}" [ ! -f "/data/${nbdkit_volume}.img" ]; then
CMD truncate -s "${nbdkit_size}" "/data/${nbdkit_volume}.img"
CMD chown nobody:nobody "/data/${nbdkit_volume}.img"
CMD chmod 600 "/data/${nbdkit_volume}.img"
RAW fi
COPY ${OVERLORD_METADATA}/nbdkit.sh /nbdkit.sh
CMD chmod +x /nbdkit.sh
CMD mkdir -p /data/.config
CMD echo "${nbdkit_volume}" > /data/.config/volume
CMD echo "${nbdkit_fadvise}" > /data/.config/fadvise
STOP
STAGE start
CMD /nbdkit.sh
sh: |
#!/bin/sh
set -xe
set -o pipefail
nbdkit_volume=`head -1 -- /data/.config/volume`
nbdkit_fadvise=`head -1 -- /data/.config/fadvise`
daemon \
-f \
-t "nbdkit (volume:${nbdkit_volume})" \
-u nobody \
-p /var/run/nbdkit.pid \
-o /var/log/nbdkit.log \
nbdkit -f file "file=/data/${nbdkit_volume}.img" "fadvise=${nbdkit_fadvise}"
pkg.conf: |
FreeBSD: {
url: "pkg+https://pkg.FreeBSD.org/${ABI}/latest",
mirror_type: "srv",
signature_type: "fingerprints",
fingerprints: "/usr/share/keys/pkg",
enabled: yes
}
FreeBSD-kmods: {
enabled: no
}nbdkit/test-volume.yml:
kind: appConfig
datacenters:
main:
entrypoint: !ENV '${ENTRYPOINT}'
access_token: !ENV '${TOKEN}'
deployIn:
labels:
- desktop
appName: nbdkit-test-volume
appFrom: 'nbdkit.appConfig'
appConfig:
nbdkit_address: 10.0.0.61pubVM/metadata.yml:
kind: metadata
datacenters:
main:
entrypoint: !ENV '${ENTRYPOINT}'
access_token: !ENV '${TOKEN}'
deployIn:
labels:
- vm-only
metadataPrefix: pubVM
metadata:
resolv.conf: |
nameserver 172.16.0.1
loader.conf: |
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
nvme_load="YES"
if_bridge_load="YES"
bridgestp_load="YES"
kern.racct.enable=1
zfs_load="YES"
geom_nbd_load="YES"
sshd_config: |
# Ports
Port 22
# Authentication
PubkeyAuthentication yes
AuthenticationMethods publickey
PermitRootLogin prohibit-password
PrintMotd no
# Forwarding
X11Forwarding no
AllowAgentForwarding yes
# Connection checks
ClientAliveCountMax 3
ClientAliveInterval 15
# Compression
Compression no
# Limits
LoginGraceTime 40
# Public keys
AuthorizedKeysFile /etc/ssh/authorized_keys
# SFTP
Subsystem sftp internal-sftp
sysctl.conf: |
# A bit of hardening
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.see_jail_proc=0
kern.randompid=1
# Allow packet filtering in if_bridge(4)
net.link.bridge.pfil_member=1
net.link.bridge.pfil_bridge=1
FreeBSD-base.conf: |
FreeBSD-base {
url = "pkg+https://pkg.freebsd.org/${ABI}/base_latest";
mirror_type = "srv";
signature_type = "fingerprints";
fingerprints = "/usr/share/keys/pkg";
enabled = yes;
}
mountdata.rc: |
#!/bin/sh
# PROVIDE: mountdata
# REQUIRE: zpool zfs gnbd
. /etc/rc.subr
name="mountdata"
rcvar="${name}_enable"
start_cmd="mountdata_start"
load_rc_config $name
: ${mountdata_enable:="NO"}
: ${mountdata_pool:="zdata"}
mountdata_start()
{
zpool import -f -o cachefile=none "${mountdata_pool}" || return $?
if [ -f "${firstboot_sentinel}" ]; then
zpool upgrade "${mountdata_pool}" || return $?
fi
}
run_rc_command "$1"
appConfig: |
<%page args="poweroff='0',timezone='UTC',expose='0',iface='tailscale0'"/>
kind: 'vmJail'
vmName: ${appName}
makejail: 'gh+DtxdF/vm-makejail'
overwrite: true
build-arguments:
- is_public_vm: '1'
- exclude_ports: '10000:65535'
% if poweroff != "0":
poweroff: true
% endif
options:
- fstab: '/var/cache/pkg /var/cache/pkg'
- priority: '5'
% if expose != "0":
- expose: '${expose}:22 ext_if:${iface} on_if:${iface}'
% endif
template:
loader: 'bhyveload'
cpu: '${ncpu}'
memory: '${memory}'
network0_type: 'virtio-net'
network0_switch: 'public'
wired_memory: 'YES'
diskLayout:
driver: 'nvme'
size: '${disk}'
from:
type: 'pkgbase'
packages:
- FreeBSD-set-minimal
- FreeBSD-kernel-generic
- FreeBSD-ssh
osArch: amd64
osVersion: '${fbsdver}'
disk:
scheme: 'gpt'
partitions:
- type: 'freebsd-boot'
size: '512k'
alignment: '1m'
- type: 'freebsd-swap'
size: '${swap}'
alignment: '1m'
- type: 'freebsd-zfs'
alignment: '1m'
format:
script: |
<%text filter="n">
set -xe
set -o pipefail
POOL="tank"
if [ "${OVERLORD_PHASE}" = "format" ]; then
zpool create -o altroot="${OVERLORD_MNTDIR}" -O compress=lz4 -O atime=off -m none -t "${OVERLORD_UUID}" ${POOL} "${OVERLORD_DEVICE}"
zfs create -o mountpoint=none ${OVERLORD_UUID}/ROOT
zfs create -o mountpoint=/ ${OVERLORD_UUID}/ROOT/default
zfs create -o mountpoint=/home ${OVERLORD_UUID}/home
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off ${OVERLORD_UUID}/tmp
chmod 1777 "${OVERLORD_MNTDIR}/tmp"
zfs create -o mountpoint=/usr -o canmount=off ${OVERLORD_UUID}/usr
zfs create -o setuid=off ${OVERLORD_UUID}/usr/ports
zfs create ${OVERLORD_UUID}/usr/src
zfs create -o mountpoint=/var -o canmount=off ${OVERLORD_UUID}/var
zfs create -o exec=off -o setuid=off ${OVERLORD_UUID}/var/audit
zfs create -o exec=off -o setuid=off ${OVERLORD_UUID}/var/crash
zfs create -o exec=off -o setuid=off ${OVERLORD_UUID}/var/log
zfs create -o atime=on ${OVERLORD_UUID}/var/mail
zfs create -o setuid=off ${OVERLORD_UUID}/var/tmp
chmod 1777 "${OVERLORD_MNTDIR}/var/tmp"
zfs set mountpoint=/${POOL} ${OVERLORD_UUID}
zpool set bootfs=${OVERLORD_UUID}/ROOT/default ${OVERLORD_UUID}
mkdir -p "${OVERLORD_MNTDIR}/boot/zfs"
zpool set cachefile="${OVERLORD_MNTDIR}/boot/zfs/zpool.cache" ${OVERLORD_UUID}
zfs set canmount=noauto ${OVERLORD_UUID}/ROOT/default
zpool export "${OVERLORD_UUID}"
elif [ "${OVERLORD_PHASE}" = "mount" ]; then
zpool import -t -R "${OVERLORD_MNTDIR}" -d "${OVERLORD_DEVICE}" "${POOL}" "${OVERLORD_UUID}"
zfs mount "${OVERLORD_UUID}/ROOT/default"
mkdir -p "${OVERLORD_MNTDIR}/var/cache/pkg"
</%text>
mount -t nullfs "/var/cache/pkgcache/FreeBSD:${fbsdver}:amd64" <%text filter="n">"${OVERLORD_MNTDIR}/var/cache/pkg"</%text>
<%text filter="n">
elif [ "${OVERLORD_PHASE}" = "umount" ]; then
umount "${OVERLORD_MNTDIR}/var/cache/pkg"
zpool export "${OVERLORD_UUID}"
fi
</%text>
bootcode:
bootcode: '/boot/pmbr'
partcode: '/boot/gptzfsboot'
index: 1
fstab:
- device: '/dev/nda0p2'
mountpoint: 'none'
type: 'swap'
options: 'sw'
dump: 0
pass: 0
script-environment:
- HOSTNAME: '${hostname}'
- TIMEZONE: '${timezone}'
- SSH_KEY: '${ssh_key}'
- VOLUME: '${volume}'
- ASSUME_ALWAYS_YES: 'yes'
- IGNORE_OSVERSION: 'yes'
- PKG_ABI: 'FreeBSD:${fbsdver}:amd64'
script: |
<%text filter="n">
set -xe
set -o pipefail
. "/metadata/environment"
sysrc -f /mnt/etc/rc.conf ifconfig_vtnet0="inet 192.168.8.2/24"
sysrc -f /mnt/etc/rc.conf defaultrouter="192.168.8.1"
sysrc -f /mnt/etc/rc.conf fsck_y_enable="YES"
sysrc -f /mnt/etc/rc.conf clear_tmp_enable="YES"
sysrc -f /mnt/etc/rc.conf dumpdev="NO"
sysrc -f /mnt/etc/rc.conf moused_nondefault_enable="NO"
sysrc -f /mnt/etc/rc.conf hostname="${HOSTNAME}"
if [ -f "/metadata/pubVM.resolv.conf" ]; then
cp -a /metadata/pubVM.resolv.conf /mnt/etc/resolv.conf
fi
if [ -f "/metadata/pubVM.loader.conf" ]; then
cp /metadata/pubVM.loader.conf /mnt/boot/loader.conf
fi
ln -fs "/usr/share/zoneinfo/${TIMEZONE}" /mnt/etc/localtime
if [ -f "/metadata/pubVM.sshd_config" ]; then
sysrc -f /mnt/etc/rc.conf sshd_enable="YES"
cp /metadata/pubVM.sshd_config /mnt/etc/ssh/sshd_config
fi
sysrc -f /mnt/etc/rc.conf zfs_enable="YES"
echo "${SSH_KEY}" > /mnt/etc/ssh/authorized_keys
if [ -f "/metadata/pubVM.sysctl.conf" ]; then
cp /metadata/pubVM.sysctl.conf /mnt/etc/sysctl.conf
fi
if [ -f "/metadata/pubVM.FreeBSD-base.conf" ]; then
mkdir -p /mnt/usr/local/etc/pkg/repos
cp /metadata/pubVM.FreeBSD-base.conf /mnt/usr/local/etc/pkg/repos/FreeBSD-base.conf
fi
if [ -n "${VOLUME}" ]; then
pkg bootstrap
env ABI="${PKG_ABI}" pkg --chroot /mnt update -r FreeBSD-ports
env ABI="${PKG_ABI}" pkg --chroot /mnt install -Ur FreeBSD-ports nbd-client-kmod
sysrc -f /mnt/etc/rc.conf gnbd_enable="YES"
sysrc -f /mnt/etc/rc.conf gnbd_devices="nbd0"
sysrc -f /mnt/etc/rc.conf gnbd_nbd0_host="${VOLUME}"
mkdir -p /mnt/usr/local/etc/rc.d
cp /metadata/pubVM.mountdata.rc /mnt/usr/local/etc/rc.d/mountdata
chmod +x /mnt/usr/local/etc/rc.d/mountdata
sysrc -f /mnt/etc/rc.conf mountdata_enable="YES"
fi
sysrc -f /mnt/etc/rc.conf zpool_upgrade+="tank"
touch /mnt/firstboot
</%text>
metadataPrefix: pubVM
metadata:
- resolv.conf
- loader.conf
- sshd_config
- sysctl.conf
- FreeBSD-base.conf
- mountdata.rcpubVM/bsdtest.yml:
kind: appConfig
datacenters:
main:
entrypoint: !ENV '${ENTRYPOINT}'
access_token: !ENV '${TOKEN}'
deployIn:
labels:
- desktop
appName: 'bsdtest'
appFrom: 'pubVM.appConfig'
appConfig:
ncpu: '2'
memory: '2G'
disk: '10G'
swap: '2G'
fbsdver: '15'
hostname: 'bsdtest'
timezone: 'America/Caracas'
ssh_key: !ENV '${SSH_KEY}'
volume: 'nbdkit-test-volume'
Notes:
- We use appConfigs to create templates for the different workflows we might have.
- Priority is set to
5for vmJails because nbdkit must start before the virtual machines. - PkgBase is used to provision the virtual machine, but only a minimal system is created. If you want to create a more complete system, choose the
FreeBSD-set-basepackage. - Attentive readers will have noticed that this is a public virtual machine due to the arguments used. All packets passed to the jail will be redirected to the virtual machine, except for ports
10000to65535(due tonet.inet.ip.portrange.firstandnet.inet.ip.portrange.last, which are used by applications that connect to servers within the jail, such aspkg(8), which is the one we will use). This does not mean that, by default, all packets from the host are redirected to the jail (and therefore to the virtual machine). It requires you to explicitly expose the ports like any other jail. See theexposeparameter in theappConfigtemplate of thepubVM/metadata.ymlfile to expose SSH. -
/var/cache/pkgare mounted inside the jail using the same path, which greatly speeds up recreation. Please note that you should only do this when two requirements are met: 1. The jail's ABI is the same as the host, 2. Both the host and jails use the same branch (latest). If you do not meet these requirements, use another directory to avoid potential conflicts with binaries that do not match your host's ABI and branch. -
/var/cache/pkgcache/${ABI}is used to cache PkgBase packages in order to speed up vmJail recreation when using PkgBase to provision the virtual machine. This is similar to step 5, however, the requirements are more relaxed because we use a different directory for each ABI. - A static IP is reserved on the virtual network for nbdkit; however, as we do in this document, you can use a domain name to connect to nbdkit, but this, of course, requires that a DNS server be configured for AppJail.
- The first partition of the NBD device is used to host ZFS, although you can modify the rc script
mountdata.rcto use another partition or the entire device, if you prefer. - ZFS is used everywhere, including the NBD device. See below.
overlord apply -f nbdkit/metadata.yml
overlord apply -f nbdkit/test-volume.ymlAs mentioned, the first partition of the NBD device is used to host ZFS, so we are going to format it. First, install the nbd-client-kmod package to create our NBD device. This can be done on our host.
# host nbdkit-test-volume
nbdkit-test-volume has address 10.0.0.61
# pkg install -y nbd-client-kmod
...
# gnbd load
# gnbd connect nbdkit-test-volume
nbd0
# gpart create -s gpt nbd0
nbd0 created
# gpart add -t freebsd-zfs -a 1m nbd0
nbd0p1 added
# zpool create -f -O compress=lz4 -t zdata zdata /dev/nbd0p1
# zpool list zdata
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zdata 9.50G 432K 9.50G - - 0% 0% 1.00x ONLINE -
# zpool export zdata
# gnbd disconnect nbd0Let's deploy our VM.
$ overlord apply -f pubVM/metadata.yml
$ overlord apply -f pubVM/bsdtest.yml
$ overlord get-info -f pubVM/bsdtest.yml -t projects --filter-per-project
datacenter: http://controller.namespace.lan:8888
entrypoint: main
chain: None
labels:
- all
- desktop
- services
- vm-only
- dc-air
projects:
bsdtest:
state: DONE
last_log: 2025-10-28_18h56m25s
locked: False
services:
- {'name': 'vm', 'status': 0, 'jail': 'bsdtest'}
up:
operation: COMPLETED
output:
rc: 0
stdout: {'errlevel': 0, 'message': None, 'failed': []}
last_update: 4 minutes and 24.6 seconds
job_id: 1
restarted: False
$ overlord get-info -f pubVM/bsdtest.yml -t vm --filter-per-project
datacenter: http://controller.namespace.lan:8888
entrypoint: main
chain: None
labels:
- all
- desktop
- services
- vm-only
- dc-air
projects:
bsdtest:
virtual-machines:
operation: COMPLETED
output: |
md0 created
md0p1 added
md0p2 added
md0p3 added
+ set -o pipefail
+ POOL=tank
+ [ format '=' format ]
+ zpool create -o 'altroot=/usr/local/appjail/jails/bsdtest/jail/mnt' -O 'compress=lz4' -O 'atime=off' -m none -t overlord-0059dd52-4743-41fc-97ac-ca001e3c7321 tank /dev/md0p3
+ zfs create -o 'mountpoint=none' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/ROOT
+ zfs create -o 'mountpoint=/' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/ROOT/default
+ zfs create -o 'mountpoint=/home' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/home
+ zfs create -o 'mountpoint=/tmp' -o 'exec=on' -o 'setuid=off' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/tmp
+ chmod 1777 /usr/local/appjail/jails/test-vm/jail/mnt/tmp
+ zfs create -o 'mountpoint=/usr' -o 'canmount=off' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/usr
+ zfs create -o 'setuid=off' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/usr/ports
+ zfs create overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/usr/src
+ zfs create -o 'mountpoint=/var' -o 'canmount=off' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/var
+ zfs create -o 'exec=off' -o 'setuid=off' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/var/audit
+ zfs create -o 'exec=off' -o 'setuid=off' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/var/crash
+ zfs create -o 'exec=off' -o 'setuid=off' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/var/log
+ zfs create -o 'atime=on' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/var/mail
+ zfs create -o 'setuid=off' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/var/tmp
+ chmod 1777 /usr/local/appjail/jails/test-vm/jail/mnt/var/tmp
+ zfs set 'mountpoint=/tank' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321
+ zpool set 'bootfs=overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/ROOT/default' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321
+ mkdir -p /usr/local/appjail/jails/bsdtest/jail/mnt/boot/zfs
+ zpool set 'cachefile=/usr/local/appjail/jails/bsdtest/jail/mnt/boot/zfs/zpool.cache' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321
+ zfs set 'canmount=noauto' overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/ROOT/default
+ zpool export overlord-0059dd52-4743-41fc-97ac-ca001e3c7321
bootcode written to md0
partcode written to md0p1
+ set -o pipefail
+ POOL=tank
+ [ mount '=' format ]
+ [ mount '=' mount ]
+ zpool import -t -R /usr/local/appjail/jails/bsdtest/jail/mnt -d /dev/md0p3 tank overlord-0059dd52-4743-41fc-97ac-ca001e3c7321
+ zfs mount overlord-0059dd52-4743-41fc-97ac-ca001e3c7321/ROOT/default
+ mkdir -p /usr/local/appjail/jails/bsdtest/jail/mnt/var/cache/pkg
+ mount -t nullfs /var/cache/pkgcache/FreeBSD:15:amd64 /usr/local/appjail/jails/bsdtest/jail/mnt/var/cache/pkg
pkg: Setting ABI requires setting OSVERSION, guessing the OSVERSION as: 1500000
pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
Updating FreeBSD-base repository catalogue...
Fetching meta.conf: . done
Fetching data.pkg: ..... done
Processing entries: .......... done
FreeBSD-base repository update completed. 490 packages processed.
All repositories are up to date.
The following 61 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
FreeBSD-at: 15.snap20251023095006
FreeBSD-audit-lib: 15.snap20251023095006
FreeBSD-blocklist: 15.snap20251023095006
FreeBSD-bootloader: 15.snap20251023095006
FreeBSD-bzip2: 15.snap20251015201327
FreeBSD-bzip2-lib: 15.snap20251011015136
FreeBSD-caroot: 15.snap20251023234421
FreeBSD-certctl: 15.snap20251023095006
FreeBSD-clibs: 15.snap20251027162413
FreeBSD-cron: 15.snap20251023095006
FreeBSD-ctf-lib: 15.snap20251023095006
FreeBSD-devd: 15.snap20251023095006
FreeBSD-devmatch: 15.snap20251023095006
FreeBSD-dhclient: 15.snap20251023095006
FreeBSD-efi-tools: 15.snap20251023095006
FreeBSD-fetch: 15.snap20251023095006
FreeBSD-firmware-iwm: 15.snap20251011015136
FreeBSD-fwget: 15.snap20251020202731
FreeBSD-geom: 15.snap20251023095006
FreeBSD-hyperv-tools: 15.snap20251023095006
FreeBSD-kerberos-lib: 15.snap20251023095006
FreeBSD-kernel-generic: 15.snap20251028095230
FreeBSD-kernel-man: 15.snap20251028023512
FreeBSD-libarchive: 15.snap20251028074900
FreeBSD-libbsdstat: 15.snap20251011015136
FreeBSD-libcasper: 15.snap20251023095006
FreeBSD-libldns: 15.snap20251023095006
FreeBSD-libmagic: 15.snap20251023095006
FreeBSD-libucl: 15.snap20251023095006
FreeBSD-libyaml: 15.snap20251011015136
FreeBSD-locales: 15.snap20251011015136
FreeBSD-mandoc: 15.snap20251023095006
FreeBSD-natd: 15.snap20251023095006
FreeBSD-ncurses: 15.snap20251023004132
FreeBSD-ncurses-lib: 15.snap20251023095006
FreeBSD-newsyslog: 15.snap20251023095006
FreeBSD-nuageinit: 15.snap20251011015136
FreeBSD-openssl: 15.snap20251023095006
FreeBSD-openssl-lib: 15.snap20251023095006
FreeBSD-periodic: 15.snap20251014010208
FreeBSD-pkg-bootstrap: 15.snap20251023095006
FreeBSD-powerd: 15.snap20251023095006
FreeBSD-ppp: 15.snap20251023095006
FreeBSD-rc: 15.snap20251020174900
FreeBSD-rescue: 15.snap20251028074900
FreeBSD-resolvconf: 15.snap20251011015136
FreeBSD-runtime: 15.snap20251028074900
FreeBSD-set-minimal: 15.snap20251015130244
FreeBSD-ssh: 15.snap20251023095006
FreeBSD-syslogd: 15.snap20251023095006
FreeBSD-tcpd: 15.snap20251023095006
FreeBSD-ufs: 15.snap20251028095230
FreeBSD-ufs-lib: 15.snap20251023095006
FreeBSD-utilities: 15.snap20251028111053
FreeBSD-vi: 15.snap20251023095006
FreeBSD-vt-data: 15.snap20251011015136
FreeBSD-wpa: 15.snap20251023095006
FreeBSD-xz-lib: 15.snap20251023095006
FreeBSD-zfs: 15.snap20251025084859
FreeBSD-zfs-lib: 15.snap20251025092321
FreeBSD-zoneinfo: 15.snap20251011015136
Number of packages to be installed: 61
The process will require 331 MiB more space.
67 MiB to be downloaded.
[1/7] Fetching FreeBSD-rescue-15.snap20251028074900.pkg: .......... done
[2/7] Fetching FreeBSD-runtime-15.snap20251028074900.pkg: .......... done
[3/7] Fetching FreeBSD-libarchive-15.snap20251028074900.pkg: .......... done
[4/7] Fetching FreeBSD-kernel-generic-15.snap20251028095230.pkg: .......... done
[5/7] Fetching FreeBSD-kernel-man-15.snap20251028023512.pkg: .......... done
[6/7] Fetching FreeBSD-ufs-15.snap20251028095230.pkg: ......... done
[7/7] Fetching FreeBSD-utilities-15.snap20251028111053.pkg: .......... done
Checking integrity... done (0 conflicting)
[1/61] Installing FreeBSD-audit-lib-15.snap20251023095006...
[1/61] Extracting FreeBSD-audit-lib-15.snap20251023095006: . done
[2/61] Installing FreeBSD-blocklist-15.snap20251023095006...
[2/61] Extracting FreeBSD-blocklist-15.snap20251023095006: .......... done
[3/61] Installing FreeBSD-bootloader-15.snap20251023095006...
[3/61] Extracting FreeBSD-bootloader-15.snap20251023095006: ......... done
[4/61] Installing FreeBSD-bzip2-15.snap20251015201327...
[4/61] Extracting FreeBSD-bzip2-15.snap20251015201327: ........ done
[5/61] Installing FreeBSD-bzip2-lib-15.snap20251011015136...
[5/61] Extracting FreeBSD-bzip2-lib-15.snap20251011015136: . done
[6/61] Installing FreeBSD-clibs-15.snap20251027162413...
[6/61] Extracting FreeBSD-clibs-15.snap20251027162413: .......... done
[7/61] Installing FreeBSD-cron-15.snap20251023095006...
[7/61] Extracting FreeBSD-cron-15.snap20251023095006: ......... done
[8/61] Installing FreeBSD-ctf-lib-15.snap20251023095006...
[8/61] Extracting FreeBSD-ctf-lib-15.snap20251023095006: . done
[9/61] Installing FreeBSD-devmatch-15.snap20251023095006...
[9/61] Extracting FreeBSD-devmatch-15.snap20251023095006: .......... done
[10/61] Installing FreeBSD-dhclient-15.snap20251023095006...
[10/61] Extracting FreeBSD-dhclient-15.snap20251023095006: .......... done
[11/61] Installing FreeBSD-efi-tools-15.snap20251023095006...
[11/61] Extracting FreeBSD-efi-tools-15.snap20251023095006: .......... done
[12/61] Installing FreeBSD-fetch-15.snap20251023095006...
[12/61] Extracting FreeBSD-fetch-15.snap20251023095006: .......... done
[13/61] Installing FreeBSD-firmware-iwm-15.snap20251011015136...
[13/61] Extracting FreeBSD-firmware-iwm-15.snap20251011015136: ......... done
[14/61] Installing FreeBSD-fwget-15.snap20251020202731...
[14/61] Extracting FreeBSD-fwget-15.snap20251020202731: .......... done
[15/61] Installing FreeBSD-geom-15.snap20251023095006...
[15/61] Extracting FreeBSD-geom-15.snap20251023095006: .......... done
[16/61] Installing FreeBSD-hyperv-tools-15.snap20251023095006...
[16/61] Extracting FreeBSD-hyperv-tools-15.snap20251023095006: .......... done
[17/61] Installing FreeBSD-kerberos-lib-15.snap20251023095006...
[17/61] Extracting FreeBSD-kerberos-lib-15.snap20251023095006: .......... done
[18/61] Installing FreeBSD-kernel-generic-15.snap20251028095230...
[18/61] Extracting FreeBSD-kernel-generic-15.snap20251028095230: .......... done
[19/61] Installing FreeBSD-kernel-man-15.snap20251028023512...
[19/61] Extracting FreeBSD-kernel-man-15.snap20251028023512: .......... done
[20/61] Installing FreeBSD-libarchive-15.snap20251028074900...
[20/61] Extracting FreeBSD-libarchive-15.snap20251028074900: .......... done
[21/61] Installing FreeBSD-libbsdstat-15.snap20251011015136...
[21/61] Extracting FreeBSD-libbsdstat-15.snap20251011015136: . done
[22/61] Installing FreeBSD-libcasper-15.snap20251023095006...
[22/61] Extracting FreeBSD-libcasper-15.snap20251023095006: .......... done
[23/61] Installing FreeBSD-libldns-15.snap20251023095006...
[23/61] Extracting FreeBSD-libldns-15.snap20251023095006: . done
[24/61] Installing FreeBSD-libmagic-15.snap20251023095006...
[24/61] Extracting FreeBSD-libmagic-15.snap20251023095006: ...... done
[25/61] Installing FreeBSD-libucl-15.snap20251023095006...
[25/61] Extracting FreeBSD-libucl-15.snap20251023095006: .. done
[26/61] Installing FreeBSD-libyaml-15.snap20251011015136...
[26/61] Extracting FreeBSD-libyaml-15.snap20251011015136: . done
[27/61] Installing FreeBSD-locales-15.snap20251011015136...
[27/61] Extracting FreeBSD-locales-15.snap20251011015136: ......... done
[28/61] Installing FreeBSD-mandoc-15.snap20251023095006...
[28/61] Extracting FreeBSD-mandoc-15.snap20251023095006: .......... done
[29/61] Installing FreeBSD-natd-15.snap20251023095006...
[29/61] Extracting FreeBSD-natd-15.snap20251023095006: .......... done
[30/61] Installing FreeBSD-ncurses-15.snap20251023004132...
[30/61] Extracting FreeBSD-ncurses-15.snap20251023004132: .......... done
[31/61] Installing FreeBSD-ncurses-lib-15.snap20251023095006...
[31/61] Extracting FreeBSD-ncurses-lib-15.snap20251023095006: ..... done
[32/61] Installing FreeBSD-newsyslog-15.snap20251023095006...
[32/61] Extracting FreeBSD-newsyslog-15.snap20251023095006: ....... done
[33/61] Installing FreeBSD-nuageinit-15.snap20251011015136...
[33/61] Extracting FreeBSD-nuageinit-15.snap20251011015136: ....... done
[34/61] Installing FreeBSD-openssl-15.snap20251023095006...
[34/61] Extracting FreeBSD-openssl-15.snap20251023095006: .......... done
[35/61] Installing FreeBSD-certctl-15.snap20251023095006...
[35/61] Extracting FreeBSD-certctl-15.snap20251023095006: .. done
[36/61] Installing FreeBSD-caroot-15.snap20251023234421...
[36/61] Extracting FreeBSD-caroot-15.snap20251023234421: .......... done
[37/61] Installing FreeBSD-openssl-lib-15.snap20251023095006...
[37/61] Extracting FreeBSD-openssl-lib-15.snap20251023095006: ....... done
[38/61] Installing FreeBSD-periodic-15.snap20251014010208...
[38/61] Extracting FreeBSD-periodic-15.snap20251014010208: ......... done
[39/61] Installing FreeBSD-pkg-bootstrap-15.snap20251023095006...
[39/61] Extracting FreeBSD-pkg-bootstrap-15.snap20251023095006: ..... done
[40/61] Installing FreeBSD-powerd-15.snap20251023095006...
[40/61] Extracting FreeBSD-powerd-15.snap20251023095006: ... done
[41/61] Installing FreeBSD-ppp-15.snap20251023095006...
[41/61] Extracting FreeBSD-ppp-15.snap20251023095006: ......... done
[42/61] Installing FreeBSD-rescue-15.snap20251028074900...
[42/61] Extracting FreeBSD-rescue-15.snap20251028074900: .......... done
[43/61] Installing FreeBSD-resolvconf-15.snap20251011015136...
[43/61] Extracting FreeBSD-resolvconf-15.snap20251011015136: .......... done
[44/61] Installing FreeBSD-runtime-15.snap20251028074900...
[44/61] Extracting FreeBSD-runtime-15.snap20251028074900: .......... done
[45/61] Installing FreeBSD-at-15.snap20251023095006...
[45/61] Extracting FreeBSD-at-15.snap20251023095006: ......... done
[46/61] Installing FreeBSD-devd-15.snap20251023095006...
[46/61] Extracting FreeBSD-devd-15.snap20251023095006: .......... done
[47/61] Installing FreeBSD-rc-15.snap20251020174900...
[47/61] Extracting FreeBSD-rc-15.snap20251020174900: .......... done
[48/61] Installing FreeBSD-ssh-15.snap20251023095006...
[48/61] Extracting FreeBSD-ssh-15.snap20251023095006: .......... done
[49/61] Installing FreeBSD-syslogd-15.snap20251023095006...
[49/61] Extracting FreeBSD-syslogd-15.snap20251023095006: ...... done
[50/61] Installing FreeBSD-tcpd-15.snap20251023095006...
[50/61] Extracting FreeBSD-tcpd-15.snap20251023095006: .......... done
[51/61] Installing FreeBSD-ufs-15.snap20251028095230...
[51/61] Extracting FreeBSD-ufs-15.snap20251028095230: .......... done
[52/61] Installing FreeBSD-ufs-lib-15.snap20251023095006...
[52/61] Extracting FreeBSD-ufs-lib-15.snap20251023095006: . done
[53/61] Installing FreeBSD-utilities-15.snap20251028111053...
[53/61] Extracting FreeBSD-utilities-15.snap20251028111053: .......... done
[54/61] Installing FreeBSD-vi-15.snap20251023095006...
[54/61] Extracting FreeBSD-vi-15.snap20251023095006: .......... done
[55/61] Installing FreeBSD-vt-data-15.snap20251011015136...
[55/61] Extracting FreeBSD-vt-data-15.snap20251011015136: .......... done
[56/61] Installing FreeBSD-wpa-15.snap20251023095006...
[56/61] Extracting FreeBSD-wpa-15.snap20251023095006: ......... done
[57/61] Installing FreeBSD-xz-lib-15.snap20251023095006...
[57/61] Extracting FreeBSD-xz-lib-15.snap20251023095006: . done
[58/61] Installing FreeBSD-zfs-15.snap20251025084859...
[58/61] Extracting FreeBSD-zfs-15.snap20251025084859: .......... done
[59/61] Installing FreeBSD-zfs-lib-15.snap20251025092321...
[59/61] Extracting FreeBSD-zfs-lib-15.snap20251025092321: .......... done
[60/61] Installing FreeBSD-zoneinfo-15.snap20251011015136...
[60/61] Extracting FreeBSD-zoneinfo-15.snap20251011015136: .......... done
[61/61] Installing FreeBSD-set-minimal-15.snap20251015130244...
+ set -o pipefail
+ . /metadata/environment
+ export 'HOSTNAME=bsdtest'
+ export 'TIMEZONE=America/Caracas'
+ export 'SSH_KEY=ssh-ed25519 ...'
+ export 'VOLUME=nbdkit-test-volume'
+ export 'ASSUME_ALWAYS_YES=yes'
+ export 'IGNORE_OSVERSION=yes'
+ export 'PKG_ABI=FreeBSD:15:amd64'
+ sysrc -f /mnt/etc/rc.conf 'ifconfig_vtnet0=inet 192.168.8.2/24'
ifconfig_vtnet0: -> inet 192.168.8.2/24
+ sysrc -f /mnt/etc/rc.conf 'defaultrouter=192.168.8.1'
defaultrouter: NO -> 192.168.8.1
+ sysrc -f /mnt/etc/rc.conf 'fsck_y_enable=YES'
fsck_y_enable: NO -> YES
+ sysrc -f /mnt/etc/rc.conf 'clear_tmp_enable=YES'
clear_tmp_enable: NO -> YES
+ sysrc -f /mnt/etc/rc.conf 'dumpdev=NO'
dumpdev: NO -> NO
+ sysrc -f /mnt/etc/rc.conf 'moused_nondefault_enable=NO'
moused_nondefault_enable: YES -> NO
+ sysrc -f /mnt/etc/rc.conf 'hostname=bsdtest'
hostname: -> bsdtest
+ [ -f /metadata/pubVM.resolv.conf ]
+ cp -a /metadata/pubVM.resolv.conf /mnt/etc/resolv.conf
+ [ -f /metadata/pubVM.loader.conf ]
+ cp /metadata/pubVM.loader.conf /mnt/boot/loader.conf
+ ln -fs /usr/share/zoneinfo/America/Caracas /mnt/etc/localtime
+ [ -f /metadata/pubVM.sshd_config ]
+ sysrc -f /mnt/etc/rc.conf 'sshd_enable=YES'
sshd_enable: NO -> YES
+ cp /metadata/pubVM.sshd_config /mnt/etc/ssh/sshd_config
+ sysrc -f /mnt/etc/rc.conf 'zfs_enable=YES'
zfs_enable: NO -> YES
+ echo 'ssh-ed25519 ...'
+ [ -f /metadata/pubVM.sysctl.conf ]
+ cp /metadata/pubVM.sysctl.conf /mnt/etc/sysctl.conf
+ [ -f /metadata/pubVM.FreeBSD-base.conf ]
+ mkdir -p /mnt/usr/local/etc/pkg/repos
+ cp /metadata/pubVM.FreeBSD-base.conf /mnt/usr/local/etc/pkg/repos/FreeBSD-base.conf
+ [ -n nbdkit-test-volume ]
+ pkg bootstrap
[bsdtest.appjail] Installing pkg-2.3.1_1...
[bsdtest.appjail] Extracting pkg-2.3.1_1: .......... done
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
+ env 'ABI=FreeBSD:15:amd64' pkg --chroot /mnt update -r FreeBSD-ports
pkg: Setting ABI requires setting OSVERSION, guessing the OSVERSION as: 1500000
pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
Updating FreeBSD-ports repository catalogue...
[bsdtest.appjail] Fetching meta.conf: . done
[bsdtest.appjail] Fetching data.pkg: .......... done
Processing entries: .......... done
FreeBSD-ports repository update completed. 36578 packages processed.
FreeBSD-ports is up to date.
+ env 'ABI=FreeBSD:15:amd64' pkg --chroot /mnt install -Ur FreeBSD-ports nbd-client-kmod
pkg: Setting ABI requires setting OSVERSION, guessing the OSVERSION as: 1500000
pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
nbd-client-kmod: g20251024.1500068 [FreeBSD-ports]
Number of packages to be installed: 1
[bsdtest.appjail] [1/1] Installing nbd-client-kmod-g20251024.1500068...
[bsdtest.appjail] [1/1] Extracting nbd-client-kmod-g20251024.1500068: ......... done
+ sysrc -f /mnt/etc/rc.conf 'gnbd_enable=YES'
gnbd_enable: -> YES
+ sysrc -f /mnt/etc/rc.conf 'gnbd_devices=nbd0'
gnbd_devices: -> nbd0
+ sysrc -f /mnt/etc/rc.conf 'gnbd_nbd0_host=nbdkit-test-volume'
gnbd_nbd0_host: -> nbdkit-test-volume
+ mkdir -p /mnt/usr/local/etc/rc.d
+ cp /metadata/pubVM.mountdata.rc /mnt/usr/local/etc/rc.d/mountdata
+ chmod +x /mnt/usr/local/etc/rc.d/mountdata
+ sysrc -f /mnt/etc/rc.conf 'mountdata_enable=YES'
mountdata_enable: -> YES
+ sysrc -f /mnt/etc/rc.conf 'zpool_upgrade+=tank'
zpool_upgrade: -> tank
+ touch /mnt/firstboot
+ set -o pipefail
+ POOL=tank
+ [ umount '=' format ]
+ [ umount '=' mount ]
+ [ umount '=' umount ]
+ umount /usr/local/appjail/jails/bsdtest/jail/mnt/var/cache/pkg
+ zpool export overlord-0059dd52-4743-41fc-97ac-ca001e3c7321
vm_list: -> bsdtest
Starting bsdtest
* found guest in /vm/bsdtest
* booting...
last_update: 1 minute and 25.13 seconds
job_id: 1The virtual machine is deployed on my own PC, so I don't need to expose SSH or use a public IP address. I can use the jail's IPv4 address.
$ cat ~/.ssh/config
...
Host control-bsdtest
HostName bsdtest
User root
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
$ ssh control-bsdtest
Warning: Permanently added 'bsdtest' (ED25519) to the list of known hosts.
root@bsdtest:~ # freebsd-version
15.0-STABLE
root@bsdtest:~ # zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 7.50G 237M 7.27G - - 1% 3% 1.00x ONLINE -
zdata 9.50G 612K 9.50G - - 0% 0% 1.00x ONLINE -
root@bsdtest:~ # echo "Hi again!" > /zdata/README.txt
root@bsdtest:~ # ^D
Shared connection to bsdtest closed.
$ overlord apply -f pubVM/bsdtest.yml
$ ssh control-bsdtest ls /zdata
README.txt
$ ssh control-bsdtest cat /zdata/README.txt
Hi again!
$ overlord destroy -Ff pubVM/bsdtest.yml