Skip to content

DebugCollector: first class file archival #9555

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
davepacheco wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

DebugCollector: first class file archival #9555

davepacheco wants to merge 5 commits into from

Conversation

@davepacheco
Copy link
Collaborator

@davepacheco davepacheco commented Dec 20, 2025

This PR adds a more first-class file archival mechanism inside the debug collector within sled agent. The reason I did this is that in the past when I wanted to modify the files that Sled Agent collects, I found it tricky to do because:

  • it's hard to add support for collecting new kinds of files without touching the code related to collecting other files
  • that code combines both decision-making and execution logic, which makes it tricky to change
  • the tests for it are good end-to-end tests but very coarse

Altogether I basically felt like even when making a pretty small change, you essentially had to test in a real deployment, which is a much slower dev workflow than it needs to be (and it'd be very easy to break without breaking CI).

This is coming up because I'm planning to implement RFD 613 Debug Dropbox shortly.

After this PR, there's a new file_archiver module:

  • There's a list of rules (in the rules submodule) that describe what files to collect. I hope it's easy to add new things to this.
  • Archival is now implemented with the plan-execute pattern. Nearly all the behavior is in the planner so that it can be exhaustively tested without having to actually construct directory trees. (There's still an end-to-end smoke test that does actually verify what happens with files on disk.)
  • There's a pretty comprehensive test suite that's based on a list of paths found on real systems (from dogfood). It includes checks to make sure the dataset itself covers all the different rules so that if we extend the rules but forget to update the test data, it will fail tests.

It's arguably overengineered at this point but I'm hopeful that this will make it a lot easier to augment the set of files that get archived in this way.

As a first step, I tried to preserve the existing behavior as much as possible. There are several oddities that we might want to fix in follow-up work:

  • There's no debouncing, meaning that it's possible that the system could try to archive a rotated log file or core file or crash dump while it's still being written, resulting in losing the original file and having only a partial copy in the debug dataset.
  • I need to double-check this but it looks to me like the existing implementation overwrites existing crash dumps and core files. This would be buggy but not a huge deal for core files because their names are pretty unusual (they have pids and execnames and zonenames in them), but it seems likely to result in at most one crash dump kept per sled ever since they'll all be called vmdump.0.
  • Rotated log files include their original mtime as a Unix timestamp in the filename, but if there's already a file with that name, then the mtime is incremented until we find a file that does not exist. This is a little weird because it means this value is close to the mtime but not actually the mtime. (One behavior change in this PR is that we'll only check a max number of possible filenames -- 30 -- after that, we give up.)
  • Live log files wind up being called something.mtime instead of something.log.mtime the way that rotated log files do. This isn't a huge deal but does break oxlog (oxlog does not find archived live log files #9271). I'm not sure what we should do here. We could use the same convention but then we'd lose the distinction between live vs. rotated log files. I'm not sure if that's important.

@davepacheco davepacheco marked this pull request as draft December 20, 2025 04:37
@davepacheco
Copy link
Collaborator Author

davepacheco commented Dec 20, 2025

Marking this draft because I still want to do some testing on a4x2 or a racklette.

@davepacheco
Copy link
Collaborator Author

davepacheco commented Dec 22, 2025

I deployed this to london and:

  • induced a panic on sled 15
  • induced a process core dump on sled 14

Regarding sled 14, I did briefly see the process core dump here:

BRM42220036 # find /pool/int/a15c936d-bf01-4411-aab2-d401b66910c6/crash
/pool/int/a15c936d-bf01-4411-aab2-d401b66910c6/crash
/pool/int/a15c936d-bf01-4411-aab2-d401b66910c6/crash/core.global.sleep.18626.1766442497
BRM42220036

As expected, that disappeared and the archived copy showed up:

BRM42220036 # find /pool/int/a15c936d-bf01-4411-aab2-d401b66910c6/crash
/pool/int/a15c936d-bf01-4411-aab2-d401b66910c6/crash
BRM42220036 # ls /pool/ext/*/crypt/debug/core*
/pool/ext/0544a89a-a2f8-494c-ace8-099b7fbd4e13/crypt/debug/core.global.sleep.18626.1766442497

On the sled where I induced a system panic:

BRM42220030 # ls -l /pool/ext/*/crypt/debug | grep -v ^d
/pool/ext/0990c846-28ad-40f8-bb73-5b19d4d85771/crypt/debug:
total 992

/pool/ext/6167e7b2-5a4c-420e-aa1f-8fc1be95d639/crypt/debug:
total 10409274
-rw-r--r--   1 root     root           2 Dec 28  1986 bounds
-rw-r--r--   1 root     root     10682171392 Dec 28  1986 vmdump.0

/pool/ext/6bdd15b1-bde9-4bd7-a1ea-342c0278b555/crypt/debug:
total 0

/pool/ext/72d71e4b-ddff-4cef-ac16-8707e1a732d1/crypt/debug:
total 0

/pool/ext/8b4122b0-e873-4c7c-bc6c-d2de1dfc68ba/crypt/debug:
total 0

/pool/ext/bad19632-89c0-4cd7-8430-5463dab81e4c/crypt/debug:
total 0

/pool/ext/bb7cb68b-0f7b-4994-91bc-5bd3b6527438/crypt/debug:
total 0

/pool/ext/d7edfff9-e058-4dfc-bb5f-fa1b18cb760a/crypt/debug:
total 0

/pool/ext/f47ac7ce-8c14-420c-9419-db3a3f714840/crypt/debug:
total 0

In terms of log files: back on sled 14:

BRM42220036 # ls -l /pool/ext/*/crypt/debug/
/pool/ext/0544a89a-a2f8-494c-ace8-099b7fbd4e13/crypt/debug/:
total 3154
-rw-r--r--   1 root     root     2725748 Dec 22 22:30 core.global.sleep.18626.1766442497
drwxr-xr-x   2 root     root           2 Dec 28  1986 global
drwxr-xr-x   2 root     root         110 Dec 22 21:00 oxz_cockroachdb_701e477d-effb-4ca5-954c-23629e59236b
drwxr-xr-x   2 root     root         110 Dec 22 21:00 oxz_crucible_14ae8c8d-1b04-4c29-9127-5a801540a6a4
drwxr-xr-x   2 root     root         110 Dec 22 21:00 oxz_crucible_1e7579a3-62e3-4377-9fe4-aa70cbbaa713
drwxr-xr-x   2 root     root         110 Dec 22 21:00 oxz_crucible_3cf95991-6e04-417b-b25f-f10568c2e420
drwxr-xr-x   2 root     root         110 Dec 22 21:00 oxz_crucible_55316099-f1d1-49de-a646-c041a86dea03
drwxr-xr-x   2 root     root         110 Dec 22 21:00 oxz_crucible_5af3a531-c868-4c02-a6cb-03a24ae010fc
drwxr-xr-x   2 root     root         110 Dec 22 21:00 oxz_crucible_5af6e97e-900d-4cc2-84ad-05210d6391fe
drwxr-xr-x   2 root     root         110 Dec 22 21:00 oxz_crucible_7cd97885-d96a-4e0a-9917-fa1c3a94e1bb
drwxr-xr-x   2 root     root         110 Dec 22 21:00 oxz_crucible_8aa08df3-6310-4c35-bf85-86c81dfe5549
drwxr-xr-x   2 root     root         110 Dec 22 21:00 oxz_crucible_d7d018b4-9567-46be-96e9-0f45141a9368
drwxr-xr-x   2 root     root         109 Dec 22 21:00 oxz_crucible_pantry_1a4a450b-c260-4ed8-80f6-735d26853263
drwxr-xr-x   2 root     root         110 Dec 22 21:00 oxz_external_dns_95f461ab-5307-4272-a3f3-7a3bd870792d
drwxr-xr-x   2 root     root         109 Dec 22 20:55 oxz_internal_dns_429d414f-135d-45f0-873c-2d2ec1427bb2
drwxr-xr-x   2 root     root         112 Dec 22 20:55 oxz_ntp_931d6ed6-bb5b-4afc-9b45-2fcadfd18dfa
drwxr-xr-x   2 root     root         110 Dec 22 23:00 oxz_oximeter_34208b99-36b3-4b2d-9fb1-cbd5a3f15804
drwxr-xr-x   2 root     root         119 Dec 22 20:55 oxz_switch

/pool/ext/0811ae71-4df3-49d5-9177-96ad02d13302/crypt/debug/:
total 0

/pool/ext/0a189e94-85d0-40ff-b84d-7957fb701743/crypt/debug/:
total 0

/pool/ext/26ab05fb-d580-41d4-9c89-c147ffcd2371/crypt/debug/:
total 0

/pool/ext/2f3d8653-5b88-4ccd-a91e-0e18c1098548/crypt/debug/:
total 0

/pool/ext/3510e4ab-8fff-47f5-8410-6404801ab35b/crypt/debug/:
total 0

/pool/ext/c4336926-de97-4224-8570-1fdae2e98223/crypt/debug/:
total 0

/pool/ext/c5403d97-0256-47fb-afa4-1e94479b3545/crypt/debug/:
total 0

/pool/ext/e31e6dd4-150b-4275-92c8-9fe4bc5c6f68/crypt/debug/:
total 0

BRM42220036 # ls -l /pool/ext/*/crypt/debug/oxz_crucible_14ae8c8d-1b04-4c29-9127-5a801540a6a4 | head
total 972
-rw-r--r--   1 root     root          75 Dec 22 21:00 application-management-net-snmp:default.log.1766436541
-rw-r--r--   1 root     root          30 Dec 22 21:00 application-security-tcsd:default.log.1766436536
-rw-r--r--   1 root     root         173 Dec 22 21:00 messages.1766436552
-rw-r--r--   1 root     root          80 Dec 22 21:00 milestone-devices:default.log.1766436537
-rw-r--r--   1 root     root         144 Dec 22 21:00 milestone-multi-user-server:default.log.1766436542
-rw-r--r--   1 root     root         400 Dec 22 21:00 milestone-multi-user:default.log.1766436542
-rw-r--r--   1 root     root         326 Dec 22 21:00 milestone-name-services:default.log.1766436542
-rw-r--r--   1 root     root          80 Dec 22 21:00 milestone-network:default.log.1766436537
-rw-r--r--   1 root     root         150 Dec 22 21:00 milestone-single-user:default.log.1766436542
BRM42220036 # ls -l /pool/ext/*/crypt/debug/oxz_crucible_14ae8c8d-1b04-4c29-9127-5a801540a6a4 | tail
-rw-r--r--   1 root     root         155 Dec 22 21:00 system-pkgserv:default.log.1766436538
-rw-r--r--   1 root     root          30 Dec 22 21:00 system-process-security:default.log.1766436537
-rw-r--r--   1 root     root         165 Dec 22 21:00 system-rbac:default.log.1766436540
-rw-r--r--   1 root     root         238 Dec 22 21:00 system-rmtmpfiles:default.log.1766436537
-rw-r--r--   1 root     root         179 Dec 22 21:00 system-sac:default.log.1766436542
-rw-r--r--   1 root     root          30 Dec 22 21:00 system-svc-global:default.log.1766436537
-rw-r--r--   1 root     root         206 Dec 22 21:00 system-system-log:default.log.1766436542
-rw-r--r--   1 root     root       70420 Dec 22 21:00 system-update-man-index:default.log.1766436543
-rw-r--r--   1 root     root         205 Dec 22 21:00 system-utmp:default.log.1766436542
-rw-r--r--   1 root     root          30 Dec 22 21:00 system-vtdaemon:default.log.1766436537

@davepacheco
Copy link
Collaborator Author

davepacheco commented Dec 22, 2025

I started a VM and was surprised to immediately see some rotated files in the debug dataset, but it turns out that yes, cron had run logadm despite this zone being only minutes old:

root@oxz_propolis:~# date
Mon Dec 22 23:32:51 UTC 2025
root@oxz_propolis:~# uptime
23:32:53    up 4 min(s),  1 user,  load average: 0.13, 0.06, 0.02
root@oxz_propolis:~# cat /var/cron/log 
! *** cron started ***   pid = 25187 Mon Dec 22 23:28:54 2025
>  CMD: /usr/sbin/logadm
>  root 26006 c Mon Dec 22 23:30:00 2025
<  root 26006 c Mon Dec 22 23:30:01 2025

Out in the GZ:

BRM42220030 # ls -l /pool/ext/*/crypt/debug/oxz_propolis-server_*
total 970
-rw-r--r--   1 root     root          75 Dec 22 23:30 application-management-net-snmp:default.log.1766446133
-rw-r--r--   1 root     root          30 Dec 22 23:30 application-security-tcsd:default.log.1766446130
-rw-r--r--   1 root     root         180 Dec 22 23:30 messages.1766446143
-rw-r--r--   1 root     root          80 Dec 22 23:30 milestone-devices:default.log.1766446130
-rw-r--r--   1 root     root         144 Dec 22 23:30 milestone-multi-user-server:default.log.1766446134
-rw-r--r--   1 root     root         400 Dec 22 23:30 milestone-multi-user:default.log.1766446134
-rw-r--r--   1 root     root         326 Dec 22 23:30 milestone-name-services:default.log.1766446133
-rw-r--r--   1 root     root          80 Dec 22 23:30 milestone-network:default.log.1766446130
-rw-r--r--   1 root     root         150 Dec 22 23:30 milestone-single-user:default.log.1766446134
-rw-r--r--   1 root     root          80 Dec 22 23:30 milestone-sysconfig:default.log.1766446134
-rw-r--r--   1 root     root          30 Dec 22 23:30 network-chrony:default.log.1766446130
-rw-r--r--   1 root     root         163 Dec 22 23:30 network-datalink-management:default.log.1766446129
-rw-r--r--   1 root     root         179 Dec 22 23:30 network-dns-client:default.log.1766446133
-rw-r--r--   1 root     root          30 Dec 22 23:30 network-dns-install:default.log.1766446129
-rw-r--r--   1 root     root        2787 Dec 22 23:30 network-inetd-upgrade:default.log.1766446134
-rw-r--r--   1 root     root          75 Dec 22 23:30 network-inetd:default.log.1766446134
-rw-r--r--   1 root     root         236 Dec 22 23:30 network-initial:default.log.1766446130
-rw-r--r--   1 root     root          30 Dec 22 23:30 network-install:default.log.1766446129
-rw-r--r--   1 root     root         161 Dec 22 23:30 network-ip-interface-management:default.log.1766446130
-rw-r--r--   1 root     root          75 Dec 22 23:30 network-ipmp:default.log.1766446133
-rw-r--r--   1 root     root          30 Dec 22 23:30 network-ipqos:default.log.1766446130
-rw-r--r--   1 root     root          30 Dec 22 23:30 network-ipsec-ike:default.log.1766446129
-rw-r--r--   1 root     root         157 Dec 22 23:30 network-ipsec-ipsecalgs:default.log.1766446130
-rw-r--r--   1 root     root          30 Dec 22 23:30 network-ipsec-manual-key:default.log.1766446129
-rw-r--r--   1 root     root         283 Dec 22 23:30 network-ipsec-policy:default.log.1766446130
-rw-r--r--   1 root     root         166 Dec 22 23:30 network-iptun:default.log.1766446130
-rw-r--r--   1 root     root          75 Dec 22 23:30 network-ipv4-forwarding:default.log.1766446134
-rw-r--r--   1 root     root          75 Dec 22 23:30 network-ipv6-forwarding:default.log.1766446134
-rw-r--r--   1 root     root          30 Dec 22 23:30 network-ldap-client:default.log.1766446129
-rw-r--r--   1 root     root          30 Dec 22 23:30 network-location:default.log.1766446129
-rw-r--r--   1 root     root         332 Dec 22 23:30 network-loopback:default.log.1766446130
-rw-r--r--   1 root     root         152 Dec 22 23:30 network-netcfg:default.log.1766446130
-rw-r--r--   1 root     root         162 Dec 22 23:30 network-netmask:default.log.1766446130
-rw-r--r--   1 root     root         275 Dec 22 23:30 network-physical:default.log.1766446130
-rw-r--r--   1 root     root          30 Dec 22 23:30 network-physical:nwam.log.1766446129
-rw-r--r--   1 root     root          75 Dec 22 23:30 network-routing-legacy-routing:ipv4.log.1766446134
-rw-r--r--   1 root     root          75 Dec 22 23:30 network-routing-legacy-routing:ipv6.log.1766446134
-rw-r--r--   1 root     root         188 Dec 22 23:30 network-routing-ndp:default.log.1766446144
-rw-r--r--   1 root     root          75 Dec 22 23:30 network-routing-rdisc:default.log.1766446134
-rw-r--r--   1 root     root          75 Dec 22 23:30 network-routing-ripng:default.log.1766446134
-rw-r--r--   1 root     root         120 Dec 22 23:30 network-routing-route:default.log.1766446134
-rw-r--r--   1 root     root         327 Dec 22 23:30 network-routing-setup:default.log.1766446134
-rw-r--r--   1 root     root          75 Dec 22 23:30 network-rpc-bind:default.log.1766446133
-rw-r--r--   1 root     root          30 Dec 22 23:30 network-rpc-keyserv:default.log.1766446129
-rw-r--r--   1 root     root          75 Dec 22 23:30 network-security-kadmin:default.log.1766446133
-rw-r--r--   1 root     root          75 Dec 22 23:30 network-security-krb5kdc:default.log.1766446133
-rw-r--r--   1 root     root         355 Dec 22 23:30 network-service:default.log.1766446133
-rw-r--r--   1 root     root         167 Dec 22 23:30 network-shares-group:default.log.1766446134
-rw-r--r--   1 root     root         163 Dec 22 23:30 network-shares-group:zfs.log.1766446134
-rw-r--r--   1 root     root          75 Dec 22 23:30 network-smb-client:default.log.1766446133
-rw-r--r--   1 root     root          30 Dec 22 23:30 network-smb-server:default.log.1766446130
-rw-r--r--   1 root     root         263 Dec 22 23:30 network-ssh:default.log.1766446134
-rw-r--r--   1 root     root          30 Dec 22 23:30 network-tcpkey:default.log.1766446129
-rw-r--r--   1 root     root           0 Dec 22 23:30 svc.startd.log.1766446128
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-auditd:default.log.1766446129
-rw-r--r--   1 root     root         163 Dec 22 23:30 system-auditset:default.log.1766446131
-rw-r--r--   1 root     root         170 Dec 22 23:30 system-boot-archive-update:default.log.1766446134
-rw-r--r--   1 root     root         163 Dec 22 23:30 system-boot-archive:default.log.1766446130
-rw-r--r--   1 root     root         166 Dec 22 23:30 system-boot-config:default.log.1766446134
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-consadm:default.log.1766446131
-rw-r--r--   1 root     root         115 Dec 22 23:30 system-console-login:default.log.1766446134
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-console-login:vt2.log.1766446131
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-console-login:vt3.log.1766446131
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-console-login:vt4.log.1766446131
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-console-login:vt5.log.1766446131
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-console-login:vt6.log.1766446131
-rw-r--r--   1 root     root         199 Dec 22 23:30 system-coreadm:default.log.1766446134
-rw-r--r--   1 root     root         204 Dec 22 23:30 system-cron:default.log.1766446134
-rw-r--r--   1 root     root         334 Dec 22 23:30 system-cryptosvc:default.log.1766446133
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-device-allocate:default.log.1766446130
-rw-r--r--   1 root     root         164 Dec 22 23:30 system-device-audio:default.log.1766446130
-rw-r--r--   1 root     root         241 Dec 22 23:30 system-device-local:default.log.1766446130
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-device-mpxio-upgrade:default.log.1766446129
-rw-r--r--   1 root     root          60 Dec 22 23:30 system-early-manifest-import:default.log.1766446128
-rw-r--r--   1 root     root          75 Dec 22 23:30 system-extended-accounting:flow.log.1766446133
-rw-r--r--   1 root     root          75 Dec 22 23:30 system-extended-accounting:net.log.1766446133
-rw-r--r--   1 root     root          75 Dec 22 23:30 system-extended-accounting:process.log.1766446133
-rw-r--r--   1 root     root          75 Dec 22 23:30 system-extended-accounting:task.log.1766446133
-rw-r--r--   1 root     root         159 Dec 22 23:30 system-filesystem-local:default.log.1766446134
-rw-r--r--   1 root     root         308 Dec 22 23:30 system-filesystem-minimal:default.log.1766446130
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-filesystem-reparse:default.log.1766446130
-rw-r--r--   1 root     root         158 Dec 22 23:30 system-filesystem-root:default.log.1766446129
-rw-r--r--   1 root     root         218 Dec 22 23:30 system-filesystem-usr:default.log.1766446130
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-fm-notify-params:default.log.1766446130
-rw-r--r--   1 root     root         295 Dec 22 23:30 system-fmd:default.log.1766446131
-rw-r--r--   1 root     root         161 Dec 22 23:30 system-hostid:default.log.1766446130
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-hotplug:default.log.1766446130
-rw-r--r--   1 root     root         326 Dec 22 23:30 system-identity:domain.log.1766446133
-rw-r--r--   1 root     root         164 Dec 22 23:30 system-identity:node.log.1766446130
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-idmap:default.log.1766446129
-rw-r--r--   1 root     root       81967 Dec 22 23:30 system-illumos-propolis-server:default.log.1766446195
-rw-r--r--   1 root     root         271 Dec 22 23:30 system-keymap:default.log.1766446133
-rw-r--r--   1 root     root         165 Dec 22 23:30 system-logadm-upgrade:default.log.1766446130
-rw-r--r--   1 root     root         288 Dec 22 23:30 system-manifest-import:default.log.1766446134
-rw-r--r--   1 root     root         273 Dec 22 23:30 system-name-service-cache:default.log.1766446133
-rw-r--r--   1 root     root         151 Dec 22 23:30 system-pfexec:default.log.1766446130
-rw-r--r--   1 root     root         155 Dec 22 23:30 system-pkgserv:default.log.1766446131
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-process-security:default.log.1766446130
-rw-r--r--   1 root     root         165 Dec 22 23:30 system-rbac:default.log.1766446133
-rw-r--r--   1 root     root         238 Dec 22 23:30 system-rmtmpfiles:default.log.1766446130
-rw-r--r--   1 root     root         179 Dec 22 23:30 system-sac:default.log.1766446134
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-svc-global:default.log.1766446130
-rw-r--r--   1 root     root         206 Dec 22 23:30 system-system-log:default.log.1766446134
-rw-r--r--   1 root     root       70420 Dec 22 23:30 system-update-man-index:default.log.1766446134
-rw-r--r--   1 root     root         205 Dec 22 23:30 system-utmp:default.log.1766446134
-rw-r--r--   1 root     root          30 Dec 22 23:30 system-vtdaemon:default.log.1766446130

What I've yet to confirm:

  • any GZ logs
  • logs from zone shutdown of any control plane zone

@davepacheco
Copy link
Collaborator Author

davepacheco commented Dec 22, 2025

I tried restarting sled-agent on sled 14, expecting to see that the restart would bounce control plane zones and archive their live log files, but I don't seem to see the live log files archived. I need to do more digging on that.

@davepacheco
Copy link
Collaborator Author

davepacheco commented Dec 23, 2025

I watched the global zone files get rotated as expected:

BRM42220036 # ls -l /pool/ext/*/crypt/debug/global
/pool/ext/0544a89a-a2f8-494c-ace8-099b7fbd4e13/crypt/debug/global:
total 0

/pool/ext/0811ae71-4df3-49d5-9177-96ad02d13302/crypt/debug/global:
total 0
BRM42220036 # ls -l /var/svc/log/*.log.*
-rw-r--r--   1 root     root     4335276 Dec 22 23:47 /var/svc/log/oxide-mg-ddm:default.log.0
-rw-r--r--   1 root     root     15328015 Dec 22 23:47 /var/svc/log/oxide-sled-agent:default.log.0

then a few minutes later:

BRM42220036 # ls -l /pool/ext/*/crypt/debug/global
/pool/ext/0544a89a-a2f8-494c-ace8-099b7fbd4e13/crypt/debug/global:
total 0

/pool/ext/0811ae71-4df3-49d5-9177-96ad02d13302/crypt/debug/global:
total 3731
-rw-r--r--   1 root     root      170934 Dec 22 23:52 messages.1766446712
-rw-r--r--   1 root     root     4335276 Dec 22 23:52 oxide-mg-ddm:default.log.1766447265
-rw-r--r--   1 root     root     15328015 Dec 22 23:52 oxide-sled-agent:default.log.1766447265
BRM42220036 # ls -l /var/svc/log/*.log.*
/var/svc/log/*.log.*: No such file or directory

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jgallagher jgallagher Awaiting requested review from jgallagher

@smklein smklein Awaiting requested review from smklein

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@davepacheco