[macOS][unified_log] Initial release of macOS Unified logs integration. #16170
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 tasks
Contributor
Author
|
Hi @cpascale43 , @narph , @nfritts , @leandrojmp , @P1llus , @marc-gr , @btrieger |
andrewkroh
added
New Integration
|
Does this have all of the feedback that had already been posted on the previous PR addressed? Is that being addressed or do those comments need to be re-added to this PR now? |
cpascale43
reviewed
Dec 3, 2025
cpascale43
reviewed
Dec 3, 2025
cpascale43
reviewed
Dec 3, 2025
|
|
||
| This integration requires Elastic Agent to be installed on the macOS systems you want to monitor. | ||
|
|
||
| ### Agent-based installation |
There was a problem hiding this comment.
Maybe another subheading here:
Should I use this integration if I already have Elastic Defend?
For comprehensive visibility, yes. Elastic Defend isn't designed to provide a complete capture of all system events and it's recommended to supplement it with the macOS unified logging integration.
Specifically, Elastic Defend on macOS does not capture:
- All user login/logout events
- Every user account creation, deletion, or modification
- Complete system service registration and changes
- Application diagnostic logs
This integration fills those gaps, similar to how Windows users supplement Elastic Defend with Custom Windows Event Logs integration. See Elastic Defend Event Capture documentation for details on what Elastic Defend does and doesn't capture.
There was a problem hiding this comment.
@muskan-agarwal26 are you able to add this in? Thank you
Contributor
Author
Yes, @nfritts , All comments on the previous PR are addressed. |
Contributor
|
I think we should consider removing the |
narph
added
the
Team:Security-Windows Platform
|
Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform) |
Remove custom fields.
cpascale43
approved these changes
Dec 12, 2025
marc-gr
reviewed
Dec 12, 2025
marc-gr
reviewed
Dec 12, 2025
|
@marc-gr is there anything else needed for your review? Thanks |
1. Removed predicates from agent files and added in default values in manifest. 2. Removed preserve duplicate custom toggle
muskan-agarwal26
force-pushed
the
macos-0.1.0
branch
from
39a3963 to
cb9e477
Compare
marc-gr
requested changes
Jan 13, 2026
Contributor
marc-gr
left a comment
marc-gr
left a comment
There was a problem hiding this comment.
In addition to the comments I think system_change and network_activity could be refactored to extract their common bits into a common-network.yml pipeline since ~259 lines are duplicated, and only ~13 lines are unique per pipeline.
packages/macos/data_stream/network_activity/elasticsearch/ingest_pipeline/default.yml
Show resolved
Hide resolved
1. Created common network pipeline. 2. Added missing if cond. 3. set ignore_empty_value to false.
muskan-agarwal26
force-pushed
the
macos-0.1.0
branch
from
a11ae47 to
e6c9f80
Compare
Contributor
|
/test |
🚀 Benchmarks reportTo see the full report comment with |
## 1. Removed Duplicate Grok Pattern
**File**: `_dev/shared/common-network-pipeline.yml`
**Issue**: Same grok pattern appeared twice on consecutive lines (copy-paste error).
**Fix**: Removed the duplicate line.
---
## 2. Created Proper Pipeline for `advanced_monitoring`
**File**: `data_stream/advanced_monitoring/elasticsearch/ingest_pipeline/default.yml`
**Issue**: `advanced_monitoring` used a `.link` file pointing directly to `common-pipeline.yml`, meaning it had no `event.category` or `event.type`.
**Fix**: Created a proper `default.yml` that sets `event.category: host` and `event.type: info` before calling the common pipeline.
---
## 3. Predicate Changes
### 3a. `advanced_monitoring` - Complete Rewrite
**File**: `data_stream/advanced_monitoring/manifest.yml`
**Before**:
```
- 'eventMessage CONTAINS[c] "exec" OR eventMessage CONTAINS[c] "fork" OR eventMessage CONTAINS[c] "exited" OR eventMessage CONTAINS[c] "terminated"'
- 'subsystem == "com.apple.securityd" AND (composedMessage CONTAINS "code signing" OR composedMessage CONTAINS "not valid")'
- 'composedMessage CONTAINS "com.apple.quarantine"'
```
**After**:
```
- 'subsystem == "com.apple.xpc" OR subsystem == "com.apple.launchd"'
- 'category == "performance" OR category == "diagnostics"'
- 'messageType == 16 OR messageType == 17'
```
**Why**: The original predicates were **identical** to `process_execution_monitoring`, meaning both data streams would collect the exact same events (duplicates). The new predicates focus on system internals (XPC/launchd services, performance diagnostics) which is semantically appropriate for "advanced monitoring".
### 3b. `process_execution_monitoring` - Removed Quarantine
**File**: `data_stream/process_execution_monitoring/manifest.yml`
**Removed**:
```
- 'composedMessage CONTAINS "com.apple.quarantine"'
```
**Why**: Quarantine is about downloaded file security, not process execution. The `file_read_write` data stream already captures quarantine events with `event.category: file`. Having both collect quarantine would cause duplicates with inconsistent categorization.
---
## 4. Updated Test Data
**Files**: `data_stream/*/test/pipeline/*.log`
**Issue**: Test events didn't match the new predicates after changes.
**Changes**:
- `advanced_monitoring`: Changed test events to use `com.apple.xpc` and `com.apple.launchd` subsystems
- `file_read_write`: Changed test events to include `open()` file operation and `com.apple.quarantine` subsystem
---
## 5. Fixed `system_change` Data Stream Architecture
**Files**:
- `data_stream/system_change/elasticsearch/ingest_pipeline/default.yml`
- `data_stream/system_change/manifest.yml`
- `data_stream/system_change/_dev/test/pipeline/test-system-change.log`
### 5a. Removed Unnecessary Network Pipeline
**Issue**: `system_change` was calling `common-network-pipeline.yml` which contains TCP/UDP grok patterns for parsing network connection strings. System change events (installer, security policy, software updates) don't contain network connection data.
**Fix**: Removed the `common-network-pipeline` call from the pipeline.
**Before**:
```yaml
- pipeline: { name: '{{ IngestPipeline "common-pipeline" }}' }
- pipeline: { name: '{{ IngestPipeline "common-network-pipeline" }}' }
```
**After**:
```yaml
- pipeline: { name: '{{ IngestPipeline "common-pipeline" }}' }
```
### 5b. Refined Predicate to Exclude Network Traffic
**Issue**: The predicate `process == "softwareupdated"` was too broad - it captured ALL logs from the `softwareupdated` process, including TCP/UDP network traffic from `com.apple.network` subsystem. This caused network events to incorrectly appear in the `system_change` data stream.
**Fix**: Added subsystem exclusion for `softwareupdated` matches.
**Before**:
```
process == "softwareupdated"
```
**After**:
```
(process == "softwareupdated" AND subsystem != "com.apple.network")
```
### 5c. Replaced Test Data
**Issue**: Test data contained TCP/UDP network connection logs instead of actual system change events.
**Fix**: Replaced with proper system change events:
- `com.apple.systempolicy` - Gatekeeper/system policy evaluations
- `com.apple.installer` - Package installation/removal
- `com.apple.security` - XProtect updates, security assessments
- `com.apple.SoftwareUpdate` - Catalog updates
---
## 6. Inlined Network Pipeline into `network_activity`
**Files Changed**:
- `data_stream/network_activity/elasticsearch/ingest_pipeline/default.yml` - Inlined all processing
- `_dev/shared/common-network-pipeline.yml` - **Deleted**
- `data_stream/network_activity/elasticsearch/ingest_pipeline/common-network-pipeline.yml.link` - **Deleted**
- `data_stream/system_change/elasticsearch/ingest_pipeline/common-network-pipeline.yml.link` - **Deleted**
**Rationale**: After removing `system_change`'s dependency on the network pipeline, `network_activity` was the only consumer of `common-network-pipeline.yml`. Having a shared pipeline for a single consumer adds unnecessary indirection.
**Result**: The network-specific processing (grok patterns for TCP/UDP, TLS handshake parsing, byte/packet calculations, related.hosts/related.ip enrichment) is now directly in `network_activity`'s `default.yml`.
---
## Final Predicate Overview
| Data Stream | Predicate Focus | event.category |
|-------------|-----------------|----------------|
| **advanced_monitoring** | XPC, launchd, performance/diagnostics | `host` |
| **authentication** | sudo, su, loginwindow, sshd | `authentication` |
| **file_read_write** | File ops (open/write/unlink/rename) + quarantine | `file` |
| **network_activity** | TCP/UDP connections, disconnect, NECP | `network` |
| **process_execution_monitoring** | exec/fork/exit/terminate, code signing | `process` |
| **system_change** | Security policy, installer, software updates | `configuration` |
| **user_and_account_management** | sysadminctl, dscl, admin operations | `iam` |
All predicates are now unique with no overlapping event collection.
marc-gr
approved these changes
Jan 14, 2026
Contributor
|
/test |
Contributor
|
/test |
💚 Build Succeeded
History
|
Contributor
|
we need a review from @elastic/integrations-triaging |
jsoriano
approved these changes
Jan 14, 2026
|
Package macos - 0.1.0 containing this change is available at https://epr.elastic.co/package/macos/0.1.0/ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed commit message
The initial release includes unified_log data stream and associated dashboard.
macOS fields are mapped to their corresponding ECS fields where possible.
Test samples were derived from live data samples.
Checklist
changelog.ymlfile.How to test this PR locally
To test the macOS package:
Related issues
Screenshots