Hey shocko — your Procmon read is spot on: this is the metadata/permission merge getting hammered by on-access scanning. The asymmetry is the giveaway. Searches hit big sequential index I/O and feel fine, but opening any config dialog makes splunkd walk the whole knowledge-object tree across system, app and user contexts — hundreds of thousands of tiny .meta/.conf opens plus a Get ACL on each. That small-file-plus-ACL workload is exactly what Windows on-access AV punishes hardest, and your trace shows it: the .meta reads and the Get ACL storm across etc\users, etc\apps and etc\slave-apps.
Two things to go after, in order:
1. Get Splunk out of Defender's on-access path. This is Splunk's own recommendation for Windows and almost certainly your biggest win. Exclude the Splunk processes (splunkd.exe and the splunk-*.exe helpers) and the whole install dir ($SPLUNK_HOME, plus $SPLUNK_DB if it's separate) from Defender real-time/on-access scanning — and make sure your EDR layer honors the same exclusions, since EDR often keeps scanning even when the plain AV exclusion is set. Quick proof: in a maintenance window, temporarily turn off real-time protection, open a macro dialog, and see if it snaps open. If it does, you've found it.
2. Shrink what splunkd has to scan every time. Exclusions cut the per-file scan cost; you can also cut the number of files:
- etc\users is usually the culprit. Every user directory carries its own local.meta and any private knowledge objects, and the permission merge walks all of them on each config load. Count them (dir etc\users) — if you've got hundreds of stale/orphaned user dirs from people who've left or from SSO churn, that's a lot of dead weight. Archive the orphaned ones (back up first), and reassign or clean up private KOs that don't need to be per-user.
- Trim unused apps. More apps in etc\apps means more .conf/.meta to merge on every load — remove anything you're not actually using.
- slave-apps means this box is also an indexer cluster peer (or has leftover peer apps from one). Those .meta are in the scan too, so it's worth being clear on the box's full role — a combined search head plus peer is carrying both footprints.
One more to rule out: make sure $SPLUNK_HOME\etc is on fast local disk, not a SAN or network volume — the many-tiny-files merge is brutal over any added latency.
Worth a read:
How many user directories are under etc\users, and roughly how many apps in etc\apps? That'll tell us whether the AV exclusion alone does it, or whether you've also got an etc\users cleanup on your hands.
