Already done, and fixed. Turns out that the install itself appears to be correct in all instances we could audit, but permissions were just not getting picked up.
To fix: stopped both SHs, removed all 3 apps (SA-nix, Splunk_TA, Splunk for nix app), and untarred a fresh copy. Restarted Splunk SH, went through SA-nix first time setup, then Splunk for NIX first time setup, and permissions were correct, and I could see the past 3 months of data in the os index.
Thanks to BrianO for verifying the original setup looked right, and the suggestion for the "nuke from orbit" and re-install via tar.
... View more