Finally got around to reading about SC4S and I'm fairly certain this is the route forward. I got the ok to replace the pair of UF's with a Heavy if needed. But I have a good feeling that the SC4S will be able to handle things very well. ... View more

I guess Splunk 9.x defauls to systemd again. Any way to revert to init.d? ... View more

Try running btool when this "non-impacting" error occurs.  It doesn't run. ... View more

It is a breaking issue as I cannot run btool on my forwarders that are throwing this message.  ... View more

Thank you ... View more

Getting the same warning.  I'll submit a support ticket. ... View more

Aight so, an admittal if you will. I have been ignoring the fact that all the timestamps are in Zulu time. Therefor, some of what I've claimed is incorrect. So taking the time conversion into account in the below response:   ./splunk list monitor   Shows all the files that should be monitored, are.   ./splunk list inputstatus   Shows all the files as before.. the small, less active ones are type = finished reading. The larger, problematic ones are type = open file. All are 100%. Re-running the Query, again for the past 24 hours I now have 4 empty hours (Z -> CDT; 2AM, 3AM, 4AM and 7AM), grepping the log file as before for a timestamp that's missing from SC. So far the first three missing hours: 2, 3, and 4AM are indeed empty from the log file. 7am however has events after searching randomly for, 07:10, 07:34, 07:45..many events.  So I have a question out to the networking team to verify there are events/alerts in the local log storage on a device from the group that sourced the missing data.   Still waiting for a response from networking. ... View more

Hi @Skeer-Jamf  Sounds like you're on the right track now. Note, if using the Splunk Windows app then it is probably best to not set the sourcetype in inputs.conf and let the default values flow through the system for these Windows events.  As you've seen, having the wrong sourcetype value can affect downstream configurations. The Splunk Windows app needs to be installed on the heavy forwarders and also the search head members (or Splunk Cloud - basically, where you run your search queries from).  The app docs say the inputs.conf file should be removed when deployed to the search head/Splunk Cloud.   For this type of Windows event sources I don't believe the app needs to be installed on the UF (not 100% sure on that) but no harm if it is.  Basically, just need an inputs.conf entry on the UF to forward these Windows Directory Service events (assuming the UF is forwarding data correctly, as expected), e.g. ## Application and Services Logs - Directory Service [WinEventLog://Directory Service] disabled = 1 renderXml=true If it is working correctly then the sourcetype should actually be getting set to XMLWinEventLog when it is searched and all these event fields will be auto extracted at search time OK. If the sourcetype is not XMLWinEventLog then the auto extractions will not be working and you need to review each step (UF -> HF -> SC)   ... View more

top defaults to 10 results - try | top 0 src ... View more

The "weird string" is the host name for your search head.  You can let that default because you'll need to override it, anyway. You'll need to create an app with a transforms.conf file that sets the host name to a value in the data.  See https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Overridedefaulthostassignments for a good description of that.  Use a heavy forwarder as described in the doc or build an app and upload it Splunk.com.  Either way, I recommend putting the .conf files in an app directory rather than in etc/system/local. When you've done this, you'll have a custom sourcetype and should use that name rather than the built-in csv sourcetype. I don't know why you're not seeing all of the index names. ... View more

Hey Pickle.. So yeah exploding the SPL into the APPs dir, like i mentioned, is just beautiful. Works like a champ! The playbook is done and done thanks to you guys. Luckily, in my situation, we'll likely only ever have the one APP to install, a single ansible service account across all of our machines and.. we run primarily a Linux shop.  Oh and with regards to rsyslog. I've got respect for it, but it's anything but friendly. 🙂 In either case.. much, MUCH thanks for the help! ... View more