I have this Heavy Forwarder apparently not sending its own _internal logs $SPLUNK_HOME/var/log/splunk/*.log to the indexers. What I've already checked: HF is working fine, delivering data which it's set to receive and forward. HF is phoning Deployment server fine. _audit index is being indexed fine. Using $ splunk list forward-server I see it is properly set to send data only to correct indexers. The logs are being written as expected and have proper reading permissions, e.g.: $ ls -ltr ~/var/log/splunk/splunkd.log -rw------- 1 splunk splunk 12983503 Feb 26 11:43 /opt/splunk/var/log/splunk/splunkd.log Searching for _internal index into HF returns no results as supposed to be. Any ideas about what is going on? There is already a question about it in Answers, but not satisfying answered... https://answers.splunk.com/answers/686484/why-are-internal-logs-from-heavy-forwarderhf-not-g.html Thanks, TCM ... View more

I'm trying to run Monitoring Console Health Check but it doesn't start. Splunk shows me only "Loading..." page. The checklist.conf file is ok in default directory. I'm running the brand new version 7.2.4. Can someone help me please? ... View more

Monitoring saturation of event-processing queues in Heavy Forwarders I have a distributed environment with multiple indexes, search heads, and a pair of heavy forwarders. But over the last few days, one of my heavy forwarders started to alert a issue. The Monitoring Console's Health Check is warning "Saturation of event-processing queues". Besides that, the heavy forwarders performances have decreased a lot, delaying event delivery and failing scripts execution. splunkd is consuming 100% of its CPU core full time. Checking docs (Identify and triage indexing performance problems), they suggest to determine queue fill pattern through the Monitoring Console > Indexing > Indexing Performance: Instance. But, seems it applies only to the indexers, not to the heavy forwarder. Please, how could I discover what is causing such issue? How could I monitor such an issue? How can I see when it starts and how long it takes in order to do a cross with other systems behavior? Is such info available in the Monitoring Console? Thanks in advance and regards, Tiago ... View more

I'm wondering if somebody had faced this freaking behavior. I wanna extract both key, the field name, and its value from my (pretty uncommon) log and, in order to this I did the following: In first place I made the search bellow just to test the regex, and it's working perfectly. ... | rex max_match=0 field=_raw "(?<test1>\w+)\(.+\)=(?<test2>[^\(].*)[\n|\r]" I then replaced the test1 and test2 tags by _KEY_1 and _VAL_1 to assign properly each matched group to key and value as I wanted. ... | rex max_match=0 field=_raw "(?<_KEY_1>\w+)\(.+\)=(?<_VAL_1>[^\(].*)[\n|\r]" From here ahead the extraction didn't work anymore. So, had someone handled successfully same problem using this _KEY_1 and _VAL_1 tags? It seems like a bug for me. Thanks in advance, Tiago ... View more