I am trying to use the new alert action "Log event" in a distributed environment (Search Head 6.4.0 & Indexers 6.2.2).
Unfortunately, I doesn't work properly.
For the test, I set the "main" index as the destination index.
First issue: it seems that it is writing in "main" index, but on the Search Head, not on the Indexer (there is no way to indicate onto which Search peer to write the log by the way..)
Second issue: I cannot see the written log. When I search
index=main, there is no result. I only guess that the event is written because when I go to the "Indexes" pages in the setting, the "Latest event" time is updated.
Any idea how to make it work?
Strive's comment is correct. In order for Custom log alert events to be set up in a distributed environment, you must define the index on the search head. We are looking at this as a bug, but Strive's work-around is valid. For reference, the bug number is SPL-146802
We faced similar issue.
The summarized data forwarding to indexers works fine. Other internal logs forwarding to indexers work fine. But, the Alert Action Log event alone fails.
For this to work -- We have to define index in Search Head as well.
Note: the data wont be stored on Search Head, eventually the Alert Action's Log event will be forwarded to indexer. But the definition on SH is must.
Not sure if it is a bug in Splunk or it is working as expected.
I have added a comment in Alert documentation and a discussion is on with splunk folks 🙂
Check why Search Head is not Forwarding the data to Indexers (it should forward instead of indexing locally). Check the outputs.conf. I believe second issue will resolve itself once you fix this.
Oh OK it makes sense for the first point. Unfortunately, I can't make this change due to my company policy.
But I should be able to search my local index on the search head, right? (second point) I do it for _internal index and there is no issue.