Deployment Architecture

How can I set up the "Log Event" alert action in a distributed environment?

Contributor

Hello,

I am trying to use the new alert action "Log event" in a distributed environment (Search Head 6.4.0 & Indexers 6.2.2).
Unfortunately, I doesn't work properly.

For the test, I set the "main" index as the destination index.

First issue: it seems that it is writing in "main" index, but on the Search Head, not on the Indexer (there is no way to indicate onto which Search peer to write the log by the way..)
Second issue: I cannot see the written log. When I search index=main, there is no result. I only guess that the event is written because when I go to the "Indexes" pages in the setting, the "Latest event" time is updated.

Any idea how to make it work?

0 Karma

Splunk Employee
Splunk Employee

Hi folks,
This issue will be fixed in our next release.

Thank you for your patience.

-Eve Meelan

0 Karma

Splunk Employee
Splunk Employee

Hi folks,
Strive's comment is correct. In order for Custom log alert events to be set up in a distributed environment, you must define the index on the search head. We are looking at this as a bug, but Strive's work-around is valid. For reference, the bug number is SPL-146802

0 Karma

Path Finder

Hello,
Splunk 7.1.3 same issue

0 Karma

Path Finder

Any news on fixing this?
I'm on 7.1.3 and the bug still seems to be present.

Thanks!

0 Karma

Influencer

We faced similar issue.

The summarized data forwarding to indexers works fine. Other internal logs forwarding to indexers work fine. But, the Alert Action Log event alone fails.

For this to work -- We have to define index in Search Head as well.

Note: the data wont be stored on Search Head, eventually the Alert Action's Log event will be forwarded to indexer. But the definition on SH is must.

Not sure if it is a bug in Splunk or it is working as expected.

I have added a comment in Alert documentation and a discussion is on with splunk folks 🙂

Motivator

Not very elegant. But fixes my problem. Thanks!

0 Karma

SplunkTrust
SplunkTrust

Check why Search Head is not Forwarding the data to Indexers (it should forward instead of indexing locally). Check the outputs.conf. I believe second issue will resolve itself once you fix this.

0 Karma

Contributor

Oh OK it makes sense for the first point. Unfortunately, I can't make this change due to my company policy.
But I should be able to search my local index on the search head, right? (second point) I do it for _internal index and there is no issue.

0 Karma