Deployment Architecture

How can I set up the "Log Event" alert action in a distributed environment?

ctaf
Contributor

Hello,

I am trying to use the new alert action "Log event" in a distributed environment (Search Head 6.4.0 & Indexers 6.2.2).
Unfortunately, I doesn't work properly.

For the test, I set the "main" index as the destination index.

First issue: it seems that it is writing in "main" index, but on the Search Head, not on the Indexer (there is no way to indicate onto which Search peer to write the log by the way..)
Second issue: I cannot see the written log. When I search index=main, there is no result. I only guess that the event is written because when I go to the "Indexes" pages in the setting, the "Latest event" time is updated.

Any idea how to make it work?

0 Karma

emeelan_splunk
Splunk Employee
Splunk Employee

Hi folks,
This issue will be fixed in our next release.

Thank you for your patience.

-Eve Meelan

0 Karma

emeelan_splunk
Splunk Employee
Splunk Employee

Hi folks,
Strive's comment is correct. In order for Custom log alert events to be set up in a distributed environment, you must define the index on the search head. We are looking at this as a bug, but Strive's work-around is valid. For reference, the bug number is SPL-146802

0 Karma

pkarpushin
Path Finder

Hello,
Splunk 7.1.3 same issue

0 Karma

DATEVeG
Path Finder

Any news on fixing this?
I'm on 7.1.3 and the bug still seems to be present.

Thanks!

0 Karma

strive
Influencer

We faced similar issue.

The summarized data forwarding to indexers works fine. Other internal logs forwarding to indexers work fine. But, the Alert Action Log event alone fails.

For this to work -- We have to define index in Search Head as well.

Note: the data wont be stored on Search Head, eventually the Alert Action's Log event will be forwarded to indexer. But the definition on SH is must.

Not sure if it is a bug in Splunk or it is working as expected.

I have added a comment in Alert documentation and a discussion is on with splunk folks 🙂

nick405060
Motivator

Not very elegant. But fixes my problem. Thanks!

0 Karma

somesoni2
Revered Legend

Check why Search Head is not Forwarding the data to Indexers (it should forward instead of indexing locally). Check the outputs.conf. I believe second issue will resolve itself once you fix this.

0 Karma

ctaf
Contributor

Oh OK it makes sense for the first point. Unfortunately, I can't make this change due to my company policy.
But I should be able to search my local index on the search head, right? (second point) I do it for _internal index and there is no issue.

0 Karma
Get Updates on the Splunk Community!

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...