I have an issue with a new dedicated Search Head for ES.
My Splunk architecture is quite simple. 4 clustered indexers, 3 SH, and 1 dedicated SH for ES
everything is working fine, except the ES notable events : the index notable remains empty.
In the SH, search peer list : everything is OK, indexers are defined
All the events of the SH are sent to the indexers, the notable index has been created there
I have correlation searches active with 2 actions : notable and json alerting.
- JSON alerting is OK
- notable : not OK
If I manually create a notable event on ES, I can see it in the index main, with a strange sourcetype (“stash_common_action_model-too_small”)
I have found it with the following request “index=main notable”
“1589789135, search_name="Manual Notable Event - Rule", _time="1589789135", app="SplunkEnterpriseSecuritySuite", creator="XXXX", info_max_time="+Infinity", info_min_time="0.000", info_search_time="1589789135.348247000", owner="XXXX", rule_description="hello world", rule_title="test5", security_domain="access", status="1", urgency="informational"
=> is there an issue with this sourcetype / index ?
=> or any idea for troubleshooting ?
thank you in advance
PS: a Splunk ticket was opened 10 days ago ... but I am still stuck
problem resolved with the support help
the inputs.conf from the ES app was corrupted
Splunk was not able to parse correctly the notable events.
with the correct inputs.conf file everything is working as expecting