Deployment Architecture

What does this search head cluster function alert WARN messages mean?

spectrum2035
Explorer

Every other day, we are getting following error on the internal index. Nearly 65,000 messages are generated for less than 15mins. What does this error actually mean?

_WARN  SHCFunctions - alert csv wrong action  csv = key,expire,ACTION,MD5,"__mv_key","__mv_expire","__mv_ACTION","__mv_MD5"\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n_

DavidHourani
Super Champion

Hi @spectrum2035,

Do you still have this issue ? Seems like a misconfigured lookup or alert action to generate a csv. can you try to link this to any newly added alert action ?

0 Karma

smitra_splunk
Splunk Employee
Splunk Employee

facing same problem ... no clues .... doesn't look like there is a correlation to errors reported by other splunkd logging components. just sudden spikes of SHCFunctions warnings.

0 Karma

amitm05
Builder

@spectrum2035
Only the error does not give much info. Can you try to add some more info about the error ?
I am guessing if SHC means Search Head Cluster. Please check if you are able to find any errors/warnings in Monitoring Console on your search head dashboards and any warnings on General Health checks

0 Karma

spectrum2035
Explorer

I did check the general health status of the SHC in DMC and couldnt find anything alarming...

Following are the 4 logs which was indexed just before the event happened....

I ACCESS [conn47] Successfully authenticated as principal __system on local
I NETWORK [thread1] connection accepted from 10.10.10.3:50374 #47 (23 connections now open)
127.0.0.1 - splunk-system-user [25/Jun/2019:16:16:04.090 +0100] "GET /services/data/inputs/threatlist?output_mode=json&search=disabled%3D%22false%22 HTTP/1.0" 200 41063 - - - 92ms
I ACCESS [conn20] Successfully authenticated as principal __system on local

If I look back to the earlier one's i have license usage events OR StatusMgr related events.. so there is no specific pattern..

0 Karma

skalliger
SplunkTrust
SplunkTrust

That's just a wild guess: Are you using Enterprise Security? And on Windows?

Skalli

0 Karma

spectrum2035
Explorer

Yes we are using ES but on RHEL

0 Karma

adonio
Ultra Champion

check ES version and Splunk version compatibility:
https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/CompatMatrix
contact splunk support too

0 Karma

spectrum2035
Explorer

Thanks adonio, we have upgraded our servers nearly a year back and this started showing up for last 1 month only.

0 Karma

skalliger
SplunkTrust
SplunkTrust

I've never seen that logging category and I don't see SHCFunctions in the log.cfg either. Is that some custom app that logs into your _internal index?

Skalli

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...