Deployment Architecture
Highlighted

What does this search head cluster function alert WARN messages mean?

Explorer

Every other day, we are getting following error on the internal index. Nearly 65,000 messages are generated for less than 15mins. What does this error actually mean?

_WARN  SHCFunctions - alert csv wrong action  csv = key,expire,ACTION,MD5,"__mv_key","__mv_expire","__mv_ACTION","__mv_MD5"\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n_
Highlighted

Re: What does this search head cluster function alert WARN messages mean?

SplunkTrust
SplunkTrust

I've never seen that logging category and I don't see SHCFunctions in the log.cfg either. Is that some custom app that logs into your _internal index?

Skalli

0 Karma
Highlighted

Re: What does this search head cluster function alert WARN messages mean?

Builder

@spectrum2035
Only the error does not give much info. Can you try to add some more info about the error ?
I am guessing if SHC means Search Head Cluster. Please check if you are able to find any errors/warnings in Monitoring Console on your search head dashboards and any warnings on General Health checks

0 Karma
Highlighted

Re: What does this search head cluster function alert WARN messages mean?

Explorer

I did check the general health status of the SHC in DMC and couldnt find anything alarming...

Following are the 4 logs which was indexed just before the event happened....

I ACCESS [conn47] Successfully authenticated as principal _system on local
I NETWORK [thread1] connection accepted from 10.10.10.3:50374 #47 (23 connections now open)
127.0.0.1 - splunk-system-user [25/Jun/2019:16:16:04.090 +0100] "GET /services/data/inputs/threatlist?output
mode=json&search=disabled%3D%22false%22 HTTP/1.0" 200 41063 - - - 92ms
I ACCESS [conn20] Successfully authenticated as principal __system on local

If I look back to the earlier one's i have license usage events OR StatusMgr related events.. so there is no specific pattern..

0 Karma
Highlighted

Re: What does this search head cluster function alert WARN messages mean?

SplunkTrust
SplunkTrust

That's just a wild guess: Are you using Enterprise Security? And on Windows?

Skalli

0 Karma
Highlighted

Re: What does this search head cluster function alert WARN messages mean?

Explorer

Yes we are using ES but on RHEL

0 Karma
Highlighted

Re: What does this search head cluster function alert WARN messages mean?

SplunkTrust
SplunkTrust

check ES version and Splunk version compatibility:
https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/CompatMatrix
contact splunk support too

0 Karma
Highlighted

Re: What does this search head cluster function alert WARN messages mean?

Explorer

Thanks adonio, we have upgraded our servers nearly a year back and this started showing up for last 1 month only.

0 Karma
Highlighted

Re: What does this search head cluster function alert WARN messages mean?

SplunkTrust
SplunkTrust

Hi @spectrum2035,

Do you still have this issue ? Seems like a misconfigured lookup or alert action to generate a csv. can you try to link this to any newly added alert action ?

0 Karma
Highlighted

Re: What does this search head cluster function alert WARN messages mean?

Splunk Employee
Splunk Employee

facing same problem ... no clues .... doesn't look like there is a correlation to errors reported by other splunkd logging components. just sudden spikes of SHCFunctions warnings.

0 Karma