What does this search head cluster function alert WARN messages mean?

[spectrum2035]

Every other day, we are getting following error on the internal index. Nearly 65,000 messages are generated for less than 15mins. What does this error actually mean?

_WARN  SHCFunctions - alert csv wrong action  csv = key,expire,ACTION,MD5,"__mv_key","__mv_expire","__mv_ACTION","__mv_MD5"\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n_

[DavidHourani]

Hi @spectrum2035,

Do you still have this issue ? Seems like a misconfigured lookup or alert action to generate a csv. can you try to link this to any newly added alert action ?

[smitra_splunk]

facing same problem ... no clues .... doesn't look like there is a correlation to errors reported by other splunkd logging components. just sudden spikes of SHCFunctions warnings.

[amitm05]

@spectrum2035
Only the error does not give much info. Can you try to add some more info about the error ?
I am guessing if SHC means Search Head Cluster. Please check if you are able to find any errors/warnings in Monitoring Console on your search head dashboards and any warnings on General Health checks

[spectrum2035]

I did check the general health status of the SHC in DMC and couldnt find anything alarming...

Following are the 4 logs which was indexed just before the event happened....

I ACCESS [conn47] Successfully authenticated as principal __system on local
I NETWORK [thread1] connection accepted from 10.10.10.3:50374 #47 (23 connections now open)
127.0.0.1 - splunk-system-user [25/Jun/2019:16:16:04.090 +0100] "GET /services/data/inputs/threatlist?output_mode=json&search=disabled%3D%22false%22 HTTP/1.0" 200 41063 - - - 92ms
I ACCESS [conn20] Successfully authenticated as principal __system on local

If I look back to the earlier one's i have license usage events OR StatusMgr related events.. so there is no specific pattern..

[skalliger]

That's just a wild guess: Are you using Enterprise Security? And on Windows?

Skalli

[spectrum2035]

Yes we are using ES but on RHEL

[adonio]

check ES version and Splunk version compatibility:
https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/CompatMatrix
contact splunk support too

[spectrum2035]

Thanks adonio, we have upgraded our servers nearly a year back and this started showing up for last 1 month only.

[skalliger]

I've never seen that logging category and I don't see SHCFunctions in the log.cfg either. Is that some custom app that logs into your _internal index?

Skalli