topic Re: What does this search head cluster function alert WARN messages mean? in Deployment Architecture

What does this search head cluster function alert WARN messages mean?

spectrum2035 — Mon, 24 Jun 2019 21:42:43 GMT

Every other day, we are getting following error on the internal index. Nearly 65,000 messages are generated for less than 15mins. What does this error actually mean?

_WARN  SHCFunctions - alert csv wrong action  csv = key,expire,ACTION,MD5,"__mv_key","__mv_expire","__mv_ACTION","__mv_MD5"\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n"","","","",,,,\n_

Re: What does this search head cluster function alert WARN messages mean?

skalliger — Tue, 25 Jun 2019 19:01:31 GMT

I've never seen that logging category and I don't see SHCFunctions in the log.cfg either. Is that some custom app that logs into your _internal index?

Skalli

Re: What does this search head cluster function alert WARN messages mean?

amitm05 — Tue, 25 Jun 2019 20:20:24 GMT

@spectrum2035
Only the error does not give much info. Can you try to add some more info about the error ?
I am guessing if SHC means Search Head Cluster. Please check if you are able to find any errors/warnings in Monitoring Console on your search head dashboards and any warnings on General Health checks

Re: What does this search head cluster function alert WARN messages mean?

spectrum2035 — Wed, 30 Sep 2020 01:03:21 GMT

I did check the general health status of the SHC in DMC and couldnt find anything alarming...

Following are the 4 logs which was indexed just before the event happened....

I ACCESS [conn47] Successfully authenticated as principal __system on local
I NETWORK [thread1] connection accepted from 10.10.10.3:50374 #47 (23 connections now open)
127.0.0.1 - splunk-system-user [25/Jun/2019:16:16:04.090 +0100] "GET /services/data/inputs/threatlist?output_mode=json&search=disabled%3D%22false%22 HTTP/1.0" 200 41063 - - - 92ms
I ACCESS [conn20] Successfully authenticated as principal __system on local

If I look back to the earlier one's i have license usage events OR StatusMgr related events.. so there is no specific pattern..

Re: What does this search head cluster function alert WARN messages mean?

skalliger — Wed, 26 Jun 2019 09:03:48 GMT

That's just a wild guess: Are you using Enterprise Security? And on Windows?

Skalli

Re: What does this search head cluster function alert WARN messages mean?

spectrum2035 — Wed, 26 Jun 2019 22:41:03 GMT

Yes we are using ES but on RHEL

Re: What does this search head cluster function alert WARN messages mean?

adonio — Thu, 27 Jun 2019 02:48:59 GMT

check ES version and Splunk version compatibility:
https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/CompatMatrix
contact splunk support too

Re: What does this search head cluster function alert WARN messages mean?

spectrum2035 — Thu, 27 Jun 2019 07:52:22 GMT

Thanks adonio, we have upgraded our servers nearly a year back and this started showing up for last 1 month only.

Re: What does this search head cluster function alert WARN messages mean?

DavidHourani — Thu, 27 Jun 2019 08:25:08 GMT

Hi @spectrum2035,

Do you still have this issue ? Seems like a misconfigured lookup or alert action to generate a csv. can you try to link this to any newly added alert action ?

Re: What does this search head cluster function alert WARN messages mean?

smitra_splunk — Tue, 13 Aug 2019 14:03:25 GMT

facing same problem ... no clues .... doesn't look like there is a correlation to errors reported by other splunkd logging components. just sudden spikes of SHCFunctions warnings.