Deployment Architecture

Why are there errors on new Search Head Cluster member?


I recently added a new host to my search head cluster and am receiving a continuous stream of errors as seen below from the new host. Any idea how I can determine what is causing these errors and how to fix them?

Interestingly, when I look at a count of the alerts, the number of alerts per hour has gone steadily down by about 5-10 per hour since they first started:
alt text

I also noticed that the error seems to reference 2 apps that don't currently show any data: NetApp and Palo Alto. I'm not sure if they ever displayed data or not as I have never used them, but I know that they have not displayed data for quite some time - long before these errors started. The "skipping" note in the error seems to indicate there is a lot more to the error than I can see, but I obviously don't know what so I'm not sure if other apps are referenced or not.

These are the steps I have tried to resolve the issue:

  • Rolling restart of the SHC
  • Remove, clean and re-add the newest member
  • I haven't seen any problems while using the latest member; searching works, dashboards work, etc.

Here is one of the errors:

index=_internal source="/opt/splunk/var/log/splunk/splunkd.log" "SHCMasterHTTPProxy - Low Level http request failure err=Deserialization failed."

02-12-2018 10:50:52.843 -0800 WARN SHCMasterHTTPProxy - Low Level http request failure err=Deserialization failed. Could not find expected key 'unique_guids_artifactids' (Reply: ConfigInfo: feed_name = , {\n CC2A8F3B-A392-4C0D-8914-F611CE068DFB -> ConfigItem: name=CC2A8F3B-A392-4C0D-8914-F611CE068DFB title= atomId= owner=system app= customActions={}; ArgsList: {artifacts_location_csv -> ParamType: dataType=unset _isMultiValue=false {_values: {[0]='"artifact_id","artifact_log_entry",peer,"mv_artifact_id","mv_artifact_log_entry","mv_peer"\n"scheduleradminpostfixRMD504f0506f29d1e837_at_1518456600_22508_3142118D-D20E-4C18-B6EC-EE7B69A5F00B",0,"3142118D-D20E-4C18-B6EC-EE7B69A5F00B",,,\n"scheduleradminpostfixRMD504f0506f29d1e837_at_1518456600_22508_3142118D-D20E-4C18-B6EC-EE7B69A5F00B",0,"F6E7F7FE-DC53-456F-B8EC-B624BAF5E1B4",,,\n"scheduleradminpostfixRMD504f0506f29d1e837_at_1518460200_25_3142118D-D20E-4C18-B6EC-EE7B69A5F00B",0,"3142118D-D20E-4C18-B6EC-EE7B69A5F00B",,,\n"scheduleradminpostfixRMD504f0506f29d1e837_at_1518460200_25_3142118D-D20E-4C18-B6EC-EE7B69A5F00B",0,"F6E7F7FE-DC53-456F-B8EC-B624BAF5E1B4",,,\n"scheduleradminpostfixRMD51d56dd48c3688be1_at_1518456600_26467_F6E7F7FE-DC53-456F-B8EC-B624BAF5E1B4",0,"3142118D-D20E-4C18-B6EC-EE7B69A5F00B",,,\n"scheduleradminpostfixRMD51d56dd48c3688be1_at_1518456600_26467_F6E7F7FE-DC53-456F-B8EC-B624BAF5E1B4",0,"F6E7F7FE-DC53-456F-B8EC-B624BAF5E1B4",,,\n"scheduleradminpostfix_RMD51d56dd48c3688be1_at_1518460200_0_CC2A8F3B-A392-4C0D-8914-F611CE068DFB",0,"314211 ...{skipping 103210 bytes}... _app_netapp","tsidx-perf-system-ontap",1,1518461700,,,,,\nnobody,SplunkforPaloAltoNetworks,"WildFire Reports - Retrieve Report",1,1518461460,,,,,\nadmin,"splunk_app_netapp","tsidx-perf-disk-ontap",1,1518461700,,,,,\nadmin,"splunk_app_netapp","tsidx-perf-quota-ontap",1,1518461700,,,,,\nadmin,"splunk_app_netapp","tsidx-perf-qtree-ontap",1,1518461700,,,,,\n'} (size=1)}, splunk_min_version -> ParamType: _dataType=unset _isMultiValue=false {_values: {[0]='6.5.0'} (size=1)}, } _m.size=14\n Messages:\n}\n)

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.