Deployment Architecture

Why are there errors on new Search Head Cluster member?

rrustong
Explorer

I recently added a new host to my search head cluster and am receiving a continuous stream of errors as seen below from the new host. Any idea how I can determine what is causing these errors and how to fix them?

Interestingly, when I look at a count of the alerts, the number of alerts per hour has gone steadily down by about 5-10 per hour since they first started:
alt text

I also noticed that the error seems to reference 2 apps that don't currently show any data: NetApp and Palo Alto. I'm not sure if they ever displayed data or not as I have never used them, but I know that they have not displayed data for quite some time - long before these errors started. The "skipping" note in the error seems to indicate there is a lot more to the error than I can see, but I obviously don't know what so I'm not sure if other apps are referenced or not.

These are the steps I have tried to resolve the issue:

  • Rolling restart of the SHC
  • Remove, clean and re-add the newest member
  • I haven't seen any problems while using the latest member; searching works, dashboards work, etc.

Here is one of the errors:

index=_internal source="/opt/splunk/var/log/splunk/splunkd.log" "SHCMasterHTTPProxy - Low Level http request failure err=Deserialization failed."

02-12-2018 10:50:52.843 -0800 WARN SHCMasterHTTPProxy - Low Level http request failure err=Deserialization failed. Could not find expected key 'unique_guids_artifactids' (Reply: ConfigInfo: feed_name = , {\n CC2A8F3B-A392-4C0D-8914-F611CE068DFB -> ConfigItem: name=CC2A8F3B-A392-4C0D-8914-F611CE068DFB title= atomId= owner=system app= customActions={}; ArgsList: {artifacts_location_csv -> ParamType: dataType=unset _isMultiValue=false {_values: {[0]='"artifact_id","artifact_log_entry",peer,"mv_artifact_id","mv_artifact_log_entry","mv_peer"\n"scheduleradminpostfixRMD504f0506f29d1e837_at_1518456600_22508_3142118D-D20E-4C18-B6EC-EE7B69A5F00B",0,"3142118D-D20E-4C18-B6EC-EE7B69A5F00B",,,\n"scheduleradminpostfixRMD504f0506f29d1e837_at_1518456600_22508_3142118D-D20E-4C18-B6EC-EE7B69A5F00B",0,"F6E7F7FE-DC53-456F-B8EC-B624BAF5E1B4",,,\n"scheduleradminpostfixRMD504f0506f29d1e837_at_1518460200_25_3142118D-D20E-4C18-B6EC-EE7B69A5F00B",0,"3142118D-D20E-4C18-B6EC-EE7B69A5F00B",,,\n"scheduleradminpostfixRMD504f0506f29d1e837_at_1518460200_25_3142118D-D20E-4C18-B6EC-EE7B69A5F00B",0,"F6E7F7FE-DC53-456F-B8EC-B624BAF5E1B4",,,\n"scheduleradminpostfixRMD51d56dd48c3688be1_at_1518456600_26467_F6E7F7FE-DC53-456F-B8EC-B624BAF5E1B4",0,"3142118D-D20E-4C18-B6EC-EE7B69A5F00B",,,\n"scheduleradminpostfixRMD51d56dd48c3688be1_at_1518456600_26467_F6E7F7FE-DC53-456F-B8EC-B624BAF5E1B4",0,"F6E7F7FE-DC53-456F-B8EC-B624BAF5E1B4",,,\n"scheduleradminpostfix_RMD51d56dd48c3688be1_at_1518460200_0_CC2A8F3B-A392-4C0D-8914-F611CE068DFB",0,"314211 ...{skipping 103210 bytes}... _app_netapp","tsidx-perf-system-ontap",1,1518461700,,,,,\nnobody,SplunkforPaloAltoNetworks,"WildFire Reports - Retrieve Report",1,1518461460,,,,,\nadmin,"splunk_app_netapp","tsidx-perf-disk-ontap",1,1518461700,,,,,\nadmin,"splunk_app_netapp","tsidx-perf-quota-ontap",1,1518461700,,,,,\nadmin,"splunk_app_netapp","tsidx-perf-qtree-ontap",1,1518461700,,,,,\n'} (size=1)}, splunk_min_version -> ParamType: _dataType=unset _isMultiValue=false {_values: {[0]='6.5.0'} (size=1)}, } _m.size=14\n Messages:\n}\n)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...