After upgrading to Splunk 6.1, I have searches returning no results, and a warning stating "Failed to start search on peer" What is this and how do I fix it?

[jrodman]

After upgrading to 6.1, searches fail to start. When running interactive searches from the search view, the event viewer shows "no results found" and a search warning under the "Jobs" icon appears, which upon closer inspection states "Failed to start search on peer '[peer_name]'."

Inspecting the search.log of the affected search shows the following errors just before the search process shuts down:

(...)
05-08-2014 02:02:52.892 INFO  SearchParser - PARSING: typer | tags
05-08-2014 02:02:52.908 ERROR SearchParser - Could not find macro 'ns_index' that takes 0 arguments. Expecting stanza name 'ns_index'.
05-08-2014 02:02:52.908 ERROR ProviderQueue - Error while creating result provider: Error in 'SearchParser': Could not find macro 'ns_index' that takes 0 arguments. Expecting stanza name 'ns_index'.
(...)

What is happening here? Why are searches failing? How can I fix this problem?

[hexx · Accepted solution]

UPDATE: Maintenance release 6.1.1 has been made available to resolve this issue. If you are experiencing this problem, please update to Splunk Enterprise 6.1.1.

---

This problem is caused by the system-wide export by an app of eventtypes that reference one or more macros that are not exported. When searching from a different app, the eventtype is encountered but the macro it references cannot be expanded, which causes the search failure.

The appropriate behavior here would be a soft-fail: the search should continue and emit a warning about the eventtype that could not be evaluated.

This issue has been identified as bug SPL-83868, which is specific to Splunk 6.1 and will be promptly fixed in maintenance release 6.1.1.

In the meantime, you can use the following steps as a work-around:

• Inspect the search.log of a failing search (check the peer's search.log if a distributed search) to identify the macro(s) that cannot be found

• Inspect macros.conf using btool, locate the app(s) where the macro belongs. In our example, the ns_index macro is failing, so to find the app where it lives we run:
  

  
  
  $SPLUNK_HOME/bin/splunk cmd btool macros list ns_index --debug
  
  

• Add the following stanza to the app's metadata/local.meta file to export the macro(s):
  

  
  
  [macros]
  
  export = system
  
  

• Hit the following Splunk Web refresh endpoint or restart Splunk:
  

  
  
  http[s]://:/en-US/debug/refresh?entity=admin/macros
  
  

---

Note that the following apps have been found to trigger this bug by shipping macro-based eventtypes that are exported system-wide without exporting the macros they depend on:

• Windows DHCP
• Splunk for Citrix Netscaler with AppFlow
• Splunk for Blue Coat ProxySG

[araitz]

Someone should fix that DHCP app...