Splunk Search

After upgrading to Splunk 6.1, I have searches returning no results, and a warning stating "Failed to start search on peer" What is this and how do I fix it?

jrodman
Splunk Employee
Splunk Employee

After upgrading to 6.1, searches fail to start. When running interactive searches from the search view, the event viewer shows "no results found" and a search warning under the "Jobs" icon appears, which upon closer inspection states "Failed to start search on peer '[peer_name]'."

Inspecting the search.log of the affected search shows the following errors just before the search process shuts down:

(...)
05-08-2014 02:02:52.892 INFO  SearchParser - PARSING: typer | tags
05-08-2014 02:02:52.908 ERROR SearchParser - Could not find macro 'ns_index' that takes 0 arguments. Expecting stanza name 'ns_index'.
05-08-2014 02:02:52.908 ERROR ProviderQueue - Error while creating result provider: Error in 'SearchParser': Could not find macro 'ns_index' that takes 0 arguments. Expecting stanza name 'ns_index'.
(...)

What is happening here? Why are searches failing? How can I fix this problem?

1 Solution

hexx
Splunk Employee
Splunk Employee

UPDATE: Maintenance release 6.1.1 has been made available to resolve this issue. If you are experiencing this problem, please update to Splunk Enterprise 6.1.1.




This problem is caused by the system-wide export by an app of eventtypes that reference one or more macros that are not exported. When searching from a different app, the eventtype is encountered but the macro it references cannot be expanded, which causes the search failure.

The appropriate behavior here would be a soft-fail: the search should continue and emit a warning about the eventtype that could not be evaluated.

This issue has been identified as bug SPL-83868, which is specific to Splunk 6.1 and will be promptly fixed in maintenance release 6.1.1.

In the meantime, you can use the following steps as a work-around:

  • Inspect the search.log of a failing search (check the peer's search.log if a distributed search) to identify the macro(s) that cannot be found

  • Inspect macros.conf using btool, locate the app(s) where the macro belongs. In our example, the ns_index macro is failing, so to find the app where it lives we run:


    $SPLUNK_HOME/bin/splunk cmd btool macros list ns_index --debug

  • Add the following stanza to the app's metadata/local.meta file to export the macro(s):


    [macros]
    export = system

  • Hit the following Splunk Web refresh endpoint or restart Splunk:


    http[s]://:/en-US/debug/refresh?entity=admin/macros




Note that the following apps have been found to trigger this bug by shipping macro-based eventtypes that are exported system-wide without exporting the macros they depend on:

View solution in original post

hexx
Splunk Employee
Splunk Employee

UPDATE: Maintenance release 6.1.1 has been made available to resolve this issue. If you are experiencing this problem, please update to Splunk Enterprise 6.1.1.




This problem is caused by the system-wide export by an app of eventtypes that reference one or more macros that are not exported. When searching from a different app, the eventtype is encountered but the macro it references cannot be expanded, which causes the search failure.

The appropriate behavior here would be a soft-fail: the search should continue and emit a warning about the eventtype that could not be evaluated.

This issue has been identified as bug SPL-83868, which is specific to Splunk 6.1 and will be promptly fixed in maintenance release 6.1.1.

In the meantime, you can use the following steps as a work-around:

  • Inspect the search.log of a failing search (check the peer's search.log if a distributed search) to identify the macro(s) that cannot be found

  • Inspect macros.conf using btool, locate the app(s) where the macro belongs. In our example, the ns_index macro is failing, so to find the app where it lives we run:


    $SPLUNK_HOME/bin/splunk cmd btool macros list ns_index --debug

  • Add the following stanza to the app's metadata/local.meta file to export the macro(s):


    [macros]
    export = system

  • Hit the following Splunk Web refresh endpoint or restart Splunk:


    http[s]://:/en-US/debug/refresh?entity=admin/macros




Note that the following apps have been found to trigger this bug by shipping macro-based eventtypes that are exported system-wide without exporting the macros they depend on:

View solution in original post

araitz
Splunk Employee
Splunk Employee

Someone should fix that DHCP app...

Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.