topic Re: Specific Search not working after upgrade to Splunk 5.0 in Splunk Search

Specific Search not working after upgrade to Splunk 5.0

tiny3001 — Wed, 20 Feb 2013 07:50:29 GMT

I've got a specific search, that generates two time ranges on a timechart, using the instructions found on this Splunk blog entry:
Compare two time ranges in one report

This search worked perfectly... until yesterday, when we upgraded our search heads to Splunk 5.0.2.

I've confirmed this, by running the exact search on another search head that is still running Splunk 4.3.3. The search works on Splunk 4, does not work on Splunk 5.

Here is the (edited for privacy) search:

index="summary" search_name="Event Count Summary" earliest=-28d@d latest=@d
| eval Month="This Month"
| append [search index="summary" search_name="Event Count Summary" earliest=-56d@d latest=-28d@d
  | eval Month="Last Month"
  | eval new_time=_time+2419200]
| eval _time=if(isnotnull(new_time), new_time, _time)

On the 5.0.2 search head, I only get results for "Last Month" and only "Last Month" gets graphed. On the 4.3.3 search head, everything gets graphed perfectly.

I suspected possible timeouts in the subsearches, but the limits.conf between the two search heads are almost identical.

How can a Splunk upgrade break a search? Where do I need to look?

Edit: (2013/04/15)
The comment below is correct, this definitely has to do with the 'append' command. Still no answer on this though, but in the meantime, I have taken it up with Splunk Enterprise support. Haven't heard from them in over a month now...

Re: Specific Search not working after upgrade to Splunk 5.0

jevenson — Thu, 21 Mar 2013 16:35:00 GMT

I have a similar issue, and I'm thinking it may have something to do with the 'append' command. My issue can be seen here: http://splunk-base.splunk.com/answers/74749/fillnull-not-working-on-my-search

Re: Specific Search not working after upgrade to Splunk 5.0

daniel_splunk — Mon, 15 Apr 2013 08:32:05 GMT

If your indexer is still under 4.3.x, you may hit this.

Searches that contain subsearches do not return data in environments where search heads are running version 5.0.x and indexers are running version 4.3.x. To work around this issue, upgrade indexers to the same version as the search heads. (SPL-62457)

This is documented in the known issue.

http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Knownissues

Re: Specific Search not working after upgrade to Splunk 5.0

tiny3001 — Wed, 17 Apr 2013 07:56:49 GMT

Absolutely the correct answer. Ironically, the one thing that could have solved our problem (upgrading the 4.3.3 indexer) was the one thing that could have solved our problem. We were too scared to upgrade after we picked up the issue and wanted to understand it completely before we upgraded everything.

Re: Specific Search not working after upgrade to Splunk 5.0

tiny3001 — Wed, 17 Apr 2013 07:57:42 GMT

Also, closing the Splunk Enterprise support call... not sure why they didn't pick up on this? Maybe my question wasn't clear enough?