Splunk Search

Specific Search not working after upgrade to Splunk 5.0

tiny3001
Path Finder

I've got a specific search, that generates two time ranges on a timechart, using the instructions found on this Splunk blog entry:
Compare two time ranges in one report

This search worked perfectly... until yesterday, when we upgraded our search heads to Splunk 5.0.2.

I've confirmed this, by running the exact search on another search head that is still running Splunk 4.3.3. The search works on Splunk 4, does not work on Splunk 5.

Here is the (edited for privacy) search:

index="summary" search_name="Event Count Summary" earliest=-28d@d latest=@d
| eval Month="This Month"
| append [search index="summary" search_name="Event Count Summary" earliest=-56d@d latest=-28d@d
  | eval Month="Last Month"
  | eval new_time=_time+2419200]
| eval _time=if(isnotnull(new_time), new_time, _time)

On the 5.0.2 search head, I only get results for "Last Month" and only "Last Month" gets graphed. On the 4.3.3 search head, everything gets graphed perfectly.

I suspected possible timeouts in the subsearches, but the limits.conf between the two search heads are almost identical.

How can a Splunk upgrade break a search? Where do I need to look?

Edit: (2013/04/15)
The comment below is correct, this definitely has to do with the 'append' command. Still no answer on this though, but in the meantime, I have taken it up with Splunk Enterprise support. Haven't heard from them in over a month now...

Tags (3)
1 Solution

daniel_splunk
Splunk Employee
Splunk Employee

If your indexer is still under 4.3.x, you may hit this.

Searches that contain subsearches do not return data in environments where search heads are running version 5.0.x and indexers are running version 4.3.x. To work around this issue, upgrade indexers to the same version as the search heads. (SPL-62457)

This is documented in the known issue.

http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Knownissues

View solution in original post

daniel_splunk
Splunk Employee
Splunk Employee

If your indexer is still under 4.3.x, you may hit this.

Searches that contain subsearches do not return data in environments where search heads are running version 5.0.x and indexers are running version 4.3.x. To work around this issue, upgrade indexers to the same version as the search heads. (SPL-62457)

This is documented in the known issue.

http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Knownissues

tiny3001
Path Finder

Also, closing the Splunk Enterprise support call... not sure why they didn't pick up on this? Maybe my question wasn't clear enough?

0 Karma

tiny3001
Path Finder

Absolutely the correct answer. Ironically, the one thing that could have solved our problem (upgrading the 4.3.3 indexer) was the one thing that could have solved our problem. We were too scared to upgrade after we picked up the issue and wanted to understand it completely before we upgraded everything.

0 Karma

jevenson
Path Finder

I have a similar issue, and I'm thinking it may have something to do with the 'append' command. My issue can be seen here: http://splunk-base.splunk.com/answers/74749/fillnull-not-working-on-my-search

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...