Improve speed of "append"

[therealdpk]

I am trying to use append to combine the results of two searches, but when I use append the search time explodes. Individually, the two searches take a few seconds each. Together, the search takes over five minutes. The full search:

index=foo sourcetype=foo_st | append [search index=bar sourcetype=bar_st] | timechart count by index

over a 24 hour period.

Individually, the searches find a small set of results (336k and 42k respectively). Together, with the above append command, the Search Job Inspector reports that the append component has an input count of 57,000,000,000 and an output count of 58,000,000, which seems completely insane and arbitrary. timechart ends up with an input count of 58,000,042,474. I suspect that append is somehow multiplying the result rows or something.

FWIW, the "matching event" counter on the UI says 379,453 events. And this is with Splunk 4.3.2.

Has anyone else seen this issue and, if so, what did you do to resolve it? What can I view using the Splunk UI to try to diagnose the problem?

I've left this question open in case someone knows how to diagnose the 57-58 billion result problem, or how to use the UI to diagnose the problem.

[chris]

You could just try to avoid using append and the subsearch by doing trying the following:

 (index=foo sourcetype=foo_st) OR ( index=bar  sourcetype=bar_st) | timechart count by index

I haven't run accross the exploding number of results though.

Chris

[therealdpk]

Ahh, excellent. That makes sense.

FWIW, I got the idea to use append from http://blogs.splunk.com/2012/02/19/compare-two-time-ranges-in-one-report/ , where the same search is performed against two dates in order to compare the results. I was using it to compare Monday to last Monday and two Mondays ago and so on, and had the same multiple-billions result. I gave up on a fix for that, but then I found this much simpler way to demonstrate the problem.