topic How can I efficiently diff results of two searches? in Splunk Search

How can I efficiently diff results of two searches?

juniormint — Fri, 14 Feb 2014 05:53:48 GMT

Imagine I have a bunch of indexes named app1, app2, app3, .... appN. Assuming I have search permissions on all of them, then I can run the following search to quickly (1 sec) get a list of existing app# indexes.

| eventcount summarize=f index=app* | eval index_app=index | stats count by index_app

It seems like daily someone creates a new app and doesn't bother to tell me. The way I deal with this is to reroute log events from unknown apps in a CatchAll index. Periodically I want to look at what is in the catchall index to see what new apps exist.

This search works to get a list of apps that appear in the catch all index...taking only a couple seconds to run over the last few days

index=CatchAll | rex field=_raw "index=\"(?<index_app>\w+)\"" | dedup index_app

So one search gives me the list of existing app indexes and the other is a set of app indexes which may include new ones.

My goal is to figure out app indexes I need to create, and it seems like I should be able use these two searches together to get the answer quickly.

I tried to combine

index=CatchAll | rex field=_raw "index=\"(?<index_app>\w+)\"" | dedup index_app |join left [| eventcount summarize=f index=app* | eval index_app=index | stats count by index_app]

But this both doesn't seem to work and is super slow.

It's strange because it seems like it should be 2 seconds for the first search, then 2 seconds for the second, and a fraction to diff them.

Re: How can I efficiently diff results of two searches?

kristian_kolb — Fri, 14 Feb 2014 08:45:03 GMT

If you only need to find which indexes that have not been created properly, you could check for these events in the "_internal" index;

02-14-2014 09:40:57.755 +0100 WARN  IndexProcessor - received event for unconfigured/disabled/deleted index='some_index_name' with source='source::c:\temp\blah.log' host='host::ServerX' sourcetype='sourcetype::bleh' (1 missing total)

Re: How can I efficiently diff results of two searches?

juniormint — Fri, 14 Feb 2014 12:29:59 GMT

Kristian, thanks for your idea...definitely a good way to find unconfigured indexes directly. I think I am still interested in diffing the two sets for my particular use case (I don't get these unconfigured/disabled index events)

Re: How can I efficiently diff results of two searches?

somesoni2 — Fri, 14 Feb 2014 21:32:12 GMT

Try this

| set diff [| eventcount summarize=f index=app* | eval index_app=index | stats count by index_app | fields - count] [search index=CatchAll | rex field=_raw "index=\"(?<index_app>\w+)\"" | dedup index_app]