Imagine I have a bunch of indexes named app1, app2, app3, .... appN. Assuming I have search permissions on all of them, then I can run the following search to quickly (1 sec) get a list of existing app# indexes.
| eventcount summarize=f index=app* | eval index_app=index | stats count by index_app
It seems like daily someone creates a new app and doesn't bother to tell me. The way I deal with this is to reroute log events from unknown apps in a CatchAll index. Periodically I want to look at what is in the catchall index to see what new apps exist.
This search works to get a list of apps that appear in the catch all index...taking only a couple seconds to run over the last few days
index=CatchAll | rex field=_raw "index=\"(?<index_app>\w+)\"" | dedup index_app
So one search gives me the list of existing app indexes and the other is a set of app indexes which may include new ones.
My goal is to figure out app indexes I need to create, and it seems like I should be able use these two searches together to get the answer quickly.
I tried to combine
index=CatchAll | rex field=_raw "index=\"(?<index_app>\w+)\"" | dedup index_app |join left [| eventcount summarize=f index=app* | eval index_app=index | stats count by index_app]
But this both doesn't seem to work and is super slow.
It's strange because it seems like it should be 2 seconds for the first search, then 2 seconds for the second, and a fraction to diff them.
Try this
| set diff [| eventcount summarize=f index=app* | eval index_app=index | stats count by index_app | fields - count] [search index=CatchAll | rex field=_raw "index=\"(?<index_app>\w+)\"" | dedup index_app]
If you only need to find which indexes that have not been created properly, you could check for these events in the "_internal" index;
02-14-2014 09:40:57.755 +0100 WARN IndexProcessor - received event for unconfigured/disabled/deleted index='some_index_name' with source='source::c:\temp\blah.log' host='host::ServerX' sourcetype='sourcetype::bleh' (1 missing total)
/K
Kristian, thanks for your idea...definitely a good way to find unconfigured indexes directly. I think I am still interested in diffing the two sets for my particular use case (I don't get these unconfigured/disabled index events)