Splunk Search

How can I efficiently diff results of two searches?

juniormint
Communicator

Imagine I have a bunch of indexes named app1, app2, app3, .... appN. Assuming I have search permissions on all of them, then I can run the following search to quickly (1 sec) get a list of existing app# indexes.

| eventcount summarize=f index=app* | eval index_app=index | stats count by index_app 

It seems like daily someone creates a new app and doesn't bother to tell me. The way I deal with this is to reroute log events from unknown apps in a CatchAll index. Periodically I want to look at what is in the catchall index to see what new apps exist.

This search works to get a list of apps that appear in the catch all index...taking only a couple seconds to run over the last few days

index=CatchAll | rex field=_raw "index=\"(?<index_app>\w+)\"" | dedup index_app

So one search gives me the list of existing app indexes and the other is a set of app indexes which may include new ones.

My goal is to figure out app indexes I need to create, and it seems like I should be able use these two searches together to get the answer quickly.

I tried to combine

index=CatchAll | rex field=_raw "index=\"(?<index_app>\w+)\"" | dedup index_app |join left [| eventcount summarize=f index=app* | eval index_app=index | stats count by index_app]

But this both doesn't seem to work and is super slow.

It's strange because it seems like it should be 2 seconds for the first search, then 2 seconds for the second, and a fraction to diff them.

Tags (3)
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try this

| set diff [| eventcount summarize=f index=app* | eval index_app=index | stats count by index_app | fields - count] [search index=CatchAll | rex field=_raw "index=\"(?<index_app>\w+)\"" | dedup index_app]

kristian_kolb
Ultra Champion

If you only need to find which indexes that have not been created properly, you could check for these events in the "_internal" index;

02-14-2014 09:40:57.755 +0100 WARN  IndexProcessor - received event for unconfigured/disabled/deleted index='some_index_name' with source='source::c:\temp\blah.log' host='host::ServerX' sourcetype='sourcetype::bleh' (1 missing total)

/K

0 Karma

juniormint
Communicator

Kristian, thanks for your idea...definitely a good way to find unconfigured indexes directly. I think I am still interested in diffing the two sets for my particular use case (I don't get these unconfigured/disabled index events)

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...