Need to find where index was used in lookups, reports, alerts and dashboards.

[bwheelerice]

The data coming into one of our indexers recently changed. Now the format is different, and the fields are different. The values are basically the same. I need to be able to find where this index and data was being used in our environment's lookups, reports, alerts and dashboards. Any idea how this can be accomplished?

[bwheelerice]

My main goal is to go to all locations and make sure that the SPL that is being used along with index=<my_index> is actually doing what it should because the data changed from csv format to json. And the field value pairs changed their naming convention. All because our environment chose to change the application it uses to gather said data. I have been tasked with this and I was not even the person who created all these searches/reports/alerts etc... So no tribal knowledge which led me to this forum. 

[SanjayReddy]

Hi  @bwheelerice 

recently I did similar exercise 

if you have backend access 

on Searchead CLI  navigate to $PLUNK_HOME/etc

run command

grep -Ril "index=<indexname>"

it will list wherever index=<indexname> present with location 
its covers etc/apps/ and etc/users

However, one problem is that you need to do it  based on your needs.

At least it worked for me to find index names and replace them wherever needed.

searching of index name depends on component 
for ex: on deployment server you need to run in location  $PLUNK_HOME/etc/deployment-apps 

on cluster manager $PLUNK_HOME/etc/manager-apps or master-apps

it works for any keyword that you want to lookfor

[PickleRick]

1. If you really want to brute-force your way through configs, don't just do grep -R over everything because you'll be doing - for example - searching through a whole lot of java code if you have DBConnect installed. It's enough to do

find $SPLUNK_HOME/etc -type f -name \*.conf | xargs grep "index=whatever"

2. It only finds those cases where there is an explicit index=something condition in the search. I know it's relatively uncommon, but index can be specified in another way - for example with a use of a macro. The index can also be specified with a wildcard. There are more fancy ways of dynamically specifying index to search. You won't find them this way.

[SanjayReddy]

Thanks @PickleRick  for information.

I will try use options you menioned as well 

[PickleRick]

Ok. So you _probably_ should handle it completely differently. It has nothing to do with the index itself. Data format has changed which means you should use another sourcetype. In that new sourcetype you should provide a compatibility layer for the old one - creating aliases, lookups and calculated fields to match old sourcetype's fields. It would be even better if your sourcetypes were CIM-compatible and you were searching from datamodels but I don't suppose that's the case.

[richgalloway]

@PickleRickis correct in that this is a Hard Problem.  For further discussion on it, along with ways to (partially) solve it, see the .conf24 talk I co-produced at GitHub - TheWoodRanger/presentation-conf_24_audittrail_native_telemetry 

[inventsekar]

This is where the good documentation of the Splunk project will come in handy. 

As @PickleRick suggested, this task will need the long route and manual work may give the headache. 

thanks. 

[PickleRick]

A short answer - there is no way to reliably find all its uses.

A longer answer - there are so many ways to "use" an index that while you can try to "grep" some of them (mostly by calling proper REST endpoints and filtering the output properly), you can never tell whether someone didn't use some macro that expands to "index=whatever" or a subsearch resulting in such condition when evaluated.

So you can try but they can hide 😉

BTW, how would you want to "use index in a lookup"?