topic Re: Need to find where index was used in lookups, reports, alerts and dashboards. in Splunk Search

Need to find where index was used in lookups, reports, alerts and dashboards.

bwheelerice — Fri, 12 Jul 2024 18:09:13 GMT

The data coming into one of our indexers recently changed. Now the format is different, and the fields are different. The values are basically the same. I need to be able to find where this index and data was being used in our environment's lookups, reports, alerts and dashboards. Any idea how this can be accomplished?

Re: Need to find where index was used in lookups, reports, alerts and dashboards.

PickleRick — Fri, 12 Jul 2024 19:33:40 GMT

A short answer - there is no way to reliably find all its uses.

A longer answer - there are so many ways to "use" an index that while you can try to "grep" some of them (mostly by calling proper REST endpoints and filtering the output properly), you can never tell whether someone didn't use some macro that expands to "index=whatever" or a subsearch resulting in such condition when evaluated.

So you can try but they can hide 😉

BTW, how would you want to "use index in a lookup"?

Re: Need to find where index was used in lookups, reports, alerts and dashboards.

inventsekar — Fri, 12 Jul 2024 20:36:26 GMT

This is where the good documentation of the Splunk project will come in handy.

As @PickleRick suggested, this task will need the long route and manual work may give the headache.

thanks.

Re: Need to find where index was used in lookups, reports, alerts and dashboards.

richgalloway — Fri, 12 Jul 2024 20:49:33 GMT

@PickleRickis correct in that this is a Hard Problem. For further discussion on it, along with ways to (partially) solve it, see the .conf24 talk I co-produced at GitHub - TheWoodRanger/presentation-conf_24_audittrail_native_telemetry

Re: Need to find where index was used in lookups, reports, alerts and dashboards.

bwheelerice — Fri, 12 Jul 2024 21:09:00 GMT

My main goal is to go to all locations and make sure that the SPL that is being used along with index=<my_index> is actually doing what it should because the data changed from csv format to json. And the field value pairs changed their naming convention. All because our environment chose to change the application it uses to gather said data. I have been tasked with this and I was not even the person who created all these searches/reports/alerts etc... So no tribal knowledge which led me to this forum.

Re: Need to find where index was used in lookups, reports, alerts and dashboards.

PickleRick — Sat, 13 Jul 2024 05:40:44 GMT

Ok. So you _probably_ should handle it completely differently. It has nothing to do with the index itself. Data format has changed which means you should use another sourcetype. In that new sourcetype you should provide a compatibility layer for the old one - creating aliases, lookups and calculated fields to match old sourcetype's fields. It would be even better if your sourcetypes were CIM-compatible and you were searching from datamodels but I don't suppose that's the case.

Re: Need to find where index was used in lookups, reports, alerts and dashboards.

SanjayReddy — Sun, 14 Jul 2024 18:11:03 GMT

Hi @bwheelerice

recently I did similar exercise

if you have backend access

on Searchead CLI navigate to $PLUNK_HOME/etc

run command

grep -Ril "index=<indexname>"

it will list wherever index=<indexname> present with location
its covers etc/apps/ and etc/users

However, one problem is that you need to do it based on your needs.

At least it worked for me to find index names and replace them wherever needed.

searching of index name depends on component
for ex: on deployment server you need to run in location $PLUNK_HOME/etc/deployment-apps

on cluster manager $PLUNK_HOME/etc/manager-apps or master-apps

it works for any keyword that you want to lookfor

Re: Need to find where index was used in lookups, reports, alerts and dashboards.

PickleRick — Sun, 14 Jul 2024 18:25:00 GMT

1. If you really want to brute-force your way through configs, don't just do grep -R over everything because you'll be doing - for example - searching through a whole lot of java code if you have DBConnect installed. It's enough to do

find $SPLUNK_HOME/etc -type f -name \*.conf | xargs grep "index=whatever"

2. It only finds those cases where there is an explicit index=something condition in the search. I know it's relatively uncommon, but index can be specified in another way - for example with a use of a macro. The index can also be specified with a wildcard. There are more fancy ways of dynamically specifying index to search. You won't find them this way.

Re: Need to find where index was used in lookups, reports, alerts and dashboards.

SanjayReddy — Mon, 02 Sep 2024 18:36:12 GMT

Thanks @PickleRick for information.

I will try use options you menioned as well