index investigation

[Kenny_splunk]

So we are starting a new project soon, and basically our boss is personally sending me an index (not internal) to investigate.

Investigate as far as as far as usage. We are trying to optimize the env and cut whats not being used, or checking to see what is being overused. KO'S, data intake, etc.

Any good practices, processes or tips you can lend? this would be the most perfect learning opportunity. Im excited, but nervous.

[livehybrid]

Hi @Kenny_splunk 

I think the best place to start here is by checking the _audit index to see who is using/searching aginst the index in question...

Start off with the following query and take it from there:

index=_audit search="*<yourIndexName>*" info=completed action=search

Its important to remember, however, than some people might search for index=* in order to access a particular index, which might not come up in the above search. They might also use something like win* instead of win_events. 
People can use index="yourName", index=yourName, index IN (yourName,anotherName) etc etc which is why I included the wildcards either side for the above sample query. You might want to tune to your environment etc as you see fit!

In these logs you should find a number of useful fields, such as "search" (what they ran) and "user" (Who ran it) amonst other things llike event_count and result_count.

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

[kiran_panchavat]

@Kenny_splunk 

• Find sourcetypes that are consuming a lot of data, especially unnecessary logs
• Reduce retention or delete them if they are no longer needed.
• If multiple indexes contain similar data, consolidate where possible.

[kiran_panchavat]

@Kenny_splunk 

Use the tstats command to track index usage over time. This will help you identify peaks and patterns in data usage.

• Review and adjust your index retention policies to ensure that data is stored only for as long as needed. This can help reduce storage costs.
• Review saved searches and reports to ensure they are still relevant and being used. Disable or delete those that are not needed.
• Optimize your searches by using efficient search commands and avoiding unnecessary subsearches. Use summary indexing and data models for faster results.

Index Usage Over Time: