internal index

[Kenny_splunk]

Hey guys, my el basically tells me that we're going to be deep diving on the indexes in our env to extract some usage data and optimize some of the intake. We will mostly be in the search app, writing queries to pull this info. Usually in the audit index, trying to find what KO's/indexes/searches/etc are being used, whats not being used and just overall monitoring. any advice or tips on this?

[richgalloway]

This .conf24 presentation should have some useful information.

GitHub - TheWoodRanger/presentation-conf_24_audittrail_native_telemetry

[Kenny_splunk]

Thank you brother! I'm checking it out as we speak

[Kenny_splunk]

Know who is logging into what Splunk systems
You know what systems searches are being performed on.
What searches are being performed,
What commands are being used in a search (think spl key words such as search, lookup, join, append, mvcount, etc)
What sourcetypes, lookups, eventtypes, etc are being searched
What dashboards are being visited

etc

[livehybrid]

At a high level, the following searches can be start points for the information you're looking for.

1. Audit index queries: -

• Use "index=_audit" to explore usage data
• Look for sourcetypes like "audittrail" and "searches"

2. Knowledge Object (KO) usage:

• Check for saved searches, reports, and dashboards usage
• Use "index=_audit action=search search_id=*" to find executed searches
• Check "index=_internal sourcetype=splunkd_conf" for configuration changes

3. Index usage:

• Analyze "index=_internal sourcetype=splunkd_access" for index access patterns
• Use "index=_introspection sourcetype=splunk_resource_usage" for resource usage

4. Search performance:

• Examine "index=_audit action=search" for slow searches
• Look at "index=_internal sourcetype=scheduler" for scheduled search performance

5. Data intake:

• Review "index=_internal sourcetype=splunkd" for forwarder and receiver logs

You could also look at the Alerts for Splunk Admins app on Splunkbase which has a good bunch of searches baked in (https://splunkbase.splunk.com/app/3796)

Please let me know how you get on and consider upvoting/karma this answer if it has helped.
Regards

Will

[Kenny_splunk]

Thanks! im a bit new to the splunk community forum. But if i accept this as the solution, will it prevent other users from still inputting advice?

[livehybrid]

Hi @Kenny_splunk 

other people will still be able to reply but the one accepted will be at the top to allow others to see it easily if they come across the same questions. 
Thanks!

will

[Kenny_splunk]

understood, so my el basicallly hands me an index today and tells me to investigate it. My anxiety is going through the roof. Please...and tips and advice and best practice?