topic Re: internal index in Splunk Enterprise

internal index

Kenny_splunk — Mon, 10 Feb 2025 14:38:22 GMT

Hey guys, my el basically tells me that we're going to be deep diving on the indexes in our env to extract some usage data and optimize some of the intake. We will mostly be in the search app, writing queries to pull this info. Usually in the audit index, trying to find what KO's/indexes/searches/etc are being used, whats not being used and just overall monitoring. any advice or tips on this?

Re: internal index

Kenny_splunk — Mon, 10 Feb 2025 14:51:59 GMT

Know who is logging into what Splunk systems
You know what systems searches are being performed on.
What searches are being performed,
What commands are being used in a search (think spl key words such as search, lookup, join, append, mvcount, etc)
What sourcetypes, lookups, eventtypes, etc are being searched
What dashboards are being visited

etc

Re: internal index

richgalloway — Mon, 10 Feb 2025 15:14:34 GMT

This .conf24 presentation should have some useful information.

GitHub - TheWoodRanger/presentation-conf_24_audittrail_native_telemetry

Re: internal index

livehybrid — Mon, 10 Feb 2025 16:37:22 GMT

At a high level, the following searches can be start points for the information you're looking for.

1. Audit index queries: -

Use "index=_audit" to explore usage data
Look for sourcetypes like "audittrail" and "searches"

2. Knowledge Object (KO) usage:

Check for saved searches, reports, and dashboards usage
Use "index=_audit action=search search_id=*" to find executed searches
Check "index=_internal sourcetype=splunkd_conf" for configuration changes

3. Index usage:

Analyze "index=_internal sourcetype=splunkd_access" for index access patterns
Use "index=_introspection sourcetype=splunk_resource_usage" for resource usage

4. Search performance:

Examine "index=_audit action=search" for slow searches
Look at "index=_internal sourcetype=scheduler" for scheduled search performance

5. Data intake:

Review "index=_internal sourcetype=splunkd" for forwarder and receiver logs

You could also look at the Alerts for Splunk Admins app on Splunkbase which has a good bunch of searches baked in (https://splunkbase.splunk.com/app/3796)

Please let me know how you get on and consider upvoting/karma this answer if it has helped.
Regards

Will

Re: internal index

Kenny_splunk — Mon, 10 Feb 2025 16:49:07 GMT

Thanks! im a bit new to the splunk community forum. But if i accept this as the solution, will it prevent other users from still inputting advice?

Re: internal index

Kenny_splunk — Mon, 10 Feb 2025 16:49:30 GMT

Thank you brother! I'm checking it out as we speak

Re: internal index

livehybrid — Mon, 10 Feb 2025 18:15:33 GMT

Hi @Kenny_splunk

other people will still be able to reply but the one accepted will be at the top to allow others to see it easily if they come across the same questions.
Thanks!

will

Re: internal index

Kenny_splunk — Wed, 12 Feb 2025 14:23:48 GMT

understood, so my el basicallly hands me an index today and tells me to investigate it. My anxiety is going through the roof. Please...and tips and advice and best practice?