Deployment Architecture

Understanding Introspection Logs: Difference between "historical", "historical batch" & various sid formats

anirbandasdeb
Path Finder

First question:
If I run the below command, I get four different values (historical, historical batch, real-time, or real-time indexed) for data.search_props.mode

index IN (_introspection) sourcetype=splunk_resource_usage component=PerProcess data.search_props.sid=*
| stats count by data.search_props.type data.search_props.mode
  • What exactly is the difference between "historical" & "historical batch"?
  • When is "historical" mode used over "historical batch"?
  • What impacts does these modes have over searches running on a multisite indexer cluster with a multisite stretched searchhead cluster?

Splunk Docs has this https://docs.splunk.com/Documentation/Splunk/7.2.6/Troubleshooting/Sampleplatforminstrumentationsear... , but they didnt explain what the different modes signify.

Second Question:
I found different forms of search IDs in the introspection logs. Samples:

- 1576061498.2185156_3863A2B4-1AB7-42DB-ACD3-4484EDD006D2
- 1576063020.97915
- userid__userid__search__search12_1576060747.2056595_CD71F91B-FF33-490B-8C4B-EE986A5C4E6F
- subsearch_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675_1576060142.3
- remote_hostname_1576063020.3359
- remote_hostname_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675
- remote_hostname_subsearch_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675_1576060142.3

I understand that sids with userid__userid__search__search12 spawn from dashboard panels, subsearch_userid__userid_ spawn from subsearches in dashboard panels, sids like 1576063020.97915 are searches run from the search box, etc..

But the ones like 1576061498.2185156_3863A2B4-1AB7-42DB-ACD3-4484EDD006D2, remote_hostname_1576063020.3359, remote_hostname_subsearch_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675_1576060142.3...
- What are origins of these search sids?
- How is any given sid generated?

The idea is to come up with a way to correlate these sids from the _introspection logs to the _audit so we can identify the searches/dashboards/alerts that take up most resources on our Splunk platform and tune them.

Third Question
What is the use of data.pid?
How do I leverage this?

Google didn't come up with anything solid.
Please help me understand these.
Any documentation links/answers would be greatly appreciated helpful.

0 Karma

anirbandasdeb
Path Finder
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...