Understanding Introspection Logs: Difference between "historical", "historical batch" & various sid formats

[anirbandasdeb]

First question:
If I run the below command, I get four different values (historical, historical batch, real-time, or real-time indexed) for data.search_props.mode

index IN (_introspection) sourcetype=splunk_resource_usage component=PerProcess data.search_props.sid=*
| stats count by data.search_props.type data.search_props.mode

• What exactly is the difference between "historical" & "historical batch"?
• When is "historical" mode used over "historical batch"?
• What impacts does these modes have over searches running on a multisite indexer cluster with a multisite stretched searchhead cluster?

Splunk Docs has this https://docs.splunk.com/Documentation/Splunk/7.2.6/Troubleshooting/Sampleplatforminstrumentationsear... , but they didnt explain what the different modes signify.

Second Question:
I found different forms of search IDs in the introspection logs. Samples:

- 1576061498.2185156_3863A2B4-1AB7-42DB-ACD3-4484EDD006D2
- 1576063020.97915
- userid__userid__search__search12_1576060747.2056595_CD71F91B-FF33-490B-8C4B-EE986A5C4E6F
- subsearch_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675_1576060142.3
- remote_hostname_1576063020.3359
- remote_hostname_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675
- remote_hostname_subsearch_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675_1576060142.3

I understand that sids with userid__userid__search__search12 spawn from dashboard panels, subsearch_userid__userid_ spawn from subsearches in dashboard panels, sids like 1576063020.97915 are searches run from the search box, etc..

But the ones like 1576061498.2185156_3863A2B4-1AB7-42DB-ACD3-4484EDD006D2, remote_hostname_1576063020.3359, remote_hostname_subsearch_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675_1576060142.3...
- What are origins of these search sids?
- How is any given sid generated?

The idea is to come up with a way to correlate these sids from the _introspection logs to the _audit so we can identify the searches/dashboards/alerts that take up most resources on our Splunk platform and tune them.

Third Question
What is the use of data.pid?
How do I leverage this?

Google didn't come up with anything solid.
Please help me understand these.
Any documentation links/answers would be greatly appreciated helpful.

[anirbandasdeb]

Found this cool answer: https://answers.splunk.com/answers/697506/how-to-map-introspection-datasearch-propssid-to-th.html