First question:
If I run the below command, I get four different values (historical, historical batch, real-time, or real-time indexed) for data.search_props.mode
index IN (_introspection) sourcetype=splunk_resource_usage component=PerProcess data.search_props.sid=*
| stats count by data.search_props.type data.search_props.mode
- What exactly is the difference between "historical" & "historical batch"?
- When is "historical" mode used over "historical batch"?
- What impacts does these modes have over searches running on a multisite indexer cluster with a multisite stretched searchhead cluster?
Splunk Docs has this https://docs.splunk.com/Documentation/Splunk/7.2.6/Troubleshooting/Sampleplatforminstrumentationsear... , but they didnt explain what the different modes signify.
Second Question:
I found different forms of search IDs in the introspection logs. Samples:
- 1576061498.2185156_3863A2B4-1AB7-42DB-ACD3-4484EDD006D2
- 1576063020.97915
- userid__userid__search__search12_1576060747.2056595_CD71F91B-FF33-490B-8C4B-EE986A5C4E6F
- subsearch_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675_1576060142.3
- remote_hostname_1576063020.3359
- remote_hostname_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675
- remote_hostname_subsearch_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675_1576060142.3
I understand that sids with userid__userid__search__search12 spawn from dashboard panels, subsearch_userid__userid_ spawn from subsearches in dashboard panels, sids like 1576063020.97915 are searches run from the search box, etc..
But the ones like 1576061498.2185156_3863A2B4-1AB7-42DB-ACD3-4484EDD006D2, remote_hostname_1576063020.3359, remote_hostname_subsearch_userid__userid_c3BsdW5rX21vbml0b3JpbmdfY29uc29sZQ__search1_1576060141.97675_1576060142.3...
- What are origins of these search sids?
- How is any given sid generated?
The idea is to come up with a way to correlate these sids from the _introspection logs to the _audit so we can identify the searches/dashboards/alerts that take up most resources on our Splunk platform and tune them.
Third Question
What is the use of data.pid?
How do I leverage this?
Google didn't come up with anything solid.
Please help me understand these.
Any documentation links/answers would be greatly appreciated helpful.
