Hi everyone,
I have a power user search that I am having problems with.
When the dashboard loads the search in one of the panels fails, I get a subsearch error from each indexers in my deployment.
Yet if I open the search in a separate window (Open in Search) it works.
Does anyone know what would cause this?
Thanks,
SPL Below>>>
index=testsearch sourcetype=transaction (application="SharePoint Static" OR application="SharePointStatic" OR application="SharePointUpload" OR application="SPTeamSiteMySite*") transaction="*UploadTestFile3MB" transaction_status="Success"
|eval secs=round(duration/1000)
|lookup local=1 testsearch_network_mapping_file.csv hostname OUTPUT longitude latitude UserCount DisplayName
|search (UserCount > 0 AND UserCount < 100000000) DisplayName != "Frank - Campus (" |rangemap field=secs Good=0-3 Fair=3-4 Bad=4-2000
|stats count AS TransTot max(UserCount) AS Users max(longitude) AS long max(latitude) AS lat BY DisplayName range
|join DisplayName [search index=testsearch sourcetype=transaction (application="SharePoint Static" OR applicaton="SharePointStatic" OR application="SharePointUpload" OR application="SPTeamSiteMySite") transaction="*UploadTestFile3MB"
|lookup testsearch_network_mapping_file.csv hostname OUTPUT DisplayName
|stats count AS SiteTotal by DisplayName]
|eval ChartValue=TransTot*Users/SiteTotal
|table *
|geostats latfield=lat longfield=long sum(ChartValue) by range
|eval s=Good+Bad+Fair
|eval Good%=round((Good / s) * 100,2)
|eval Bad%= round((Bad / s ) *100,2)
|eval Fair%=round((Fair / s) * 100,2)
| fields - s
Here is my fix/RCA.
Turns out the root cause was using Windows 2008R2 for the index tier.
Using SSO AD accounts that have FQDN meant the hashed value of the search sid exceeded the character limit of the server. This was identified by using the | history command to see the difference.
Ultimately, we're migrating the index tier to RHEL and as a workaround, I'm creating shared local accounts to facilitate access.
More intel.
Turns out there is a bug/fix in Splunk 6.4.5 where they shortened a temp file from 30 characters to 16.
We installed 6.4.9 on the index tier and the problem went away.
The search isn't by chance derived from a base search? If so, you might have to define which fields are going to be handed down by the base search to the sub-searches in the dashboard.
@hettervi,
good question. I am working through the syntax to figure that out. I'm thinking the join might be part of the problem.
So what is the error message you get from the indexers? Can you please check the job inspector and post the error as well, please ?
MuS,
Here are the indexer related errors..
The indexers are windows 2008R2 (to be migrated to RHEL very soon).
error : [subsearch]: [PRODASRV235] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV274] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV354] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV411] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV423] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV630] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV237] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV238] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV239] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV321] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV374] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV380] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV381] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV630] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [KULAPP235] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [KULAPP236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [LHDAPP235] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [LHDAPP236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
(SID: YXJiZWViZUBuYS54b20uY29t_YXJiZWViZUBuYS54b20uY29t_MTAwX1hPTV9FMkVfc2hhcmVwb2ludA_search1_1511303688.4262_588B113C-A2D3-42E1-ACE2-8E5D47E3C0DE) search.log
Would the length of the sid be a problem? I noticed that when I run the search manually, the sid is much shorter.