Dashboards & Visualizations

Why does this search fail when the dashboard panel loads, but runs successfully when executed manually.

Path Finder

Hi everyone,

I have a power user search that I am having problems with.

When the dashboard loads the search in one of the panels fails, I get a subsearch error from each indexers in my deployment.

Yet if I open the search in a separate window (Open in Search) it works.

Does anyone know what would cause this?

Thanks,

SPL Below>>>
index=testsearch sourcetype=transaction (application="SharePoint Static" OR application="SharePointStatic" OR application="SharePointUpload" OR application="SPTeamSiteMySite*") transaction="*UploadTestFile3MB" transaction_status="Success"

|eval secs=round(duration/1000)
|lookup local=1 testsearch_network_mapping_file.csv hostname OUTPUT longitude latitude UserCount DisplayName
|search (UserCount > 0 AND UserCount < 100000000) DisplayName != "Frank - Campus (" |rangemap field=secs Good=0-3 Fair=3-4 Bad=4-2000

|stats count AS TransTot max(UserCount) AS Users max(longitude) AS long max(latitude) AS lat BY DisplayName range
|join DisplayName [search index=testsearch sourcetype=transaction (application="SharePoint Static" OR applicaton="SharePointStatic" OR application="SharePointUpload" OR application="SPTeamSiteMySite
") transaction="*UploadTestFile3MB"
|lookup testsearch_network_mapping_file.csv hostname OUTPUT DisplayName
|stats count AS SiteTotal by DisplayName]
|eval ChartValue=TransTot*Users/SiteTotal
|table *
|geostats latfield=lat longfield=long sum(ChartValue) by range
|eval s=Good+Bad+Fair
|eval Good%=round((Good / s) * 100,2)
|eval Bad%= round((Bad / s ) *100,2)
|eval Fair%=round((Fair / s) * 100,2)
| fields - s

0 Karma

Path Finder

Here is my fix/RCA.

Turns out the root cause was using Windows 2008R2 for the index tier.

Using SSO AD accounts that have FQDN meant the hashed value of the search sid exceeded the character limit of the server. This was identified by using the | history command to see the difference.

Ultimately, we're migrating the index tier to RHEL and as a workaround, I'm creating shared local accounts to facilitate access.

0 Karma

Path Finder

More intel.

Turns out there is a bug/fix in Splunk 6.4.5 where they shortened a temp file from 30 characters to 16.

We installed 6.4.9 on the index tier and the problem went away.

0 Karma

Builder

The search isn't by chance derived from a base search? If so, you might have to define which fields are going to be handed down by the base search to the sub-searches in the dashboard.

0 Karma

Path Finder

@hettervi,

good question. I am working through the syntax to figure that out. I'm thinking the join might be part of the problem.

0 Karma

SplunkTrust
SplunkTrust

So what is the error message you get from the indexers? Can you please check the job inspector and post the error as well, please ?

0 Karma

Path Finder

MuS,

Here are the indexer related errors..

The indexers are windows 2008R2 (to be migrated to RHEL very soon).

error : [subsearch]: [PRODASRV235] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV274] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV354] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV411] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV423] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV630] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV237] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV238] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV239] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV321] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV374] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV380] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV381] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV630] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [KULAPP235] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [KULAPP236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [LHDAPP235] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [LHDAPP236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
(SID: YXJiZWViZUBuYS54b20uY29t_YXJiZWViZUBuYS54b20uY29t_MTAwX1hPTV9FMkVfc2hhcmVwb2ludA_search1_1511303688.4262_588B113C-A2D3-42E1-ACE2-8E5D47E3C0DE) search.log

0 Karma

Path Finder

Would the length of the sid be a problem? I noticed that when I run the search manually, the sid is much shorter.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!