topic Re: Why does this search fail when the dashboard panel loads, but runs successfully when executed manually. in Dashboards & Visualizations

Why does this search fail when the dashboard panel loads, but runs successfully when executed manually.

abeeber_2 — Tue, 29 Sep 2020 16:51:53 GMT

Hi everyone,

I have a power user search that I am having problems with.

When the dashboard loads the search in one of the panels fails, I get a subsearch error from each indexers in my deployment.

Yet if I open the search in a separate window (Open in Search) it works.

Does anyone know what would cause this?

Thanks,

SPL Below>>>
index=testsearch sourcetype=transaction (application="SharePoint Static" OR application="SharePointStatic" OR application="SharePointUpload" OR application="SPTeamSiteMySite*") transaction="*UploadTestFile3MB" transaction_status="Success"

|eval secs=round(duration/1000)
|lookup local=1 testsearch_network_mapping_file.csv hostname OUTPUT longitude latitude UserCount DisplayName
|search (UserCount > 0 AND UserCount < 100000000) DisplayName != "Frank - Campus (" |rangemap field=secs Good=0-3 Fair=3-4 Bad=4-2000

|stats count AS TransTot max(UserCount) AS Users max(longitude) AS long max(latitude) AS lat BY DisplayName range
|join DisplayName [search index=testsearch sourcetype=transaction (application="SharePoint Static" OR applicaton="SharePointStatic" OR application="SharePointUpload" OR application="SPTeamSiteMySite") transaction="*UploadTestFile3MB"
|lookup testsearch_network_mapping_file.csv hostname OUTPUT DisplayName
|stats count AS SiteTotal by DisplayName]
|eval ChartValue=TransTot*Users/SiteTotal
|table *
|geostats latfield=lat longfield=long sum(ChartValue) by range
|eval s=Good+Bad+Fair
|eval Good%=round((Good / s) * 100,2)
|eval Bad%= round((Bad / s ) *100,2)
|eval Fair%=round((Fair / s) * 100,2)
| fields - s

Re: Why does this search fail when the dashboard panel loads, but runs successfully when executed manually.

MuS — Tue, 21 Nov 2017 22:03:59 GMT

So what is the error message you get from the indexers? Can you please check the job inspector and post the error as well, please ?

Re: Why does this search fail when the dashboard panel loads, but runs successfully when executed manually.

abeeber_2 — Tue, 29 Sep 2020 16:51:59 GMT

MuS,

Here are the indexer related errors..

The indexers are windows 2008R2 (to be migrated to RHEL very soon).

error : [subsearch]: [PRODASRV235] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV274] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV354] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV411] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV423] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV630] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV237] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV238] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV239] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV321] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV374] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV380] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV381] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV630] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [KULAPP235] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [KULAPP236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [LHDAPP235] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [LHDAPP236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
(SID: YXJiZWViZUBuYS54b20uY29t_YXJiZWViZUBuYS54b20uY29t_MTAwX1hPTV9FMkVfc2hhcmVwb2ludA_search1_1511303688.4262_588B113C-A2D3-42E1-ACE2-8E5D47E3C0DE) search.log

Re: Why does this search fail when the dashboard panel loads, but runs successfully when executed manually.

abeeber_2 — Tue, 21 Nov 2017 22:39:17 GMT

Would the length of the sid be a problem? I noticed that when I run the search manually, the sid is much shorter.

Re: Why does this search fail when the dashboard panel loads, but runs successfully when executed manually.

hettervik — Wed, 22 Nov 2017 11:19:59 GMT

The search isn't by chance derived from a base search? If so, you might have to define which fields are going to be handed down by the base search to the sub-searches in the dashboard.

Re: Why does this search fail when the dashboard panel loads, but runs successfully when executed manually.

abeeber_2 — Wed, 22 Nov 2017 14:20:24 GMT

@hettervi,

good question. I am working through the syntax to figure that out. I'm thinking the join might be part of the problem.

Re: Why does this search fail when the dashboard panel loads, but runs successfully when executed manually.

abeeber_2 — Tue, 28 Nov 2017 16:33:12 GMT

Here is my fix/RCA.

Turns out the root cause was using Windows 2008R2 for the index tier.

Using SSO AD accounts that have FQDN meant the hashed value of the search sid exceeded the character limit of the server. This was identified by using the | history command to see the difference.

Ultimately, we're migrating the index tier to RHEL and as a workaround, I'm creating shared local accounts to facilitate access.

Re: Why does this search fail when the dashboard panel loads, but runs successfully when executed manually.

abeeber_2 — Wed, 29 Nov 2017 20:57:41 GMT

More intel.

Turns out there is a bug/fix in Splunk 6.4.5 where they shortened a temp file from 30 characters to 16.

We installed 6.4.9 on the index tier and the problem went away.