Splunk Search

Search Bar - large queries fail to execute and cause "Disconnected from Server" error?

markconlin
Path Finder

When I attempt to enter very large queries into the search bar I get errors in chrome and eventually a "disconnected" from Splunk popup.

I do not believe the search is ever started as I do not see it in the "Jobs Manager".

Steps to reproduce:
- enter a large query into the search bar, mine is 160 lines and 10.4k characters.
- hit enter key
- errors appear, search is not started, soon "Disconnected from Splunk" popup will appear.

A split second after hitting enter... these errors appears in the chrome debugger:

DELETE https://splunk.myhost.com/en-US/splunkd/__raw/services/search/jobs/rt_md_1516156026.353?output_mode=...
(failed) net::ERR_CONNECTION_CLOSED

GET https://splunk.myhost.com/en-US/splunkd/__raw/servicesNS/admin/search/saved/searches/_new?output_mod...
(failed) net::ERR_CONNECTION_CLOSED

GET https://splunk.myhost.com/en-US/splunkd/__raw/services/saved/searches/_new?output_mode=json&_=151615...
(failed) net::ERR_CONNECTION_CLOSED

POST https://splunk.myhost.com/en-US/splunkd/__raw/servicesNS/admin/search/search/ast
(failed) net::ERR_CONNECTION_CLOSED

and so on....

Errors in Chrome Debugger (this happens right away)

alt text

...and then eventually disconnected from Splunk

alt text

0 Karma

marend_umg
Observer

@markconlin Did you were able to fix the issue? I'm running over a similar issue when I click on a Dashboard the "Open in Search" glass option, I have a large query too so not quite sure if this is a limit that I need to set at limits.conf or web.conf?

0 Karma

mayurr98
Super Champion

It means the browser lost the ability to talk to SplunkWeb for more than some number of seconds.

More literally it means an HTTP request reported http status code 0, which is the browsers way of telling us that it never heard back. Im not sure how many seconds it takes of total silence before the browser gives up but I'm pretty sure it's not configurable.

Assuming someone didnt actually restart SplunkWeb out from under you, it sounds like your server or maybe just SplunkWeb, is extremely busy. I get this once in a while too.

There are some options that you can configure in web.conf. but i highly recommend you to check at which stage it is taking time to fetch results in the job inspector splunk and according you can configure any of these settings.

server.socket_timeout = <integer>
* The timeout in seconds for accepted connections between the browser and
  splunkweb
* Defaults to 10

response.timeout = <integer>
* Specifies the number of seconds to wait for the server to complete a
  response
* Some requests such as uploading large files can take a long time
* Defaults to 7200

job_min_polling_interval = <integer>
* Minimum polling interval for job in miliseconds (ms)
* The default value is 100
* This is the intial time wait for fetching results
* The poll period increases gradually from min interval to max interval when
  search is in queued state or parsing state (and not running state) for a
  some time.
* Set this value between 100 to job_max_polling_interval

job_max_polling_interval = <integer>
* Maximum polling interval for job in miliseconds (ms)
* The default value is 1000
* This is the maximum time wait for fetching results
* The recommended maximum value is 3000

splunkdConnectionTimeout = <integer>
* Number of seconds to wait before timing out when communicating with
  splunkd
* Must be at least 30
* Values smaller than 30 will be ignored, resulting in the use of the
  default value
* Defaults to 30

let me know if this helps!

0 Karma

markconlin
Path Finder

Let me add some more information to my question so that it is more clear. I can eliminate restart and Splunk being too busy, those are not the issue.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!