Why does Splunk custom endpoint time out automatically after 500 seconds

[keerthana_k]

Hi,

We have distributed Splunk deployment running version 7.3.0. We have a custom REST endpoint which runs some searches and returns the search results in JSON format. When we run searches for long time ranges, the request times out after 500 seconds with 502 error code even when we see the search running in the Job Manager page. We found this error in the splunkd.log:

ERROR HttpClientRequest - HTTP client error=Read Timeout while accessing server=http://127.0.0.1:8065 for request=http://127.0.0.1:8065

We tried adjusting various timeout values like tools.session.timeout, ui_inactivity_timeout etc. but none of them seemed to help.
Has anyone faced similar issues. How do we fix this to make the request wait till the search completes.

Thanks in advance,
Keerthana