Why does Splunk UI search hang or timeout with ResourceNotFound HTTP 404 error?

[gesman]

I've experienced situations where Splunk searches stopped in the middle without any signs of life or progress.
Job -> Inspect Job will show information something along these lines:

ResourceNotFound: [HTTP 404] https://127.0.0.1:8089/services/search/jobs/1413476626.459?message_level=debug; [{'code': None, 'text': 'Unknown sid.', 'type': 'FATAL'}]

[gesman · Accepted solution]

I found the issue is caused by Splunk's default timeout for UI-initiated searches.
To fix it, edit $SPLUNK_HOME/etc/system/local/web.conf by setting reasonably high value (in minutes) for ui_inactivity_timeout, such as:

[settings]
ui_inactivity_timeout = 180

And then restarting Splunk. Above will set timeout value to 3 hours.
See more info here: http://docs.splunk.com/Documentation/Splunk/6.1.4/admin/Webconf