Splunkd 404 Error PostProcessing

[spongmob]

All,

Over the past couple of days, I've been getting an unusual error associated with post-processing and visualization. This error rarely occurs initially, but only after a few minutes when I decide to use a pager or a drilldown.

The error message is below.

"Splunkd returned a 404 error unexpectedly. Since there is a postprocess search here, this 404 is almost certainly caused by a syntax error in the postprocess search."

[sideview · Accepted solution]

Well, it most likely is caused by some search syntax error that for whatever reason only occurs sporadically.

When there's a search syntax error on a postprocess request, unfortunately Splunk returns 404.
Since an actual 404 isn't terribly common (for instance if your search results suddenly vanished or if someone deleted your job ), it's almost certainly one of these "syntax error" 404's, which are in my experience quite common.

You can often fish out the postprocess search syntax and troubleshoot and often the nature of the error comes to light. For instance if the postprocess search makes an assumption that some field will always exist, it might generate an error when that field turns out to be absent. Manually glue the two pieces back together and experiment in the search page. I bet there's some lookup syntax that generally works and sometimes generates a "not all fields in lookup could be found" error or something like that.

[sideview]

Unfortunately that's exactly what it looks like when the sid is perfectly fine, and there's just a syntax error in the postprocess search. So I would focus on the postprocess search that is being sent. In the Table module itself there should be a little 'show details' link, or at least some way right there to see the postprocess search that was sent.

[spongmob]

Thanks for the answer. I checked in the logs and it looks like the 404 error is being caused by the sid disappearing. For example, I was able to pull this "[{'code': None, 'text': 'Unknown sid.', 'type': 'FATAL'}]" error in conjunction with these events. Do you have any idea what could cause the SID's to disappear like this?

From initial research it looks like it could be disparities within the time of nfs, which may accelerate the ttl's of these searches.