How to fix 500 internal server error on Splunk 5.0.1

[megancarney]

When I do Home->Add Data->From Files and Directories->Skip Preview or Home->Add Data->From Files and Directories->Preview Data Before Indexing I get a '500 internal server error'.

If I choose 'Preview Data Before Indexing' it happens after I've chosen the sourcetype and previewed the data, if I choose 'Skip Preview' it happens right away.

This is the line in the logs:

admin:1120 - uiHelper processValueAdd operator failed for endpoint_path=data/inputs/monitor/_new elementName=spl-ctrl_sourcetypeSelect: list index out of range

The problem first appeared in 5.0, I upgraded to 5.0.1 and I'm still seeing it. This is a Windows XP machine.

I found two hits on similar issues:
http://splunk-base.splunk.com/answers/66172/consistent-error-500s-on-file-and-directory-new-definiti...
http://splunk-base.splunk.com/answers/7854/splunk-failed-to-fetch-form-values-for-form-savedsearches

The first hit isn't clear about what was done to fix the problem and the solution in the second hit was just to reinstall. I'm hoping to avoid that.

Does anyone have any additional information?

[hexx · Accepted solution]

This problem is caused by a regression in Splunk 5.0 and 5.0.1 which impacts certain methods to list indexes, such as:

• The eventcount command
• REST API calls to the /services/data/indexes endpoint

In 5.0 and 5.0.1, these methods fail to use cached assets and end up iterating over the files of your indexes themselves. This results in a much slower execution, which eventually affects whatever consumer calls the method.

In the Manager user interface, this affects some of the pages that need to present a list of indexes such as:

• The page showing the details of a saved search (Manager > Searches & Reports > $SAVED_SEARCH_NAME)
• The setup page for most data inputs (Manager > Data Inputs > Files & Directories > Add new)
• The page where you can edit roles (Manager > Access controls > Roles)

If the index-listing method takes more than 30s to complete, Splunkweb will time out on it and display a "500 - internal server error" message.

This issue will be fixed in maintenance release 5.0.2, which we are aiming to deliver early in 2013. We are also working on intermediary patch release 5.0.1.1 to specifically address this issue.

For reference, the bugs behind this problem are SPL-57518 and SPL-58650.

IMPORTANT NOTE: We are aware of a residual issue in version 5.0.2 that is responsible for timeouts on a smaller set of Manager pages:

• Manager » Indexes
• Manager » Data inputs » * » Add new

This issue will primarily affect systems with many (typically, thousands) of buckets in the hot/warm index directories and running on Windows (which does not passively use available system memory for filesystem cache).

The bug reference for this issue is SPL-61718 and it is slated to be fixed in 5.0.3. We are also working on the release of a 5.0.2.x patch to address this issue, which you'll be able to obtain by contacting support.

[hexx]

This is due to the nature of this issue, which has to do with gathering the list of indexes present on indexers from the search-head. The indexers need to have the fix as well so that when they run the 'eventcount' command to list indexes, that doesn't take too long and cause a time out upstream.

[dabbank]

I only updated one of our search heads to 5.0.2 and expected the issue to be fixed since you maintain saved searches there. But here I was wrong. You have to update the indexers to benefit from the fix. This is quite strange because you edit saved searches stored on the search head; but that's just how it is.

[hexx]

There is, unfortunately, a remaining issue in version 5.0.2 that affects certain Manager pages. I have amended my answer to indicate that.

[dabbank]

I'm experiencing the issue whenever I try to modify saved searches and tried the now available update to 5.0.2. And I'm sorry to report: The problem is not fixed with 5.0.2!

[ChrisG]

...and this item is listed in the Known Issues.

[amrit]

The "list index out of range" appears to be a red herring, as I also see it on fresh 5.0.1 install with no observable errors in the UI. I've filed it as SPL-58736, referencing this question.

@megancarney: Please open a support case ( https://www.splunk.com/page/submit_issue ) and upload the diagnostic file generated by "splunk diag". A screenshot of the error in the web browser would also help.

[MHibbin]

@megancarney, changed the "5.1" to "5.0.1", as I assume this is what you meant.

If it's not what you meant I apologise as you must be time-travelling as Splunk v5.1 is not a current release. 🙂