[UPDATE] Pivot searches erroring out with: Error in 'TSCollectProcessor'

[gauldridge]

I have a Splunk instance out on Amazon EC2 that I have used for demo purposes for a long time. It's just indexing the Apache logs and audit logs of the EC2 so I have data for demos.

I just upgraded to Splunk 6 from Splunk 5.0.3. I was having no issues with Splunk 5.0.3 but with version 6 I am not able demo the Pivot functionality (which is the main feature I have been wanting show during the past week) because it errors out immediately. The full error message I am given is:

Error in 'TSCollectProcessor': Failed to create TSIDX event in namespace='/opt/splunk/var/run/splunk/dispatch/1380974044.171/tsidxstats' errcode=1

I get this error message for Data Models I have created as well as the two sample Data Models provided with the install. I get no errors using the regular search functions.

I should mention that I upgraded from version 5.0.3 to version 6 on physical hardware and have no issues whatsoever. It's only the EC2 demo instance that I'm having issues with post-upgrade.

Any ideas out there on how I can get my demo instance of Splunk up and running?

UPDATE: The DEBUG info in the job inspector is: DEBUG: search context: user="marty", app="search", bs-pathname="/opt/splunk/etc"

2nd UPDATE: The Pivot will work properly only if the Data Model includes one additional Attribute. For example, it will work properly if I add clientip but it will break if I add another field.

[pj]

Check the memory usage - the smallest micro AMI that Amazon provides doesn't have enough RAM to allow the Pivot interface to function correctly.

[Marklar]

Any other errors in the search.log for the dispatch directory of the tscollect job? That would be helpful in debugging this.

I can't imagine how changing the amount of attributes would affect this specific code path to yield this errcode, so we need a bit more information. Is it breaking immediately in pivot or after it's been running for a bit? If it's the latter, I'd suspect disk space issues.

[gauldridge]

The pivot breaks immediately...well after 0.42 seconds. Here's the event from the _audit index: Audit:[timestamp=10-28-2013 01:49:43.876, user=n/a, action=search, info=failed, search_id='1382924973.2275', total_run_time=0.42, event_count=1555, result_count=0, available_count=0, scan_count=1555, drop_count=0, exec_time=1382924973, api_et=N/A, api_lt=N/A, search_et=N/A, search_lt=N/A, is_realtime=0, savedsearch_name=""][n/a]

[jspears]

Are you relatively low on disk space? See the first item here:

We have increased the default amount of required available disk space for indexing and searching

Prior to version 6.0, the default amount of free space Splunk needed to index and search was 2 gigabytes. When you upgrade, Splunk raises this default requirement to 5 gigabytes. Before you upgrade, make sure you have enough free space on the volume(s) that contain Splunk indexes and search dispatch directories to ensure uninterrupted index and search operation

[gauldridge]

I had not considered the minimum disk space requirement. However, after checking the available disk space on my EC2, I have a little over 6GB free space. The logs being indexed by this instance are very low volume and shouldn't consume the 1GB+ buffer for a good while.