Installation

post 4.3.2 upgrade, the rawdata version in directory does not support rebuilding and rawsize issue still exists when forcing rebuild

FatDragon1
Explorer

After upgrade, I am seeing lots of our indexes have processtracker errors.

i.e.
06-23-2012 14:28:10.391 -0400 ERROR ProcessTracker - (child_13__Fsck) BucketBuilder - The rawdata version in directory "/mnt/splunk-data/gw/db/db_1298332798_1297901011_0/rawdata" does not support rebuilding

and

Unable to read raw size

file="/mnt/splunk-data/icore/db/db_1326927068_1299178043_28/.rawSize": No such file or directory

I found a link ( http://splunk-base.splunk.com/answers/47462/post-43-upgrade-unable-to-read-raw-size ) for rawsize issue and it didn't help!

Tags (1)

FatDragon1
Explorer

I upgraded from 4.1.6 to 4.3.2.

Yes, Pretty much I have been all over the splunk base posts and splunk docs/wiki.

I installed splunk on splunk (sos) and reviewed most of the structures of the splunk operation.

disabled SplunkDeploymentMonitor 4.3.2 until I find time later to check on it's behavior.
disabled SplunkDeploymentMonitor_4.1.x 1.0 .

Monitored the splunkd log and other splunk logs and restarted splunk many times after every tuning changes.

some of the changs:
I increased FD to 100,000 for soft and hard for user executing splunkd and also tweaked the limits.conf in splunk sub-directories for processes per cpu and percentage of searches and then cleaned up dispatch directory of 119 stale saved searches and reports/schedules and then few other smaller tweaks here and there..

I had lots of patience. I don't like to see errors /warnings in any system/application and etc logs. INFO/notices entries are okay.

Seems like all my splunk logs are now clean of errors/warnings. We'll monitor more during busy splunk usage and update here with more information.

0 Karma

FatDragon1
Explorer

More information also on troubleshooting your buckets at bottom of the page in http://docs.splunk.com/Documentation/Splunk/4.3.2/admin/HowSplunkstoresindexes

FatDragon1
Explorer

oh yea..I also attempted to clean several of the buckets for the manifest/rawsize errors.

I think when splunk restarted it checks all the buckets and savedsearches and schedule reports and dispatches leftover and you'll get lots of errors and warnings as it tries to clean itself off and catch up with cold/hot/frozen buckets and update the rawdata and manifested rawsize and etc.

0 Karma

Drainy
Champion

Also, have you seen, http://splunk-base.splunk.com/answers/46839/splunkd-down-why to see if that perhaps is linked? (system running out of file descriptors).

Actually, another worthy question is what version did you upgrade from and to? If it was 4.1.x to 4.3.x then it is recommended to do an intermediary upgrade from 4.1.x to 4.2.x and then on to 4.3.x.

EDIT: Oh and of course, if you have a support contract it is always worth getting in touch with them for a quick resolution as they may have come across this many times before. Always best to be safe rather than sorry if it is critical data.

Drainy
Champion

Right, if memory serves me right you may need to do a 4.2 upgrade in the middle. So get back to a 4.1 install and then upgrade to 4.2 and from there upgrade to 4.3. I appreciate that the release notes suggest it should be fine to upgrade directly, there have been cases where support recommend a stepped upgrade

0 Karma

erga00
Path Finder

I have the same issue. Following the instructions in ( http://splunk-base.splunk.com/answers/47462/post-43-upgrade-unable-to-read-raw-size ) didn't help me either.

I'm on Windows 2008 R2 and upgraded from 4.1.6 to 4.3.2 (x64).

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...