After upgrade, I am seeing lots of our indexes have processtracker errors.
06-23-2012 14:28:10.391 -0400 ERROR ProcessTracker - (child_13__Fsck) BucketBuilder - The rawdata version in directory "/mnt/splunk-data/gw/db/db_1298332798_1297901011_0/rawdata" does not support rebuilding
Unable to read raw size
file="/mnt/splunk-data/icore/db/db_1326927068_1299178043_28/.rawSize": No such file or directory
I found a link ( http://splunk-base.splunk.com/answers/47462/post-43-upgrade-unable-to-read-raw-size ) for rawsize issue and it didn't help!
I upgraded from 4.1.6 to 4.3.2.
Yes, Pretty much I have been all over the splunk base posts and splunk docs/wiki.
I installed splunk on splunk (sos) and reviewed most of the structures of the splunk operation.
disabled SplunkDeploymentMonitor 4.3.2 until I find time later to check on it's behavior.
disabled SplunkDeploymentMonitor_4.1.x 1.0 .
Monitored the splunkd log and other splunk logs and restarted splunk many times after every tuning changes.
some of the changs:
I increased FD to 100,000 for soft and hard for user executing splunkd and also tweaked the limits.conf in splunk sub-directories for processes per cpu and percentage of searches and then cleaned up dispatch directory of 119 stale saved searches and reports/schedules and then few other smaller tweaks here and there..
I had lots of patience. I don't like to see errors /warnings in any system/application and etc logs. INFO/notices entries are okay.
Seems like all my splunk logs are now clean of errors/warnings. We'll monitor more during busy splunk usage and update here with more information.
More information also on troubleshooting your buckets at bottom of the page in http://docs.splunk.com/Documentation/Splunk/4.3.2/admin/HowSplunkstoresindexes
oh yea..I also attempted to clean several of the buckets for the manifest/rawsize errors.
I think when splunk restarted it checks all the buckets and savedsearches and schedule reports and dispatches leftover and you'll get lots of errors and warnings as it tries to clean itself off and catch up with cold/hot/frozen buckets and update the rawdata and manifested rawsize and etc.
Also, have you seen, http://splunk-base.splunk.com/answers/46839/splunkd-down-why to see if that perhaps is linked? (system running out of file descriptors).
Actually, another worthy question is what version did you upgrade from and to? If it was 4.1.x to 4.3.x then it is recommended to do an intermediary upgrade from 4.1.x to 4.2.x and then on to 4.3.x.
EDIT: Oh and of course, if you have a support contract it is always worth getting in touch with them for a quick resolution as they may have come across this many times before. Always best to be safe rather than sorry if it is critical data.
Right, if memory serves me right you may need to do a 4.2 upgrade in the middle. So get back to a 4.1 install and then upgrade to 4.2 and from there upgrade to 4.3. I appreciate that the release notes suggest it should be fine to upgrade directly, there have been cases where support recommend a stepped upgrade
I have the same issue. Following the instructions in ( http://splunk-base.splunk.com/answers/47462/post-43-upgrade-unable-to-read-raw-size ) didn't help me either.
I'm on Windows 2008 R2 and upgraded from 4.1.6 to 4.3.2 (x64).