Deployment Architecture

recover-padding-# from *nix app

chicodeme
Communicator

In using the *nix app, I had cpu.sh/iostat.sh/vmstat.sh and all was working fine across 300 rhel4/5 boxes.

I added in ps.sh & top.sh and now I have 219 recover-padding=# host entries. recover-padding-[1-5] as sources & sourcetypes.

Why did this happen and how can I make them go away?

I noticed this question but I couldn't find a Hosts.metadata file. http://answers.splunk.com/questions/10032/wierd-hosts-recover-padding-listed

1 Solution

chicodeme
Communicator

After I got everything cleaned up from above and my *nix app deployment. I was able to get rid of the recover-padding entries after working with support.

I had to stop Splunk and delete *.data, .*manifest file(s) from /opt/splunk/var/lib/splunk/os/db Then start Splunk..

I didn't have any .*manifest files but that cleared them up for me.

Also note it is NOT *.manifest.. It is .*manifest. I validated that by looking in some of my other db dirs.

After the restart it rebuilt the *.data files.

They said if the above didn't work then I should "run recover-metadata on all of your os index and identify the bad index" But only if the above didn't work.

View solution in original post

0 Karma

chicodeme
Communicator

After I got everything cleaned up from above and my *nix app deployment. I was able to get rid of the recover-padding entries after working with support.

I had to stop Splunk and delete *.data, .*manifest file(s) from /opt/splunk/var/lib/splunk/os/db Then start Splunk..

I didn't have any .*manifest files but that cleared them up for me.

Also note it is NOT *.manifest.. It is .*manifest. I validated that by looking in some of my other db dirs.

After the restart it rebuilt the *.data files.

They said if the above didn't work then I should "run recover-metadata on all of your os index and identify the bad index" But only if the above didn't work.

View solution in original post

0 Karma

chicodeme
Communicator

I found the recover-padding entries in here:
/opt/splunk/var/lib/splunk/os/db/Hosts.data
Can i just delete them and restart?

0 Karma

chicodeme
Communicator

I think it might have happened because when I added ps.sh & top.sh to /opt/splunk/etc/deployment-apps/unix/local/inputs.conf I forgot to add them to /opt/splunk/etc/deployment-apps/unix/default/inputs.conf
I had removed the entries I didn't want at first. Anyways I put them back and deployed, but the recover-padding entries still didn't go away. Now i'm back to no ps.sh & top.sh but those darn recover-padding entries are all over even after a restart.

0 Karma

jrodman
Splunk Employee
Splunk Employee

You shouldn't need to make them go away, they're intended to be behavior that's essentially internal to the .data files in your system. They get added for cases where you experienced a problem that required recovery, ie crashes, hardware failings, or operating system failings. They're part of the recovery process for handling incompletely written out data.

This problem should be fixed in Splunk 4.1.2, as I read our bug database. What version are you using?

jrodman
Splunk Employee
Splunk Employee

Most likely you are using a modified dashboard which doesnt contain the workaround to hide the padding entries, but the issue seems resolved in any event.

chicodeme
Communicator

i upgraded the indexer to 4.1.7. didn't make the recover-padding go away

0 Karma

chicodeme
Communicator

Will they go away after they do their recovery thing? They are most annoying visually..

0 Karma

chicodeme
Communicator

4.1.6 for both forwarder and indexer

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!