Deployment Architecture

recover-padding-# from *nix app

Communicator

In using the *nix app, I had cpu.sh/iostat.sh/vmstat.sh and all was working fine across 300 rhel4/5 boxes.

I added in ps.sh & top.sh and now I have 219 recover-padding=# host entries. recover-padding-[1-5] as sources & sourcetypes.

Why did this happen and how can I make them go away?

I noticed this question but I couldn't find a Hosts.metadata file. http://answers.splunk.com/questions/10032/wierd-hosts-recover-padding-listed

1 Solution

Communicator

After I got everything cleaned up from above and my *nix app deployment. I was able to get rid of the recover-padding entries after working with support.

I had to stop Splunk and delete *.data, .*manifest file(s) from /opt/splunk/var/lib/splunk/os/db Then start Splunk..

I didn't have any .*manifest files but that cleared them up for me.

Also note it is NOT *.manifest.. It is .*manifest. I validated that by looking in some of my other db dirs.

After the restart it rebuilt the *.data files.

They said if the above didn't work then I should "run recover-metadata on all of your os index and identify the bad index" But only if the above didn't work.

View solution in original post

0 Karma

Communicator

After I got everything cleaned up from above and my *nix app deployment. I was able to get rid of the recover-padding entries after working with support.

I had to stop Splunk and delete *.data, .*manifest file(s) from /opt/splunk/var/lib/splunk/os/db Then start Splunk..

I didn't have any .*manifest files but that cleared them up for me.

Also note it is NOT *.manifest.. It is .*manifest. I validated that by looking in some of my other db dirs.

After the restart it rebuilt the *.data files.

They said if the above didn't work then I should "run recover-metadata on all of your os index and identify the bad index" But only if the above didn't work.

View solution in original post

0 Karma

Communicator

I found the recover-padding entries in here:
/opt/splunk/var/lib/splunk/os/db/Hosts.data
Can i just delete them and restart?

0 Karma

Communicator

I think it might have happened because when I added ps.sh & top.sh to /opt/splunk/etc/deployment-apps/unix/local/inputs.conf I forgot to add them to /opt/splunk/etc/deployment-apps/unix/default/inputs.conf
I had removed the entries I didn't want at first. Anyways I put them back and deployed, but the recover-padding entries still didn't go away. Now i'm back to no ps.sh & top.sh but those darn recover-padding entries are all over even after a restart.

0 Karma

Splunk Employee
Splunk Employee

You shouldn't need to make them go away, they're intended to be behavior that's essentially internal to the .data files in your system. They get added for cases where you experienced a problem that required recovery, ie crashes, hardware failings, or operating system failings. They're part of the recovery process for handling incompletely written out data.

This problem should be fixed in Splunk 4.1.2, as I read our bug database. What version are you using?

Splunk Employee
Splunk Employee

Most likely you are using a modified dashboard which doesnt contain the workaround to hide the padding entries, but the issue seems resolved in any event.

Communicator

i upgraded the indexer to 4.1.7. didn't make the recover-padding go away

0 Karma

Communicator

Will they go away after they do their recovery thing? They are most annoying visually..

0 Karma

Communicator

4.1.6 for both forwarder and indexer

0 Karma