Empty index after deployment of application

[bgaignon]

Hi guys,

Here is quickly the situation:

• We have qualys_app on the Search Head with our dashboard.
• We have qualys_inputs on the Indexer with inputs.conf monitoring /tmp/qualys folder (we upload manually our files directly on the indexer)
• We have all_indexes application with indexes.conf with our config for our index (qualys).
• All apps are pushed by our deployment-server.

When we upload our new .csv files by putting them into our folder /tmp/qualys, they are well indexed by the Indexer and our Dashboard show us what we want.

Our issue is: when on the deployment-server we run ./splunk reload deploy-server
After the reboot of the Indexer the index qualys is empty. So we loose our data previously indexed.

My tests:

• I suppose the problem can only comes from the Indexer and the apps deployed on the Indexer.
• I tried to see if there is another indexes.conf in the other deployed apps that might override the configuration.
• There is no errors in splunkd.log
• I re-import my data, so they have been re-indexed and then after a new deployment => Empty index again.

So I'm open to any suggestion.
Thanks.

[bgaignon · Accepted solution]

OK that was really stupid:
The parameter in indexes.conf: maxVolumeDataSizeMB for the entire volume was as big as the size of 1 index.

So that triggered the rolling data to frozen.
Now fixed.

[bgaignon]

Thanks Greg,

The problem happened recently, I just capture an interesting log and based on the result I think the problem comes from:
[volume:primary]
path = /index
maxVolumeDataSizeMB = 5000

maxVolumeDataSizeMB = 5000
I put the size of my volume and reload the app.
I 'll gave a feedback if it's solve the problem.

[gregbujak]

Can you clarify the "index is empty"? Exactly what command are you executing to determine the index is empty?

Once its indexed, unless the index on disk is being blown away somehow, you should not lose any data. However, if you are using a lookuptable, thats another story.