Missing logs from splunk?

[sarwshai]

Specifically the winEventlog:security have vanished from my search results for approximately two three months, but currently all the logs are being indexed and also are searchable and retention period is set to 6 months.
As per retention period i should be getting logs for last 6 months, but it isn't same case. For e.g from August the logs should be saved till Feb , but i have logs for last 3 months but not for prior 3 months(approximately also there are logs for some day and for other days its count is zero), What might be the problem here.

[Richfez]

In addition to settings for retention by time, there are also settings that specify maximum size of data. Since you seem to have confirmed the length of times involved, I'd say that's the most likely remaining option.

You don't even need to edit a file on the file system. Probably. 🙂

Just click Settings, then Indexes. In there find the index you are having trouble with and compare the Current Size column with the Max Size column. I expect you'll find that it's filled up, so Splunk is deleting older data.

To fix, if that's the case, just edit the index and make it bigger. Obviously, only after confirming you have the extra disk space available!

[sarwshai]

If that may be the case then splunk would have deleted all logs during entire month, but it isn't the case, we have logs for some days lets say 15 of that month but prior and later to that date the count of logs is zero.
But sure i'll try up your method and would answer you back, thanks.

[Richfez]

It's sounding more and more like you don't have a simple "time or space" issue. But, that screenshot is suspcious. Even a DAY'S worth of one system's winevents is likely bigger than 1 MB. So I agree with DalJeanis something else is going on here.

We'll need to see the configuration of those indexes, could you do this for us?

From an elevated command prompt, assuming your splunk is installed in the default location, do

cd \program files\splunk\bin
splunk indexes list wineventlog --debug

Then paste that output back into here? Here's a link on using btool to troubleshoot but hopefully all you need is what I've written.

[s2_splunk]

I think you meant splunk cmd btool indexes list wineventlog --debug 🙂

[DalJeanis]

@sarwshai - upvoted you for that thought, then I realized that would only be true if you had only a single indexer. If you have multiple indexers, then it would be possible for one of them to run out of space and require something to be rolled off, while others had not.

To test that idea, do a timechart by splunkserver and see if there's one that starts having data much later than the others.

[sarwshai]

https://drive.google.com/open?id=0B_nR3_Mk2Sh0VGc3YXZTZlVvY2s , the link foe snapshot of current vs maximum data size, the current size is negligible compared to maximum data size. Can we have other solution please?

[s2_splunk]

If you share your indexes.conf for the index in question, it will be easier to help you, I think.