Data Gets Deleted

[Opher]

Hi,

Not a pro, but I've configured a Splunk Enterprise on my non-profit's Azure server.

I'm conducting an educational course, and I've uploaded some zeek logs from CIC-IDS2017 dataset.

For some reason, about 60 minutes after uploading using oneshot the entire index gets deleted:

/opt/splunk/bin/splunk add oneshot "$f" -index "$INDEX" -sourcetype "$st" -host "$HOST"

I really can't understand what I'm doing wrong, since the installation was really vanilla.

Can anyone help me?

[livehybrid]

Hi @Opher 

Can you confirm the time period you are searching for the data across when you are searching? 

Are there any other indexes with data? 

When you are one-shotting it will import the data at a point in time but will not continue to monitor those files (if they are changing) but the data should always exist if you search for the time period when you imported it, unless something else is configured incorrectly. 

Have you made any changes to the default retention (frozenTimePeriodInSecs) for the index?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PickleRick · Accepted solution]

What do you mean by "index gets deleted". Does the whole index indeed disappear (which is very very unlikely unless it's been explicitly deleted) or does simply data from it get rolled out (which can happen if the data is old enough and the retention policy warrants it).

Check the _internal log for phrase "freeze succeeded".

[Opher]

I'll mention that the data disappeared from the index.

Since the data I've uploaded was from 2017, I'm guessing it just got frozen.

Though I didn't see a data freeze from the day I uploaded (which was yesterday)

And when I changed my retention policy I noticed it came back.

Weird, maybe I understood something wrong.

Anyhow - seems to work now 🙂

[PickleRick]

Data isn't getting rolled to frozen immediately. There's a bit of fancy processing going on underneath. (search for "splunk bucket life cycle" for more info).

So it's perfectly normal that your data would disappear some time after initial ingestion.

But I don't understand what you mean by "data came back". It shouldn't. As it gets deleted because it's rolled to frozen it's not available in Splunk anymore.