Getting Data In

How and why did all events but one get deleted from an index?

Sebastian2
Path Finder

Hey folks,

I'm new to Splunk and I am currently reading the "Big Data Analytics Using Splunk" Book published by apress. However, I imported and indexed some of the csv-flightdata from http://transtats.bts.gov/ to understand what's going on and for testing etc. The indexing process was successful and the events seemed to be correct (I imported one month of data). I tried to get a bigger search working (writing macros in the CLI and restarting Splunk afterwards) and never got any results - until I noticed that all events, but one where gone. The only event left is the CSV headline (with the field names); all other events (the actual flight data) somehow disappeared. I have installed S.o.S. on my test instance but couldn't find any suspicious stuff.

Do you have any idea why all the data is gone, and why is only the headline left in the index? Of course I could just re-index the data; but I really want to understand what happened.

Cheers

Tags (3)
0 Karma
1 Solution

Sebastian2
Path Finder

After some digging I found the (very simple) answer. That's something like a newbie mistake: The flightdata I imported was from 1987. A while after the indexing was successfull (I suppose) Splunk moved data from Hot to Frozen. Since I didn't have a frozen script the data was simply deleted. The event containing the CSV headers, how ever, got a timestamp from 2015 and wasn't deleted (of course).

To "fix" that I added:

frozenTimePeriodInSecs = 4294967294

to my indexes.conf in the appropriatestanza

View solution in original post

Sebastian2
Path Finder

After some digging I found the (very simple) answer. That's something like a newbie mistake: The flightdata I imported was from 1987. A while after the indexing was successfull (I suppose) Splunk moved data from Hot to Frozen. Since I didn't have a frozen script the data was simply deleted. The event containing the CSV headers, how ever, got a timestamp from 2015 and wasn't deleted (of course).

To "fix" that I added:

frozenTimePeriodInSecs = 4294967294

to my indexes.conf in the appropriatestanza

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...