How and why did all events but one get deleted from an index?

Sebastian2 — Thu, 12 Nov 2015 14:14:23 GMT

Hey folks,

I'm new to Splunk and I am currently reading the "Big Data Analytics Using Splunk" Book published by apress. However, I imported and indexed some of the csv-flightdata from http://transtats.bts.gov/ to understand what's going on and for testing etc. The indexing process was successful and the events seemed to be correct (I imported one month of data). I tried to get a bigger search working (writing macros in the CLI and restarting Splunk afterwards) and never got any results - until I noticed that all events, but one where gone. The only event left is the CSV headline (with the field names); all other events (the actual flight data) somehow disappeared. I have installed S.o.S. on my test instance but couldn't find any suspicious stuff.

Do you have any idea why all the data is gone, and why is only the headline left in the index? Of course I could just re-index the data; but I really want to understand what happened.

Cheers

Re: How and why did all events but one get deleted from an index?

Sebastian2 — Mon, 16 Nov 2015 09:30:02 GMT

After some digging I found the (very simple) answer. That's something like a newbie mistake: The flightdata I imported was from 1987. A while after the indexing was successfull (I suppose) Splunk moved data from Hot to Frozen. Since I didn't have a frozen script the data was simply deleted. The event containing the CSV headers, how ever, got a timestamp from 2015 and wasn't deleted (of course).

To "fix" that I added:

frozenTimePeriodInSecs = 4294967294

to my indexes.conf in the appropriatestanza

topic Re: How and why did all events but one get deleted from an index? in Getting Data In

How and why did all events but one get deleted from an index?

Re: How and why did all events but one get deleted from an index?