Getting Data In

How and why did all events but one get deleted from an index?

Sebastian2
Path Finder

Hey folks,

I'm new to Splunk and I am currently reading the "Big Data Analytics Using Splunk" Book published by apress. However, I imported and indexed some of the csv-flightdata from http://transtats.bts.gov/ to understand what's going on and for testing etc. The indexing process was successful and the events seemed to be correct (I imported one month of data). I tried to get a bigger search working (writing macros in the CLI and restarting Splunk afterwards) and never got any results - until I noticed that all events, but one where gone. The only event left is the CSV headline (with the field names); all other events (the actual flight data) somehow disappeared. I have installed S.o.S. on my test instance but couldn't find any suspicious stuff.

Do you have any idea why all the data is gone, and why is only the headline left in the index? Of course I could just re-index the data; but I really want to understand what happened.

Cheers

Tags (3)
0 Karma
1 Solution

Sebastian2
Path Finder

After some digging I found the (very simple) answer. That's something like a newbie mistake: The flightdata I imported was from 1987. A while after the indexing was successfull (I suppose) Splunk moved data from Hot to Frozen. Since I didn't have a frozen script the data was simply deleted. The event containing the CSV headers, how ever, got a timestamp from 2015 and wasn't deleted (of course).

To "fix" that I added:

frozenTimePeriodInSecs = 4294967294

to my indexes.conf in the appropriatestanza

View solution in original post

Sebastian2
Path Finder

After some digging I found the (very simple) answer. That's something like a newbie mistake: The flightdata I imported was from 1987. A while after the indexing was successfull (I suppose) Splunk moved data from Hot to Frozen. Since I didn't have a frozen script the data was simply deleted. The event containing the CSV headers, how ever, got a timestamp from 2015 and wasn't deleted (of course).

To "fix" that I added:

frozenTimePeriodInSecs = 4294967294

to my indexes.conf in the appropriatestanza

Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...