Where did the test data go in the SPL tutorial?

[SplunkAdmin69]

In going through the SplunkCloud SPL tutorial, we are told to upload California drought data into Splunk, and we create a Dashboard from it.  That worked just as explained, but the next day the data is gone.  I was using the "AllTime" filter, so it was not that I missed data that was getting older, and skipped by filters.

source="us_drought_monitor.csv" State = CA date_year=2018| rex field=County "(?<County>.+) County"| eval droughtscore = D1 + D2*2 + D3*3 + D4*4| stats avg(droughtscore) as "2018 Drought Score" by County| geom ca_county_lookup featureIdField=County

above is the search SPL for the demo. 

While I have your attention, in the tutorial they add a min and max function, ""2018 Drought Score" max(droughtscore) as "Max 2018 Drought Score" min(droughtscore) as "Min 2018 Drought Score" by County"  This provided SPL code broke the Dashboard yesterday when it was working. is there something wrong with this SPL, that was provided?

[SplunkAdmin69]

@danielcj, if you log on to SplunkCloud you can't miss the SPL tutorial.  It was the first thing presented to me.

In SplunkCloud go to the upper right corner, and click on "Help with this page" while you are in Search.

[PickleRick · Accepted solution]

Since the data seems to be from way back it is possible that the index you're ingesting the samples to has retention period shorter than the data age.

In that case the data wouldh initially get ingested and indexed properly but when the bucket gets closed and moved from hot to cold splunk's maintenance thread would noticd that the youngest event in the bucket was older thsn retention period and would freeze (effectively deleting if you don't have any other freezing method configured) the bucket with sample data.

[SplunkAdmin69]

@PickleRick sounds correct to me.  What is this "freezing" you speak of, sounds like not deleted, but hidden?  What is the point of hiding data that is still retained?  I don't necessarily expect an answer to the second question.  Also I will search for freezing.

Thanks for answering my question.

[PickleRick]

In Splunk data lifecycle there are several stages the bucket can be in.

First the bucket is hot. Splunk is writing new data to it.

Then after the bucket meets some parameters (size, age), the "housekeeping" thread closes the bucket and moves it to another directory on the same storage - it's called warm bucket.

After the bucket is old enough, if you have separate storage defined for it, the bucked is moved to the cold storage - it's a cold bucket.

And finally after the bucket reaches the retention period or the index gets too big, the bucket is frozen - if you have special storage defined for it, the bucket is moved there and removed from splunk (it's meant for archiving and not for direct use anymore). But if you don't have the frozen storage defined (which is the default config), the bucket simply gets deleted.

[danielcj]

Hello @SplunkAdmin69 ,

Sorry, I am not familiar with this tutorial, is this a public Splunk tutorial? If yes, could you please post the link of this tutorial?

In the first case, if the data is not being shown now you would need to verify: 1) if the role of the user that is searching the data have permissions to search this index. 2) The retention size/time of the index (if the retention is exceeded, this data will be deleted).

For the second case, could you please share the complete query?

Thanks.