Splunk Search

Where did the test data go in the SPL tutorial?

SplunkAdmin69
Engager

In going through the SplunkCloud SPL tutorial, we are told to upload California drought data into Splunk, and we create a Dashboard from it.  That worked just as explained, but the next day the data is gone.  I was using the "AllTime" filter, so it was not that I missed data that was getting older, and skipped by filters.

source="us_drought_monitor.csv" State = CA date_year=2018| rex field=County "(?<County>.+) County"| eval droughtscore = D1 + D2*2 + D3*3 + D4*4| stats avg(droughtscore) as "2018 Drought Score" by County| geom ca_county_lookup featureIdField=County

above is the search SPL for the demo. 

While I have your attention, in the tutorial they add a min and max function, ""2018 Drought Score" max(droughtscore) as "Max 2018 Drought Score" min(droughtscore) as "Min 2018 Drought Score" by County"  This provided SPL code broke the Dashboard yesterday when it was working. is there something wrong with this SPL, that was provided?

Labels (1)
0 Karma
1 Solution

PickleRick
Ultra Champion

Since the data seems to be from way back it is possible that the index you're ingesting the samples to has retention period shorter than the data age.

In that case the data wouldh initially get ingested and indexed properly but when the bucket gets closed and moved from hot to cold splunk's maintenance thread would noticd that the youngest event in the bucket was older thsn retention period and would freeze (effectively deleting if you don't have any other freezing method configured) the bucket with sample data.

View solution in original post

0 Karma

SplunkAdmin69
Engager

@danielcj, if you log on to SplunkCloud you can't miss the SPL tutorial.  It was the first thing presented to me.

In SplunkCloud go to the upper right corner, and click on "Help with this page" while you are in Search.

0 Karma

PickleRick
Ultra Champion

Since the data seems to be from way back it is possible that the index you're ingesting the samples to has retention period shorter than the data age.

In that case the data wouldh initially get ingested and indexed properly but when the bucket gets closed and moved from hot to cold splunk's maintenance thread would noticd that the youngest event in the bucket was older thsn retention period and would freeze (effectively deleting if you don't have any other freezing method configured) the bucket with sample data.

0 Karma

SplunkAdmin69
Engager

@PickleRick sounds correct to me.  What is this "freezing" you speak of, sounds like not deleted, but hidden?  What is the point of hiding data that is still retained?  I don't necessarily expect an answer to the second question.  Also I will search for freezing.

 

Thanks for answering my question.

0 Karma

PickleRick
Ultra Champion

In Splunk data lifecycle there are several stages the bucket can be in.

First the bucket is hot. Splunk is writing new data to it.

Then after the bucket meets some parameters (size, age), the "housekeeping" thread closes the bucket and moves it to another directory on the same storage - it's called warm bucket.

After the bucket is old enough, if you have separate storage defined for it, the bucked is moved to the cold storage - it's a cold bucket.

And finally after the bucket reaches the retention period or the index gets too big, the bucket is frozen - if you have special storage defined for it, the bucket is moved there and removed from splunk (it's meant for archiving and not for direct use anymore). But if you don't have the frozen storage defined (which is the default config), the bucket simply gets deleted.

0 Karma

danielcj
Communicator

Hello @SplunkAdmin69 ,

Sorry, I am not familiar with this tutorial, is this a public Splunk tutorial? If yes, could you please post the link of this tutorial?

In the first case, if the data is not being shown now you would need to verify: 1) if the role of the user that is searching the data have permissions to search this index. 2) The retention size/time of the index (if the retention is exceeded, this data will be deleted).

For the second case, could you please share the complete query?

Thanks.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...