Dashboard and data not remaining after 1 day?

[markitsecure]

I'm using the free version, I just started using it. I created a dashboard and had data from another box being sent to splunk via UDP and the dashboard I created with the 3 charts is gone.

I had it happen 2x now. I had created 3 charts in the dashboard yesterday, I saved and rebooted splunk a couple times yesterday to verify it was still there and saving properly but today when I check the dashboard is gone and the only data that exists is the data that splunk sees just this morning when I booted it up.

There must be a config I'm missing or something? Can someone please help.

[woodcock]

I don't know what version you are using but you should be using the full Enterprise splunk with a Dev license (also free) which is good for a year. Upgrade to that and you will be fine:
10GB = http://dev.splunk.com/page/developer_license_sign_up/

[markitsecure]

Thanks, I am using Splunk Enterprise free version.. 7.3.1

Don't know if it's with the Dev license how do I find that out?

I just logged in this morning and see that Sunday's data IS there from yesterday.
I didn't shut down the splunk instance last night like I did Saturday night when I found the prior days data missing.

I DID happen to find a "marks_dashboard" object under Settings > All Configurations. But nothing is in that XML file.

[Sukisen1981]

hi @markitsecure - Are you trying to say that you have a local version of splunk free enterprise AND that you shut down your system each night? Splunk won;t run if your laptop is shutdown right? I am getting a bit confused to be honest. Let's digress a bit here - say I am ingesting a rest api data in my local host splunk, it works, but if i shut it down for the day any new events from the rest api won;t be ingested and subsequently when i start my system the next day, the api has a back fill I will receive back the data during the time my system was shut down, failing which the data will be missing

[markitsecure]

Yeah I have a local version of Splunk Enterprise running on my laptop.

I had data from Saturday and a dashboard of charts saved.
I shut down Saturday night, turned it back on Sunday and Saturday data was gone including the dashboard and charts.

I understand that if you shut down the instance it won't ingest new data but the old data should remain.

[woodcock]

Seriously, just get the full version and a Dev License; that's what anybody serious about this does.

[woodcock]

The link is in my answer. You need to ensure that you have enough disk space to retain your events and also that the frozenTimeSeconds is set correctly for your index.

[markitsecure]

That was my issue !

I forgot I set that value to a very low value initially.

I had set frozenTimePeriodInSecs under "index specific defaults" in my indexes.conf file to 10800, just 3 hours worth of indexing . Set it to this because I was worried I would go over go over the data limit. I read something about the limits and just wanted to be extra careful I wouldn't make the app stop working on me.

Thank you! Still not sure why / where the dashboard went as that shouldn't be affected by this.

[woodcock]

If something is really deleting your dashboard (which is STILL not clear to me because of the way you keep phrasing things so unclearly), it must be either the Deployment Server or some other deployment tool that is enforcing idempotency of the app where you are working. If it is the DS, then it cannot modify user-level settings, only app or global, so if you are editing your own stuff, this cannot be the case. If you are modifying an existing dashboard/panel, this could be the case. In this case, clone the dashboard and keep the permissions at user level. Check with your Splunk admin to find out where you should be doing your work. We generally give each group a separate app which is NOT controlled by the DS/ansible/puppet/whatever.

[markitsecure]

I stated initially this is a free version of splunk. There are no user level settings in this version I'm told. This is merely a free version I downloaded for home use. I have no deployment server going on here.

[woodcock]

What exactly is "gone"? Are the events that drive the dashboard gone? Are the dashboard panels gone? Is the dashboard XML gone?

[markitsecure]

All prior days data is gone as well as dashboard. All I can see that remains is search history

[niketn]

@markitsecure is this clustered environment that you are using? How many SHs you have? Which version of Splunk are you using? Are you pushing the changes through Deployment Server?

[markitsecure]

Nope. Single splunk vm free version. Version 7.3.1 just downloaded the other day.
No deployment server involved. I have my home Firewall sending udp syslog to this single splunk instance running on a VM on my laptop. Really simple setup.

[Sukisen1981]

if you query just the index for your UDP data, are you able to see both days' data?

[markitsecure]

Nope. When I access my data through the search only this mornings data is there. I zoom out yesterday's data is totally gone.

[diogofgm]

regarding the missing data, have you limited the index size? This might explain data being removed (thus missing).
regarding the dashboard, splunk doesn't remove dashboards. You might be missing them because, depending on how and where the dashboards where created you might not see them. Check setting > user interface > views. in the App dropdown select "All" and check if you find you dashboards in the list.

[markitsecure]

Don't know how I could have used up the index space I turned the splunk vm off yesterday after working with it.

Turned it back on this morning and hokus pokus happened.. puff all gone. I don't have much data being sent to it. Just some logs from my home firewall is all so it's very minimal. I have only have a couple of home computers and phones so the data is very minimal.

How do you limit the index size ?

I did find something alarming. Under Settings > Monitoring > Indexing >
Indexes and Volumes > Index and Volumes Instance. Event Indexes Only. I don't see ANY indexes listed. Index(0) is shown. I'm guessing this is bad.

Settings > User Interface (under the KNOWLEDGE topic) > Views. I see many items here including alerts, alert, charting, data_model_editor, etc.. I see two "dashboard" , "dashboards", "dashboard_live". Some are sharing APP, one is sharing GLOBAL. I opened each of them. Some are short XML files and a couple are longer but none contain information I had used and no dashboards are the ones I saved; "Marks Dashboard".

I saved the dashboard many times, saved and re-opened yesterday fine so this is a mystery.

I did find my Search History from yesterday is still there. So that's interesting.