topic Every 1st of month Splunk stops showing data indexed from Splunk DB Connect in Splunk Search

Every 1st of month Splunk stops showing data indexed from Splunk DB Connect

alvaromoraes — Thu, 01 Aug 2013 13:16:08 GMT

Hello,

I have some queries running at Splunk DB Connect, when month changes, like today (from July to August), it always stop showing new indexed data. I am sure that queries are running and events are being indexed (see logs below), because "Summary" dashboard shows it. Every 1st day of month all the charts and dashboards that use data from Splunk DB Connect stop working... every month is the same problem. Anyone know some workaround? I already tried to reinstall Splunk DB Connect without any success.

dbx.log:
2013-08-01 10:11:43.089 monsch1:INFO:Scheduler - Execution of input=[dbmon-dump://PSAV/SYSTEMDELAY ] finished in duration=0 ms with resultCount=1 success=true continueMonitoring=true

Summary Dashboard:
sourcetype Count Last Update
1 SYSTEMDELAY 8,409 Thu Aug 1 10:12:43 2013

Re: Every 1st of month Splunk stops showing data indexed from Splunk DB Connect

lukejadamec — Thu, 01 Aug 2013 13:23:47 GMT

How do you get it working again?
Do you get data when you do a simple search?
Are the charts and dashboards scheduled?

Re: Every 1st of month Splunk stops showing data indexed from Splunk DB Connect

alvaromoraes — Thu, 01 Aug 2013 13:30:09 GMT

1) To get working again I reinstall everything without data backup (with data the problem usually occurs again).
2) Even in a simple search the problem is the same, if I run a search of Last 15 minutes no data is found ("No matching events found. Inspect ...").
3) No, they aren't.

Thank you for the answer.

Re: Every 1st of month Splunk stops showing data indexed from Splunk DB Connect

lukejadamec — Thu, 01 Aug 2013 13:38:17 GMT

Is it broken right now? Or have you reinstalled everything?

Re: Every 1st of month Splunk stops showing data indexed from Splunk DB Connect

alvaromoraes — Thu, 01 Aug 2013 13:44:34 GMT

It is broken at moment, I will not try to reinstall until tomorrow.

Re: Every 1st of month Splunk stops showing data indexed from Splunk DB Connect

lukejadamec — Thu, 01 Aug 2013 13:51:44 GMT

Good (for troubleshooting that is).
When you run a query from the DBX console, does it timeout or show any errors?
Also, what type of database?

Re: Every 1st of month Splunk stops showing data indexed from Splunk DB Connect

alvaromoraes — Thu, 01 Aug 2013 13:57:53 GMT

We use an Oracle database.
Yes, when I run queries from the DBX in "Database Query" option everything works ok and fast.

Re: Every 1st of month Splunk stops showing data indexed from Splunk DB Connect

lukejadamec — Thu, 01 Aug 2013 14:05:31 GMT

With a simple search, do a search for the most recent session ID or other unique field that is listed in the dbx query, and search all time.

Re: Every 1st of month Splunk stops showing data indexed from Splunk DB Connect

alvaromoraes — Thu, 01 Aug 2013 14:38:35 GMT

Man, what kind of black magic was that? lol
Now it works like a charm!

What I did:
1) In Splunk DB Connect I went to "Search" option (not in the Search app).
2) Run the search for "All the time" range (without any search string). Some events from today returned \o/
3) Verified my dashboards and 1st August is showing ok now!

lukejadamec, thank you so much for the help! 🙂

If you can, put your answer below so I can vote to this as best answer!
:D

Thank you!

Re: Every 1st of month Splunk stops showing data indexed from Splunk DB Connect

lukejadamec — Thu, 01 Aug 2013 14:49:30 GMT

When you run a query from the DBX console, does it timeout or show any errors?
Also, what type of database?
With a simple search, do a search for the most recent session ID or other unique field that is listed in the dbx query, and search all time.

Re: Every 1st of month Splunk stops showing data indexed from Splunk DB Connect

alvaromoraes — Thu, 01 Aug 2013 14:52:00 GMT

Thank you again! 🙂