Every 1st of month Splunk stops showing data indexed from Splunk DB Connect

[alvaromoraes]

Hello,

I have some queries running at Splunk DB Connect, when month changes, like today (from July to August), it always stop showing new indexed data. I am sure that queries are running and events are being indexed (see logs below), because "Summary" dashboard shows it. Every 1st day of month all the charts and dashboards that use data from Splunk DB Connect stop working... every month is the same problem. Anyone know some workaround? I already tried to reinstall Splunk DB Connect without any success.

dbx.log:
2013-08-01 10:11:43.089 monsch1:INFO:Scheduler - Execution of input=[dbmon-dump://PSAV/SYSTEMDELAY ] finished in duration=0 ms with resultCount=1 success=true continueMonitoring=true

Summary Dashboard:
sourcetype Count Last Update
1 SYSTEMDELAY 8,409 Thu Aug 1 10:12:43 2013

[lukejadamec · Accepted solution]

When you run a query from the DBX console, does it timeout or show any errors?
Also, what type of database?
With a simple search, do a search for the most recent session ID or other unique field that is listed in the dbx query, and search all time.

[alvaromoraes]

Thank you again! 🙂

[alvaromoraes]

Man, what kind of black magic was that? lol
Now it works like a charm!

What I did:
1) In Splunk DB Connect I went to "Search" option (not in the Search app).
2) Run the search for "All the time" range (without any search string). Some events from today returned \o/
3) Verified my dashboards and 1st August is showing ok now!

lukejadamec, thank you so much for the help! 🙂

If you can, put your answer below so I can vote to this as best answer!
:D

Thank you!

[lukejadamec]

With a simple search, do a search for the most recent session ID or other unique field that is listed in the dbx query, and search all time.

[alvaromoraes]

We use an Oracle database.
Yes, when I run queries from the DBX in "Database Query" option everything works ok and fast.

[lukejadamec]

Good (for troubleshooting that is).
When you run a query from the DBX console, does it timeout or show any errors?
Also, what type of database?

[alvaromoraes]

It is broken at moment, I will not try to reinstall until tomorrow.

[lukejadamec]

Is it broken right now? Or have you reinstalled everything?

[alvaromoraes]

1) To get working again I reinstall everything without data backup (with data the problem usually occurs again).
2) Even in a simple search the problem is the same, if I run a search of Last 15 minutes no data is found ("No matching events found. Inspect ...").
3) No, they aren't.

Thank you for the answer.

[lukejadamec]

How do you get it working again?
Do you get data when you do a simple search?
Are the charts and dashboards scheduled?