All Apps and Add-ons

Splunk DB Connect - data input updates

sabinayousoubuv
New Member

Hello,
I have an IDM DB in my organization that is connected to Splunk by DB Connect app.
The DB holds data about workflows in the system, their status (in process, completed, etc..), request ID and so on..
Today I noticed that the data inputs are added when the system has new requests, but when a request gets updated in the system (for example the status turns from in process to completed), the data does not change in Splunk.
So the status in my DB is completed, but in Splunk, it's still in process.
I tried running the SQL query again through the DATA LAB INPUTS, Splunk tells me that everything is updated.

Would love to get some help,
Sabina.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

This will happen if the DB replaces a field in an existing record without changing the rising column value. Maybe you just have the wrong rising column. What are you using for the rising column field and does it change when a row is updated?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

sabinayousoubuv
New Member

The data I have includes info about requests.
That means that I have 3 main values:
1. requestId (the ID of the request in the system)
2.status (completed, failed, in_process...)
3.statusUpdate (the date that the status changed)

So I have one row per every requestId. The input of the status and statusUpdate update according to the actions taken in each request, or creates new rows when new requests are made.
The update of each request does not create a new row, it updates the input in the existing row of the relevant requestId.

the input type is rising, and the rising column I picked is statusUpdate (which again, creates new rows only in new events, but only updates in existing ones).

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Since you have a good candidate for a rising column (statusUpdate) you next need to ensure it's used in your query. The query should look something like SELECT * FROM foo WHERE statusUpdate > bar ORDER BY statusUpdate. This will ensure the rows are returned in the order in which they were updated and then Splunk will index the changes.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

nickhills
Ultra Champion

When using a rising column, Splunk expects the data in that column to be updated when rows are updated.
If your database does not update the statusUpdate with a new (more recent) date when the row is changed, Splunk will not be able to identify the row has changed.

In that case, you will need to have the database logic amended so that statusUpdate IS modified, or you will not be able to use rising column - instead you may need to revert to a batch process.

Check that statusUpdate is a valid date, and that DBX is correctly configured to interpret it and it has a valid checkpoint.

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...