All Apps and Add-ons

DBConnect 3 - Need to use multiple columns to distinguish Rising Column filter

ferdbiffle
Explorer

We would like to use Splunk and DBConnect to archive unique records for several datasets. Each of these datasets are produced on a daily basis through batch processing. Over 95% of the records each day have not changed or are not new records. We would like to collect and index only those changed and new records from each daily run.

Has anyone created an effective, working method to do this type of filtering?

0 Karma

ferdbiffle
Explorer

Thanks for your reply grittonc 🙂

In each case, the inputs that need to be filtered are single table/view database sources. Therefore, we need a "blended" rising column (with multiple column values) that can look for changes in daily batch records vs. existing records on previously indexed data in Splunk.

Does that make sense?

0 Karma

grittonc
Contributor

I would think about creating a db input for each rising column that you might use. You can have the same basic query for every input, but use a different rising column in each input.

For example, if you use data from tables A, B, and C in your query, could you have one input that will ingest new/updated records from table A, and another input to ingest new/updated records from table B?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...