In Splunk DB Connect, can we give a composite key as a rising column?

[rahulnarang2107]

Can we give a composite key ( a combination of two primary keys) as a rising column? And If we can what should be the initial checkpoint value taking into consideration that it will be having GUID.

[martin_mueller · Accepted solution]

You can concatenate both columns into one, and use that combined column as your rising column.
That way you get to choose how a change in just one or just the other is supposed to work out, ie which is the "high bits" and which is the "low bits".

[martin_mueller]

Timestamps are often good enough. You just need to take care of what happens if two rows have the same timestamp, but one is read before the other is written to the DB. If missing such rare events is an issue you could for example introduce a small delay into your input by adding a where clause "timestamp is older than foo seconds", where foo is your expected maximum writing lag.

[martin_mueller]

A GUID as in a random-ish string of characters? That's not a rising column.
You'll need a column where the second value is greater-than the first value. Then DB Connect can store the first value and later ask for WHERE that_column > stored_checkpoint.

As for choosing the initial checkpoint value, that depends. If you want a full backfill, leave the checkpoint empty. If you don't want a full backfill, set the checkpoint value in a way such that all rows you want have a rising column greater-than your checkpoint value and all rows you don't want have a rising column not-greater-than your checkpoint value.

[rahulnarang2107]

Thank you Martin.

I am actually monitoring some logs which doesn't seem to have any values to compare to the last value (checkpoint). Would timestamp be a good option to do so for the rising column selection. as I think it creates more problems than it solves in rising column selection. any suggestions?

[rahulnarang2107]

Thank You Martin,

I agree with you. Let's say we write a query to create a composite key. but how do we specify the checkpoint value when we run for the very first time. and if we are choosing the rising column as ID let's say which is not an integer, a GUID, how do we select the checkpoint value.