What are the requirements for the DB Connect Rising Column?

[Adam_Marx]

I'm learning how to utilize the DB Connect process and I'm somewhat hung up on the "Rising Column". On each execution of this query, there will potentially be duplicate data which will already have been indexed and I want to avoid re-indexing the data.

What are the requirements for this column?

Thanks

[Adam_Marx]

Ah, thanks I was looking for that...

[gjanders · Accepted solution]

Quoting from the Splunk add on for DB connect manual

"
Rising input

A rising input has a column that DB Connect uses to keep track of what rows are new from one input execution to the next. When you create a rising input type, you must specify the rising column. You can specify rising column as any column whose value increases or decreases over time, such as a timestamp or sequential ID. For example, you can use columns such as row_id, transaction_id, employee_id, customer_id, last_updated, and so on.

Note that timestamps are not ideal for rising columns, though they often are the best available choice. Using a timestamp for rising column can produce the following problem conditions:

• A high rate of event generation can cause data duplication or loss, because checkpointing in a stream of timestamp-distinguished records assumes there is never more than one row created in a given time. If you set the time to a one second level of resolution and get five records per second, you lose or duplicate four records from every run.
• Clock skew, NTP skew corrections, physical moves between timezones, and daylight savings events can cause data mis-ordering, duplication, or loss. If the skew is towards the future, then the resulting checkpoint value may temporarily or permanently stop data collection.
• Non-numeric datetime values cannot be evaluated numerically, and lexical sorting can produce unpredictable results. If time series data is ordered lexically, then the resulting checkpoint value may temporarily or permanently stop data collection.

"