In DB Connect, I only want to collect the new rows since the last update. Now each time it collects the last 1000 records because that is the Max Rows to Retrieve.
This is generally done by using Rising Column
. I see that you have configured BOTH tail_rising_column_name
and tail_rising_column_number
which may be your problem (if they are not the same and splunk is giving the wrong one precedence). I would delete one of those settings and make sure that the other one is correctly pointing to a value that rises with time (such as an epoch timestamp).
this is my inputs.conf
[mi_input://opentunnel_acc_metrics]
connection = opentunnel_acc
index = acc_opentunnel_db
interval = 150
max_rows = 10
mode = tail
output_timestamp_format = epoch
query = SELECT * FROM "OPENTUNNEL_RUNTIME"."METRICS"
source = /opt/splunk/var/log/splunk/acc_opentunnel_db.log
sourcetype = acc_opentunnel_db
tail_follow_only = 1
tail_rising_column_name = TUNNEL_ENTRY_TS
tail_rising_column_number = 1
ui_query_catalog = NULL
ui_query_mode = advanced
ui_query_schema = OPENTUNNEL_RUNTIME
ui_query_table = METRICS
tail_rising_column_checkpoint_value = 1443682436344
input_timestamp_column_name = TUNNEL_ENTRY_TS
input_timestamp_column_number = 17