Duplications from ServiceNow into Splunk

[cardanid]

It appears when trying to pull the sys_transaction table into Splunk (still looking at other tables), I am getting duplications.

1st issue I see is that the "sys_created_on" field from the Splunk ServiceNow app is not matching what is in the actual rest call to ServiceNow via the logs. Also it pulling the same records based on the SN search.

See screen shot attached.

[snow://syslog_transaction]
account = ServiceNow SAND2
duration = 60
filter_data = sysparm_query=sys_created_byCONTAINSsys_rest
id_field = sys_id
index = be03_service_now
since_when = 2019-09-19 00:00:00
table = syslog_transaction
timefield = sys_created_on
disabled = 0

2019-09-19 15:04:38,199 INFO pid=27829 tid=Thread-5 file=snow_job_factory.py:__call__:50 | End collecting data from table syslog_transaction for input syslog_transaction
2019-09-19 15:05:30,685 INFO pid=27829 tid=Thread-2 file=snow_job_factory.py:__call__:34 | Start collecting data from table syslog_transaction for input syslog_transaction
2019-09-19 15:05:30,685 INFO pid=27829 tid=Thread-2 file=snow_data_loader.py:_do_collect:160 | Initiating request to https://allstatesand2.service-now.com/api/now/table/syslog_transaction?sysparm_query=sys_created_byCONTAINSsys_rest&sysparm_display_value=all&sysparm_limit=1000&sysparm_exclude_reference_link=true&sysparm_query=sys_created_on>=2019-08-15+09:05:28^ORDERBYsys_created_on
2019-09-19 15:05:40,093 INFO pid=27829 tid=Thread-2 file=snow_data_loader.py:_do_collect:178 | Ending request to https://allstatesand2.service-now.com/api/now/table/syslog_transaction?sysparm_query=sys_created_byCONTAINSsys_rest&sysparm_display_value=all&sysparm_limit=1000&sysparm_exclude_reference_link=true&sysparm_query=sys_created_on>=2019-08-15+09:05:28^ORDERBYsys_created_on
2019-09-19 15:05:40,307 INFO pid=27829 tid=Thread-2 file=snow_data_loader.py:collect_data:150 | Data collection completed for input syslog_transaction. Got 1000 records from https://allstatesand2.service-now.com/syslog_transaction.
2019-09-19 15:05:40,476 INFO pid=27829 tid=Thread-2 file=snow_job_factory.py:__call__:50 | End collecting data from table syslog_transaction for input syslog_transaction
2019-09-19 15:06:30,686 INFO pid=27829 tid=Thread-1 file=snow_job_factory.py:__call__:34 | Start collecting data from table syslog_transaction for input syslog_transaction
2019-09-19 15:06:30,686 INFO pid=27829 tid=Thread-1 file=snow_data_loader.py:_do_collect:160 | Initiating request to https://allstatesand2.service-now.com/api/now/table/syslog_transaction?sysparm_query=sys_created_byCONTAINSsys_rest&sysparm_display_value=all&sysparm_limit=1000&sysparm_exclude_reference_link=true&sysparm_query=sys_created_on>=2019-08-15+09:05:28^ORDERBYsys_created_on
2019-09-19 15:06:37,861 INFO pid=27829 tid=Thread-1 file=snow_data_loader.py:_do_collect:178 | Ending request to https://allstatesand2.service-now.com/api/now/table/syslog_transaction?sysparm_query=sys_created_byCONTAINSsys_rest&sysparm_display_value=all&sysparm_limit=1000&sysparm_exclude_reference_link=true&sysparm_query=sys_created_on>=2019-08-15+09:05:28^ORDERBYsys_created_on
2019-09-19 15:06:38,077 INFO pid=27829 tid=Thread-1 file=snow_data_loader.py:collect_data:150 | Data collection completed for input syslog_transaction. Got 1000 records from https://allstatesand2.service-now.com/syslog_transaction.
2019-09-19 15:06:38,233 INFO pid=27829 tid=Thread-1 file=snow_job_factory.py:__call__:50 | End collecting data from table syslog_transaction for input syslog_transaction

[cardanid]

I am not able to upload screen shots

[cardanid]

This is interesting, its changing the times of the pull from somewhere.

** start_process_at>=2019-09-20+11:00:00^ORDERBYstart_process_at

2019-09-20 14:27:01,450 INFO pid=3635 tid=Thread-7 file=snow_data_loader.py:_do_collect:178 | Ending request to https://allstatesand2.service-now.com/api/now/table/syslog_transaction?sysparm_query=sys_created_byC...
2019-09-20 14:27:01,650 INFO pid=3635 tid=Thread-7 file=snow_data_loader.py:collect_data:150 | Data collection completed for input syslog_transaction. Got 1000 records from https://allstatesand2.service-now.com/syslog_transaction.

**start_process_at>=2019-08-15+09:05:28^ORDERBYstart_process_at
2019-09-20 14:28:02,114 INFO pid=3635 tid=Thread-1 file=snow_data_loader.py:_do_collect:178 | Ending request to https://allstatesand2.service-now.com/api/now/table/syslog_transaction?sysparm_query=sys_created_byC...