Monitoring Splunk

Duplicate logs.

Path Finder

Hi Splunker;

Splunk monitoring logs from URL by reset API, but we noticed that there duplicate logs, I mean Splunk read more than one for one event, you can see the below inputs.conf which I used?

[rest://port scanner from cloud ps.log]
auth_type = none
endpoint =
host =
http_method = GET
http_proxy =
index = ps
index_error_response_codes = 0
response_handler = DefaultResponseHandler
response_type = json
sequential_mode = 0
sourcetype = ps:ports
streaming_request = 0
polling_interval = 420

Please help me in that...


Tags (1)
0 Karma


hmm the issue is with the polling. You are polling at an interval of 420 s but it is still pulling the historical data. Before we get into the complexities of the checkpoints, is it possible for you to tune your endpoint, so that when its is queried it only returns the 'delta' data ? In most APIs it is possible to set some sort of start/end timings and you can probably do a smoke test by manually changing / modifying the time start/end values in the api endpoint url itself?

0 Karma


You are not monitoring a log in splunk's sense of monitoring. You are just GETting a file and index it. Thats why you get the duplicated logs. When you are if fact monitoring (e.g. log file available locally) splunk is able to established where he did stop indexing in the file and restarts from there.
Usually with rest you should be able too use the query string to pass additional parameters (e.g. start data, end data, etc.) to filter data before indexing.

Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

Path Finder

Thank you @diogofgm and @Sukisen1981 for reply;

These logs contains the timestamp and I set configuration about timestamp on props.conf, you can see the below, Mean that Splunk must look in the timestamp to read the new logs coming depends on the new timestamp, why read the historical logs?

category = Application
description = Ports logs produced by ps
pulldown_type = true

LINE_BREAKER = ({)\'timestamp
TIME_FORMAT = %Y-%m-%d %H:%M:%S
TIME_PREFIX = \'timestamp\':\s\'

And you can see the sample logs from these logs in the below:

'tcp'}]}\n{'timestamp': '2019-09-01 10:00:56', 'data': [{'status': 'open', 'host': '', 'port': '80', 'proto': 'tcp'}]}\n{'timestamp': '2019-09-01 10:00:56', 'data': [{'status': 'open', 'host': '', 'port': '443', 'proto':
'tcp'}]}\n{'timestamp': '2019-09-01 10:00:56', 'data': [{'status': 'open', 'host': '', 'port': '443', 'proto': 'tcp'}]}\n{'timestamp': '2019-09-01 10:00:56', 'data': [{'status': 'open', 'host': '', 'port': '80', 'proto':

Then, how can I add parameters?


0 Karma


Do you own the api endpoint or know who does? Do you have any documentation Regarding the api? If so, check if there’s a way to filter the data you are retrieving. Like I said by passing parameter in the query string along the lines
Depending on how the api was coded these will probably be different or even non existent.

Hope I was able to help you. If so, some karma would be appreciated.
0 Karma


Hi @aalhabbash1
None of the time parameters in your props.conf is related to purging of the historical data.
What you are showing in the logs,is an example of what your endpoint data retrieves , what we are asking is something like this - end?=yyyy

We refer to the query string that you are using to make the GET request through your endpoint.
My strong suggestion is to make changes in your GET endpoint rather than try to use Splunk to filter out stuff

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...