topic Re: Splunk_TA_aws duplicated events in Getting Data In

Splunk_TA_aws duplicated events

rayar — Sun, 09 Aug 2020 13:15:57 GMT

I have Splunk_TA_aws installed on the heave forwarder

the input are

[aws_s3://aws_dome9_logs_amdocsdome9logs]
aws_account = IS account
bucket_name = amdocsdome9logs
character_set = auto
ct_blacklist = ^$
host_name = s3.amazonaws.com
index = aws_dome9_logs
initial_scan_datetime = 2018-01-01T21:54:23-0700
interval = 30
is_secure = True
max_items = 100000
max_retries = 3
recursion_depth = -1
sourcetype = _json_current_time

[aws_s3://aws_dome9_logs_amdocsdome9remediationlogs]
aws_account = IS account
bucket_name = amdocsdome9remediationlogs
character_set = auto
ct_blacklist = ^$
host_name = s3.amazonaws.com
index = aws_dome9_logs
initial_scan_datetime = 2018-01-01T21:54:23-0700
interval = 30
is_secure = True
max_items = 100000
max_retries = 3
recursion_depth = -1
sourcetype = _json_current_time

what can be the reason the same event if indexed twice (day after day )
according to the json file diff the files are identical

Re: Splunk_TA_aws duplicated events

richgalloway — Sun, 09 Aug 2020 16:01:36 GMT

Are you saying you have the same file in two places and Splunk is indexing it from both places? If so, that is normal. Splunk is merely doing what it was asked to do. It doesn't know the same data is already indexed.
The solution is to not put the same data in two places that are monitored by Splunk.

Re: Splunk_TA_aws duplicated events

thambisetty — Sun, 09 Aug 2020 16:44:48 GMT

You could say its problem/issue with TA if you are observing same behavior for all the inputs you have created using the TA.

I would recommend creating a new input which you don’t think the events of this input are not duplicated with other input in AWS.

Re: Splunk_TA_aws duplicated events

rayar — Mon, 10 Aug 2020 09:03:53 GMT

I am checking now with Dome9 team

will update