Splunk_TA_aws duplicated events

[rayar]

Hi

I have Splunk_TA_aws installed on the heave forwarder  

the input are 

[aws_s3://aws_dome9_logs_amdocsdome9logs]
aws_account = IS account
bucket_name = amdocsdome9logs
character_set = auto
ct_blacklist = ^$
host_name = s3.amazonaws.com
index = aws_dome9_logs
initial_scan_datetime = 2018-01-01T21:54:23-0700
interval = 30
is_secure = True
max_items = 100000
max_retries = 3
recursion_depth = -1
sourcetype = _json_current_time

[aws_s3://aws_dome9_logs_amdocsdome9remediationlogs]
aws_account = IS account
bucket_name = amdocsdome9remediationlogs
character_set = auto
ct_blacklist = ^$
host_name = s3.amazonaws.com
index = aws_dome9_logs
initial_scan_datetime = 2018-01-01T21:54:23-0700
interval = 30
is_secure = True
max_items = 100000
max_retries = 3
recursion_depth = -1
sourcetype = _json_current_time

what can be the reason the same event if indexed twice (day after day ) 
according to the json file diff the files are identical 

[thambisetty]

You could say its problem/issue with TA if you are observing same behavior for all the inputs you have created using the TA.

I would recommend creating a new input which you don’t think the events of this input are not duplicated with other input in AWS.

[rayar]

Hi

I am checking now with Dome9 team 

will update 

[richgalloway]

Are you saying you have the same file in two places and Splunk is indexing it from both places? If so, that is normal. Splunk is merely doing what it was asked to do. It doesn't know the same data is already indexed.
The solution is to not put the same data in two places that are monitored by Splunk.