Getting Data In

Splunk_TA_aws duplicated events

rayar
Communicator

Hi

I have Splunk_TA_aws installed on the heave forwarder  

the input are 

[aws_s3://aws_dome9_logs_amdocsdome9logs]
aws_account = IS account
bucket_name = amdocsdome9logs
character_set = auto
ct_blacklist = ^$
host_name = s3.amazonaws.com
index = aws_dome9_logs
initial_scan_datetime = 2018-01-01T21:54:23-0700
interval = 30
is_secure = True
max_items = 100000
max_retries = 3
recursion_depth = -1
sourcetype = _json_current_time

[aws_s3://aws_dome9_logs_amdocsdome9remediationlogs]
aws_account = IS account
bucket_name = amdocsdome9remediationlogs
character_set = auto
ct_blacklist = ^$
host_name = s3.amazonaws.com
index = aws_dome9_logs
initial_scan_datetime = 2018-01-01T21:54:23-0700
interval = 30
is_secure = True
max_items = 100000
max_retries = 3
recursion_depth = -1
sourcetype = _json_current_time

 

what can be the reason the same event if indexed twice (day after day ) 
according to the json file diff the files are identical 

Labels (1)
Tags (1)
0 Karma

thambisetty
Super Champion

You could say its problem/issue with TA if you are observing same behavior for all the inputs you have created using the TA.

I would recommend creating a new input which you don’t think the events of this input are not duplicated with other input in AWS.

————————————
If this helps, give a like below.
0 Karma

rayar
Communicator

Hi

I am checking now with Dome9 team 

will update 

0 Karma

richgalloway
SplunkTrust
SplunkTrust
Are you saying you have the same file in two places and Splunk is indexing it from both places? If so, that is normal. Splunk is merely doing what it was asked to do. It doesn't know the same data is already indexed.
The solution is to not put the same data in two places that are monitored by Splunk.
---
If this reply helps you, an upvote would be appreciated.
0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!