Getting Data In

Splunk_TA_aws duplicated events

rayar
Contributor

Hi

I have Splunk_TA_aws installed on the heave forwarder  

the input are 

[aws_s3://aws_dome9_logs_amdocsdome9logs]
aws_account = IS account
bucket_name = amdocsdome9logs
character_set = auto
ct_blacklist = ^$
host_name = s3.amazonaws.com
index = aws_dome9_logs
initial_scan_datetime = 2018-01-01T21:54:23-0700
interval = 30
is_secure = True
max_items = 100000
max_retries = 3
recursion_depth = -1
sourcetype = _json_current_time

[aws_s3://aws_dome9_logs_amdocsdome9remediationlogs]
aws_account = IS account
bucket_name = amdocsdome9remediationlogs
character_set = auto
ct_blacklist = ^$
host_name = s3.amazonaws.com
index = aws_dome9_logs
initial_scan_datetime = 2018-01-01T21:54:23-0700
interval = 30
is_secure = True
max_items = 100000
max_retries = 3
recursion_depth = -1
sourcetype = _json_current_time

 

what can be the reason the same event if indexed twice (day after day ) 
according to the json file diff the files are identical 

Labels (1)
Tags (1)
0 Karma

thambisetty
Super Champion

You could say its problem/issue with TA if you are observing same behavior for all the inputs you have created using the TA.

I would recommend creating a new input which you don’t think the events of this input are not duplicated with other input in AWS.

————————————
If this helps, give a like below.
0 Karma

rayar
Contributor

Hi

I am checking now with Dome9 team 

will update 

0 Karma

richgalloway
SplunkTrust
SplunkTrust
Are you saying you have the same file in two places and Splunk is indexing it from both places? If so, that is normal. Splunk is merely doing what it was asked to do. It doesn't know the same data is already indexed.
The solution is to not put the same data in two places that are monitored by Splunk.
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...