AWS Missing feeds

realtimetechnol — Wed, 10 Jun 2020 08:48:07 GMT

Hi All,

I am in the process of creating an app for AWS sources and one of the objectives is to alert when an account stops sending events. As there are a number of sources per account I have settled on the one source that all accounts provide, CloudTrail. There may be a flaw in my logic but I will continue for now. I have looked at the TA's data however we aggregate into a single account so this would not be representative. The feeds are via either the AWS TA or HEC.

The problem I am having is to alert when the source stops, I have created a similar app to DMC i.e. a list of accounts feed into a lookup but the problem I am having is that unlike DMC there is no avg__tcp_kbps etc to understand if the feed has stopped. I also looked at taking the date_second, date_mday etc but need a way of updating the lookup csv with the values.

My questions is, has anyone done anything similar with AWS feeds or are there some values around time that I could use?

Thanks

Re: AWS Missing feeds

livehybrid — Thu, 11 Jun 2020 23:27:01 GMT

I might have the wrong end of the stick (apologies)...

We collect Cloudtrail (and others) from a number (120+) of accounts into a single index and extract the account id at index time. You can then use tstats to quickly look when an account was last seen...

something like this might do it:

#props.conf [set_aws_account_id] SOURCE_KEY = _raw REGEX = \"accountId\": \"([0-9]{12})\" FORMAT = aws_account_id::$1 #transforms.conf [aws:cloudtrail] TRANSFORMS-get_aws_account_id=set_aws_account_id

Then search:

| tstats latest(_time) as latest where index=<your_aws_index> earliest=-24h by aws_account_id | eval recent = if(latest > relative_time(now(),"-5m"),1,0), realLatest = strftime(latest,"%c") | where recent=0

topic AWS Missing feeds in Monitoring Splunk

AWS Missing feeds

Re: AWS Missing feeds