AWS Missing feeds

[realtimetechnol]

Hi All,

I am in the process of creating an app for AWS sources and one of the objectives is to alert when an account stops sending events. As there are a number of sources per account I have settled on the one source that all accounts provide, CloudTrail. There may be a flaw in my logic but I will continue for now. I have looked at the TA's data however we aggregate into a single account so this would not be representative. The feeds are via either the AWS TA or HEC.

The problem I am having is to alert when the source stops, I have created a similar app to DMC i.e. a list of accounts feed into a lookup but the problem I am having is that unlike DMC there is no avg__tcp_kbps etc to understand if the feed has stopped. I also looked at taking the date_second, date_mday etc but need a way of updating the lookup csv with the values.

My questions is, has anyone done anything similar with AWS feeds or are there some values around time that I could use?

Thanks

[livehybrid · Accepted solution]

I might have the wrong end of the stick (apologies)...

We collect Cloudtrail (and others) from a number (120+) of accounts into a single index and extract the account id at index time. You can then use tstats to quickly look when an account was last seen...

something like this might do it:

#props.conf
[set_aws_account_id]
SOURCE_KEY = _raw
REGEX =  \"accountId\": \"([0-9]{12})\"
FORMAT = aws_account_id::$1

#transforms.conf
[aws:cloudtrail]
TRANSFORMS-get_aws_account_id=set_aws_account_id

 Then search:

| tstats latest(_time) as latest where index=<your_aws_index> earliest=-24h by aws_account_id
| eval recent = if(latest > relative_time(now(),"-5m"),1,0), realLatest = strftime(latest,"%c")
| where recent=0