AWS Guardduty - Splunk Integration

[splunkmagu]

Hello,

we have a following integration in place:
GuardDuty -> EventBridge (no transformation)-> Firehose (no transformation) -> Splunk (cloud)

HEC sourcetype = aws:cloudwatch:guardduty
HEC source override = aws_cloudwatchevents_guardduty

Despite source override sometimes we see events with aws.guardduty source, and in those cases the message format is different (thus search outputs no results so we do not get an alert).

Single source events start with:
{"schemaVersion":"2.0","accountId":"<some_account>","region":"<aws_region>","partition":...}

Double source events start with (additional header/metadata preceding schemaVersion):
{"version":"0","id":"<some_id>","detail-type":"GuardDuty Finding","source":"aws.guardduty","account":"<some_account>","time":"2022-09-07T11:55:02Z","region":"<aws_region>","resources":[],"detail":{"schemaVersion":"2.0","accountId":"<some_account>","region":"<aws_region>","partition":...}

AWS has been excluded as the source of the issue.
Any ideas on how to have only one message format (Splunk Support ticket has also been submitted) ?

[rkantamaneni · Accepted solution]

@splunkmaguIn case you haven't resolved this yet, the problem is with the props/transforms. There's an extraction in the Add-On's default transforms.conf:

[extract_detail_from_cloudwatch_events]
DEST_KEY = _raw
REGEX = ^{[^{]+"detail":(.*)}$
FORMAT = $1

PROBLEM:

what's happening is that all the events are formatted the same from AWS, however the transform is not being applied 100% to extract the nested details JSON field and replace _raw (i.e. remove the outer wrapper). This is because some of the GuardDuty events can be large in size (up to 20k characters), and the parsing is using the default LOOKAHEAD of 4k (4096). You can see this by using the btool command: ./splunk btool transforms list --debug extract_detail_from_cloudwatch_events

SOLUTION:

If you create a transforms.conf in a local folder for the add-on, you can add the following stanza:

[extract_detail_from_cloudwatch_events]
LOOKAHEAD = 20480

This will tell Splunk to read an event up to the LOOKAHEAD length (20k in this case) to apply the transformation, which it needs to do or the end of the "detail" JSON field won't be read in full for large events and the extraction won't happen.

You can verify the longest guardduty event in your events by searching for your guardduty data and appending the search:

| eval raw_length = len(_raw) | stats max(raw_length)

Or if you're curious to see the event distribution, use a visualization to see the distribution in size of events:

index=<YOUR_aws_index> sourcetype=<YOUR_guardduty sourcetype> version=* OR schemaVersion=* 
| eval raw_length_kb=ceil(len(_raw)/1024) | stats count by raw_length_kb | sort raw_length_kb

I'd classify this as a bug, where the Splunk Add-On should have the LOOKAHEAD value defined with an appropriate value. The largest event I've seen in a recent 90 day search is around 20k (used a value of 20480).

[splunkmagu]

Thank you @rkantamaneni for that reply.

The issue has been 'resolved' some time ago but I believe your solution is the correct one.
Additionally your logic confirmed my observations that something 'wrong' happens when events are larger than 4k.

I'm just wondering why Splunk Support wasn't able to find it the way you did lol (I took around 6 months and a lot of persistence from my side to fix this :D).
At the time of creating this post I didn't know about an admin helper app that would allow us to run btool commands from the search head (we cannot access searchheads and indexers via ssh as this is a cloud deployment managed by Splunk).

I think I will submit another ticket regarding this because currently we strip the details 'header' using SEDCMD commands on a sourcetype level and I believe that increasing LOOKAHEAD value won't impact the performance that much as those two additional SEDCMD commands ?
1. SEDCMD-alter1    s/{"[\s\S]*"detail"://g
2. SEDCMD-alter2    s/}}$/}/g

[rkantamaneni]

@splunkmaguYeah, I believe increasing the LOOKAHEAD is probably better since the extraction is already in use, and wouldn't impact the events less than 4k.

For data already ingested, the Coalesce function can be used to resolve events to a common view during search time if needed. An example with a few fields:

index=<aws_index> sourcetype=aws:cloudwatch:guardduty 
| eval TITLE =coalesce('detail.title',title), ACCOUNT_ID = coalesce('detail.accountId',accountId), FINDING_TYPE = coalesce(findingType,'detail.type') 
| table _time, TITLE, ACCOUNT_ID, FINDING_TYPE

 NOTE: The single quotes for json fields matter in the coalesce function.