Getting Data In

How to parse Windows logs from log source in AWS CloudWatch via Lambda?

wvalente
Explorer

Hi,

I'm sending logs from Windows machines to a log group in CloudWatch that sends to Splunk via Lambda function.

These logs are arriving in Splunk in the wineventlog sourcetype, but the parse is not correct.

In the raw source logs, I can view that the logs come in one line, and differently than the parse understands.

Example:

[Security] [4776] [Microsoft-Windows-Security-Auditing] [XXXXX] [The computer attempted to validate the credentials for an account. Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon Account: 000000 Source Workstation: CBBBB Error Code: 0x0]

I've tried to change the sourcetype, the format to CSV, deleted the line_breaker, but until now it does not work.

Does anyone know how I can parse these kinds of logs coming from log groups in AWS CloudWatch?

Thank you a lot. 

0 Karma

to4kawa
Ultra Champion
index=_internal | head 1 | fields _raw _time | eval _raw="[Security] [4776] [Microsoft-Windows-Security-Auditing] [XXXXX] [The computer attempted to validate the credentials for an account. Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon Account: 000000 Source Workstation: CBBBB Error Code: 0x0]"
| rex  "\[(?<category>.*?)\]\s\[(?<eventID>.*?)\]\s\[(?<eventlog>.*?)\]\s\[.*?\]\s\[(?<data>.*?)\]"
| rex field=data max_match=0 " (?<fieldname>[A-Z].*?):\s(?<fieldvalue>\S+)"
| eval tmp=mvzip(fieldname,fieldvalue,"="), raw=_raw
| rename tmp as _raw
| kv
| rename raw as _raw
| fields - field* raw

Everything was alright in the end. 

0 Karma

wvalente
Explorer

Hi @to4kawa 

Thanks for your answer.

I try this regex, I modified some parts and works well.

The problem is that all logs from windows is coming in this way. That's is not just this log.

Is that any way to use props and transforms or exist a sourcetype specific for this type of log coming from AWS log group?

Tags (1)
0 Karma

to4kawa
Ultra Champion
0 Karma

wvalente
Explorer

Yes @to4kawa . I have this app to parse other types of events coming from AWS.

There is no sourcetype that match with my type of event.

I'm trying to do parsing with splunk within the raw log, but many errors appear.

I'm working on this regex:

\[(?<category>.*?)\]\s+\[(?P<EventCode>\d+)]\s+\[(?P<SourceName>.*?])\s+\[(?P<host>.*?)]\s+\[(?P<subject>[^.]+.)\s([^Account]+)

0 Karma

to4kawa
Ultra Champion

reference:
https://qiita.com/toshikawa/items/926c63a9f77a0835c94e
my japanese blog

transforms.conf

[your TRANSFORMS stanza]
SOURCE_KEY = field:data
REGEX = (?m)^\s*(?<name>[^:]+):[\t ]+(?<value>.*)$
FORMAT = "$1"::$2
REPEAT_MATCH = true
WRITE_META = true

 

this transforms.conf setting is aim to extract fields from data field.

[yours]
REGEX = \[(?<category>.*?)\]\s+\[(?P<EventCode>\d+)]\s+\[(?P<SourceName>.*?])\s+\[(?P<host>.*?)]\s+\[(?P<subject>[^.]+.)\s(?P<data>.*)
WRITE_META = true


it needs this, too.

pratik_18
Explorer

Polishing the config files above. Below conf files worked perfectly for me

#props.conf

[aws:cloudwatch:s3]
TRANSFORMS-field_extraction_aws_windows_logs = parse_windows_logs_prefix,parse_windows_logs_suffix

#Transforms.conf

[parse_windows_logs_prefix]
REGEX = \[(?<LogName>.*?)\]\s+\[(?<Type>.*?)\]\s+\[(?P<EventCode>\d+)]\s+\[(?P<SourceName>.*?)\]\s+\[(?P<ComputerName>.*?)]\s+\[(?P<message>[^.]+.)\s(?P<body>[^]]+.)
FORMAT = LogName::"$1" Type::"$2" EventCode::"$3" Sourcename::"$4" ComputerName::"$5" message::"$6" body::"$7"
WRITE_META = true

[parse_windows_logs_suffix]
SOURCE_KEY = field:body
REGEX = (?m)^\s*(?<name>[^:]+):[\t ]+(?<value>.*)$
FORMAT = "$1"::"$2"
REPEAT_MATCH = true
WRITE_META = true

0 Karma

thambisetty
SplunkTrust
SplunkTrust

The sample event you posted doesn't have keys ( field names).  can you map them to field names and I can write regex for you to extract them and keep in respective field. Regex will work if they are in always same structure.

————————————
If this helps, give a like below.
0 Karma

lznger88_2
Path Finder

Not sure on your current Splunk set up, but I have done this recently using Splunk Cloud and for AWS Microsoft AD, but not using Lambda, rather the Splunk Add-on for AWS. I configured the inputs as CloudWatch Logs. Logs are parsed correctly though I am missing things such as eventtypes.

I was also looking at giving Trumpet a go https://github.com/splunk/splunk-aws-project-trumpet

0 Karma

wvalente
Explorer

I see your point.

But even if I send to the aws:cloudwatchlogs the parse is not correct.

I read the github and the method is similar to the lambda, via HEC.

Tags (1)
0 Karma

lznger88_2
Path Finder

Do you have the Splunk Add-on for windows installed on your SH and IDX?

I've sent the Microsoft AD logs to aws:cloudwatchlogs:vpcflow. Though better practice would have to been to use a Kinesis stream. 

0 Karma
Get Updates on the Splunk Community!

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...