topic Re: Parse Windows Logs from log source AWS Cloudwatch in Getting Data In

How to parse Windows logs from log source in AWS CloudWatch via Lambda?

wvalente — Wed, 12 Aug 2020 18:02:27 GMT

Hi,

I'm sending logs from Windows machines to a log group in CloudWatch that sends to Splunk via Lambda function.

These logs are arriving in Splunk in the wineventlog sourcetype, but the parse is not correct.

In the raw source logs, I can view that the logs come in one line, and differently than the parse understands.

Example:

[Security] [4776] [Microsoft-Windows-Security-Auditing] [XXXXX] [The computer attempted to validate the credentials for an account. Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon Account: 000000 Source Workstation: CBBBB Error Code: 0x0]

I've tried to change the sourcetype, the format to CSV, deleted the line_breaker, but until now it does not work.

Does anyone know how I can parse these kinds of logs coming from log groups in AWS CloudWatch?

Thank you a lot.

Re: Parse Windows Logs from log source AWS Cloudwatch

thambisetty — Fri, 07 Aug 2020 14:27:10 GMT

The sample event you posted doesn't have keys ( field names). can you map them to field names and I can write regex for you to extract them and keep in respective field. Regex will work if they are in always same structure.

Re: Parse Windows Logs from log source AWS Cloudwatch

to4kawa — Sat, 08 Aug 2020 00:34:08 GMT

index=_internal | head 1 | fields _raw _time | eval _raw="[Security] [4776] [Microsoft-Windows-Security-Auditing] [XXXXX] [The computer attempted to validate the credentials for an account. Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon Account: 000000 Source Workstation: CBBBB Error Code: 0x0]" | rex "\[(?<category>.*?)\]\s\[(?<eventID>.*?)\]\s\[(?<eventlog>.*?)\]\s\[.*?\]\s\[(?<data>.*?)\]" | rex field=data max_match=0 " (?<fieldname>[A-Z].*?):\s(?<fieldvalue>\S+)" | eval tmp=mvzip(fieldname,fieldvalue,"="), raw=_raw | rename tmp as _raw | kv | rename raw as _raw | fields - field* raw

Everything was alright in the end.

Re: Parse Windows Logs from log source AWS Cloudwatch

wvalente — Mon, 10 Aug 2020 12:37:43 GMT

Hi @to4kawa

Thanks for your answer.

I try this regex, I modified some parts and works well.

The problem is that all logs from windows is coming in this way. That's is not just this log.

Is that any way to use props and transforms or exist a sourcetype specific for this type of log coming from AWS log group?

Re: Parse Windows Logs from log source AWS Cloudwatch

to4kawa — Mon, 10 Aug 2020 12:39:50 GMT

https://splunkbase.splunk.com/app/1274/#/overview

Have you tried this?

Re: Parse Windows Logs from log source AWS Cloudwatch

wvalente — Mon, 10 Aug 2020 15:06:14 GMT

Yes @to4kawa . I have this app to parse other types of events coming from AWS.

There is no sourcetype that match with my type of event.

I'm trying to do parsing with splunk within the raw log, but many errors appear.

I'm working on this regex:

\[(?<category>.*?)\]\s+\[(?P<EventCode>\d+)]\s+\[(?P<SourceName>.*?])\s+\[(?P<host>.*?)]\s+\[(?P<subject>[^.]+.)\s([^Account]+)

Re: Parse Windows Logs from log source AWS Cloudwatch

to4kawa — Mon, 10 Aug 2020 21:54:08 GMT

reference:
https://qiita.com/toshikawa/items/926c63a9f77a0835c94e
my japanese blog

transforms.conf

[your TRANSFORMS stanza]
SOURCE_KEY = field:data
REGEX = (?m)^\s*(?<name>[^:]+):[\t ]+(?<value>.*)$
FORMAT = "$1"::$2
REPEAT_MATCH = true
WRITE_META = true

this transforms.conf setting is aim to extract fields from data field.

[yours]
REGEX = \[(?<category>.*?)\]\s+\[(?P<EventCode>\d+)]\s+\[(?P<SourceName>.*?])\s+\[(?P<host>.*?)]\s+\[(?P<subject>[^.]+.)\s(?P<data>.*)
WRITE_META = true

it needs this, too.

Re: Parse Windows Logs from log source AWS Cloudwatch

lznger88_2 — Wed, 12 Aug 2020 06:24:32 GMT

Not sure on your current Splunk set up, but I have done this recently using Splunk Cloud and for AWS Microsoft AD, but not using Lambda, rather the Splunk Add-on for AWS. I configured the inputs as CloudWatch Logs. Logs are parsed correctly though I am missing things such as eventtypes.

I was also looking at giving Trumpet a go https://github.com/splunk/splunk-aws-project-trumpet

Re: Parse Windows Logs from log source AWS Cloudwatch

wvalente — Wed, 12 Aug 2020 14:13:41 GMT

I see your point.

But even if I send to the aws:cloudwatchlogs the parse is not correct.

I read the github and the method is similar to the lambda, via HEC.

Re: Parse Windows Logs from log source AWS Cloudwatch

lznger88_2 — Thu, 13 Aug 2020 00:34:54 GMT

Do you have the Splunk Add-on for windows installed on your SH and IDX?

I've sent the Microsoft AD logs to aws:cloudwatchlogs:vpcflow. Though better practice would have to been to use a Kinesis stream.

Re: Parse Windows Logs from log source AWS Cloudwatch

pratik_18 — Thu, 15 Jul 2021 13:45:56 GMT

Polishing the config files above. Below conf files worked perfectly for me

#props.conf

[aws:cloudwatch:s3]
TRANSFORMS-field_extraction_aws_windows_logs = parse_windows_logs_prefix,parse_windows_logs_suffix

#Transforms.conf

[parse_windows_logs_prefix]
REGEX = \[(?<LogName>.*?)\]\s+\[(?<Type>.*?)\]\s+\[(?P<EventCode>\d+)]\s+\[(?P<SourceName>.*?)\]\s+\[(?P<ComputerName>.*?)]\s+\[(?P<message>[^.]+.)\s(?P<body>[^]]+.)
FORMAT = LogName::"$1" Type::"$2" EventCode::"$3" Sourcename::"$4" ComputerName::"$5" message::"$6" body::"$7"
WRITE_META = true

[parse_windows_logs_suffix]
SOURCE_KEY = field:body
REGEX = (?m)^\s*(?<name>[^:]+):[\t ]+(?<value>.*)$
FORMAT = "$1"::"$2"
REPEAT_MATCH = true
WRITE_META = true