- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How to parse Windows logs from log source in AWS CloudWatch via Lambda?
Hi,
I'm sending logs from Windows machines to a log group in CloudWatch that sends to Splunk via Lambda function.
These logs are arriving in Splunk in the wineventlog sourcetype, but the parse is not correct.
In the raw source logs, I can view that the logs come in one line, and differently than the parse understands.
Example:
[Security] [4776] [Microsoft-Windows-Security-Auditing] [XXXXX] [The computer attempted to validate the credentials for an account. Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon Account: 000000 Source Workstation: CBBBB Error Code: 0x0]I've tried to change the sourcetype, the format to CSV, deleted the line_breaker, but until now it does not work.
Does anyone know how I can parse these kinds of logs coming from log groups in AWS CloudWatch?
Thank you a lot.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
index=_internal | head 1 | fields _raw _time | eval _raw="[Security] [4776] [Microsoft-Windows-Security-Auditing] [XXXXX] [The computer attempted to validate the credentials for an account. Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon Account: 000000 Source Workstation: CBBBB Error Code: 0x0]"
| rex "\[(?<category>.*?)\]\s\[(?<eventID>.*?)\]\s\[(?<eventlog>.*?)\]\s\[.*?\]\s\[(?<data>.*?)\]"
| rex field=data max_match=0 " (?<fieldname>[A-Z].*?):\s(?<fieldvalue>\S+)"
| eval tmp=mvzip(fieldname,fieldvalue,"="), raw=_raw
| rename tmp as _raw
| kv
| rename raw as _raw
| fields - field* rawEverything was alright in the end.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @to4kawa
Thanks for your answer.
I try this regex, I modified some parts and works well.
The problem is that all logs from windows is coming in this way. That's is not just this log.
Is that any way to use props and transforms or exist a sourcetype specific for this type of log coming from AWS log group?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
https://splunkbase.splunk.com/app/1274/#/overview
Have you tried this?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes @to4kawa . I have this app to parse other types of events coming from AWS.
There is no sourcetype that match with my type of event.
I'm trying to do parsing with splunk within the raw log, but many errors appear.
I'm working on this regex:
\[(?<category>.*?)\]\s+\[(?P<EventCode>\d+)]\s+\[(?P<SourceName>.*?])\s+\[(?P<host>.*?)]\s+\[(?P<subject>[^.]+.)\s([^Account]+)
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
reference:
https://qiita.com/toshikawa/items/926c63a9f77a0835c94e
my japanese blog
transforms.conf
[your TRANSFORMS stanza]
SOURCE_KEY = field:data
REGEX = (?m)^\s*(?<name>[^:]+):[\t ]+(?<value>.*)$
FORMAT = "$1"::$2
REPEAT_MATCH = true
WRITE_META = true
this transforms.conf setting is aim to extract fields from data field.
[yours]
REGEX = \[(?<category>.*?)\]\s+\[(?P<EventCode>\d+)]\s+\[(?P<SourceName>.*?])\s+\[(?P<host>.*?)]\s+\[(?P<subject>[^.]+.)\s(?P<data>.*)
WRITE_META = true
it needs this, too.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Polishing the config files above. Below conf files worked perfectly for me
#props.conf
[aws:cloudwatch:s3]
TRANSFORMS-field_extraction_aws_windows_logs = parse_windows_logs_prefix,parse_windows_logs_suffix
#Transforms.conf
[parse_windows_logs_prefix]
REGEX = \[(?<LogName>.*?)\]\s+\[(?<Type>.*?)\]\s+\[(?P<EventCode>\d+)]\s+\[(?P<SourceName>.*?)\]\s+\[(?P<ComputerName>.*?)]\s+\[(?P<message>[^.]+.)\s(?P<body>[^]]+.)
FORMAT = LogName::"$1" Type::"$2" EventCode::"$3" Sourcename::"$4" ComputerName::"$5" message::"$6" body::"$7"
WRITE_META = true
[parse_windows_logs_suffix]
SOURCE_KEY = field:body
REGEX = (?m)^\s*(?<name>[^:]+):[\t ]+(?<value>.*)$
FORMAT = "$1"::"$2"
REPEAT_MATCH = true
WRITE_META = true
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The sample event you posted doesn't have keys ( field names). can you map them to field names and I can write regex for you to extract them and keep in respective field. Regex will work if they are in always same structure.
If this helps, give a like below.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Not sure on your current Splunk set up, but I have done this recently using Splunk Cloud and for AWS Microsoft AD, but not using Lambda, rather the Splunk Add-on for AWS. I configured the inputs as CloudWatch Logs. Logs are parsed correctly though I am missing things such as eventtypes.
I was also looking at giving Trumpet a go https://github.com/splunk/splunk-aws-project-trumpet
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I see your point.
But even if I send to the aws:cloudwatchlogs the parse is not correct.
I read the github and the method is similar to the lambda, via HEC.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Do you have the Splunk Add-on for windows installed on your SH and IDX?
I've sent the Microsoft AD logs to aws:cloudwatchlogs:vpcflow. Though better practice would have to been to use a Kinesis stream.
