Hi,
I'm sending logs from Windows machines to a log group in CloudWatch that sends to Splunk via Lambda function.
These logs are arriving in Splunk in the wineventlog sourcetype, but the parse is not correct.
In the raw source logs, I can view that the logs come in one line, and differently than the parse understands.
Example:
[Security] [4776] [Microsoft-Windows-Security-Auditing] [XXXXX] [The computer attempted to validate the credentials for an account. Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon Account: 000000 Source Workstation: CBBBB Error Code: 0x0]
I've tried to change the sourcetype, the format to CSV, deleted the line_breaker, but until now it does not work.
Does anyone know how I can parse these kinds of logs coming from log groups in AWS CloudWatch?
Thank you a lot.
index=_internal | head 1 | fields _raw _time | eval _raw="[Security] [4776] [Microsoft-Windows-Security-Auditing] [XXXXX] [The computer attempted to validate the credentials for an account. Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon Account: 000000 Source Workstation: CBBBB Error Code: 0x0]"
| rex "\[(?<category>.*?)\]\s\[(?<eventID>.*?)\]\s\[(?<eventlog>.*?)\]\s\[.*?\]\s\[(?<data>.*?)\]"
| rex field=data max_match=0 " (?<fieldname>[A-Z].*?):\s(?<fieldvalue>\S+)"
| eval tmp=mvzip(fieldname,fieldvalue,"="), raw=_raw
| rename tmp as _raw
| kv
| rename raw as _raw
| fields - field* raw
Everything was alright in the end.
Hi @to4kawa
Thanks for your answer.
I try this regex, I modified some parts and works well.
The problem is that all logs from windows is coming in this way. That's is not just this log.
Is that any way to use props and transforms or exist a sourcetype specific for this type of log coming from AWS log group?
https://splunkbase.splunk.com/app/1274/#/overview
Have you tried this?
Yes @to4kawa . I have this app to parse other types of events coming from AWS.
There is no sourcetype that match with my type of event.
I'm trying to do parsing with splunk within the raw log, but many errors appear.
I'm working on this regex:
\[(?<category>.*?)\]\s+\[(?P<EventCode>\d+)]\s+\[(?P<SourceName>.*?])\s+\[(?P<host>.*?)]\s+\[(?P<subject>[^.]+.)\s([^Account]+)
reference:
https://qiita.com/toshikawa/items/926c63a9f77a0835c94e
my japanese blog
transforms.conf
[your TRANSFORMS stanza]
SOURCE_KEY = field:data
REGEX = (?m)^\s*(?<name>[^:]+):[\t ]+(?<value>.*)$
FORMAT = "$1"::$2
REPEAT_MATCH = true
WRITE_META = true
this transforms.conf setting is aim to extract fields from data field.
[yours]
REGEX = \[(?<category>.*?)\]\s+\[(?P<EventCode>\d+)]\s+\[(?P<SourceName>.*?])\s+\[(?P<host>.*?)]\s+\[(?P<subject>[^.]+.)\s(?P<data>.*)
WRITE_META = true
it needs this, too.
Polishing the config files above. Below conf files worked perfectly for me
#props.conf
[aws:cloudwatch:s3]
TRANSFORMS-field_extraction_aws_windows_logs = parse_windows_logs_prefix,parse_windows_logs_suffix
#Transforms.conf
[parse_windows_logs_prefix]
REGEX = \[(?<LogName>.*?)\]\s+\[(?<Type>.*?)\]\s+\[(?P<EventCode>\d+)]\s+\[(?P<SourceName>.*?)\]\s+\[(?P<ComputerName>.*?)]\s+\[(?P<message>[^.]+.)\s(?P<body>[^]]+.)
FORMAT = LogName::"$1" Type::"$2" EventCode::"$3" Sourcename::"$4" ComputerName::"$5" message::"$6" body::"$7"
WRITE_META = true
[parse_windows_logs_suffix]
SOURCE_KEY = field:body
REGEX = (?m)^\s*(?<name>[^:]+):[\t ]+(?<value>.*)$
FORMAT = "$1"::"$2"
REPEAT_MATCH = true
WRITE_META = true
The sample event you posted doesn't have keys ( field names). can you map them to field names and I can write regex for you to extract them and keep in respective field. Regex will work if they are in always same structure.
Not sure on your current Splunk set up, but I have done this recently using Splunk Cloud and for AWS Microsoft AD, but not using Lambda, rather the Splunk Add-on for AWS. I configured the inputs as CloudWatch Logs. Logs are parsed correctly though I am missing things such as eventtypes.
I was also looking at giving Trumpet a go https://github.com/splunk/splunk-aws-project-trumpet
I see your point.
But even if I send to the aws:cloudwatchlogs the parse is not correct.
I read the github and the method is similar to the lambda, via HEC.
Do you have the Splunk Add-on for windows installed on your SH and IDX?
I've sent the Microsoft AD logs to aws:cloudwatchlogs:vpcflow. Though better practice would have to been to use a Kinesis stream.