Archive

Best way to manage extra field from raw log

Communicator

I imported some custom log for file auditing. each log message is very long, it has 7 type of messages. To normalize /extra useful field from the raw log, I wrote 7 separate regex to fully extra every line of the log file. so props.conf file end up like this.

My question is : Is this a right/good way to manage field extraction in this situation, or I should write an app to manage this imperatively. Will this causing any performance issue?
Thanks

[customlog]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
TIME_FORMAT = %y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX = TimeCreated SystemTime=
category = Custom
 disabled = false
pulldown_type = 1
SHOULD_LINEMERGE = false
TZ = Australia/Canberra
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$1 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+" \w+=\"(?<UnixUID>\d+)\" \w+=\"(?<UnixGID>\d+)\" \w+=\"(?<UnixIsLocal>\w+)\"><\/\w+><\w+\s\w+=\"\w+">(?<SubjectUserSid>[\w\-\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>[\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectServer>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectType>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<HandleID>[\w\d\;]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectName>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<AccessList>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<AccessMask>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<DesiredAccess>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?(?=<)(?<Attributes>)|(?<Attribute>[^<]+))<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$2 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<AuthPackageName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$3 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$4 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+" \w+=\"(?<UnixUID>\d+)\" \w+=\"(?<UnixGID>\d+)\" \w+=\"(?<UnixIsLocal>\w+)\"><\/\w+><\w+\s\w+=\"\w+">(?<SubjectUserSid>[\w\-\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>[\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\$\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectServer>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectType>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<HandleID>[\w\d\;]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectName>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<InformationRequested>[^<]+)<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$5 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\w\$\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<AuthenticationPackageName>[\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$6 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><[^<]+<\/\w+><[\w\"\=\s]+>(?<SubjectUserSid>[\w\-]+)<\/\w+><[\w\"\=\s]+>(?<SubjectUserIsLocal>\w+)<\/\w+><[\w\"\=\s]+>(?<subjectDomainName>\w+)<\/\w+><[\w\"\=\s]+>(?<TargetUserName>[\w\_]+)<\/\w+><[\w\"\=\s]+>(?<ObjectServer>\w+)<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>(?<OldPath>[^<]+)<\/\w+><[\w\"\=\s]+>(?<NewPath>[^<]+)<\/\w+><[\w\"\=\s]+>(?<Attributes>)<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$7 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><[^<]+<\/\w+><[\w\"\=\s]+>(?<SubjectUserSid>[\w\-]+)<\/\w+><[\w\"\=\s]+>(?<SubjectUserIsLocal>\w+)<\/\w+><[\w\"\=\s]+>(?<subjectDomainName>\w+)<\/\w+><[\w\"\=\s]+>(?<TargetUserName>[\w\_]+)<\/\w+><[\w\"\=\s]+>(?<ObjectServer>\w+)<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>(?<ObjectName>[^<]+)<\/\w+><[\w\"\=\s]+>(?<WriteOffset>\d+)<\/\w+><[\w\"\=\s]+>(?<WriteCount>\d+)
Tags (2)
0 Karma

Path Finder

Wow, probably better to try and convert the message into a proper XML message and have splunk automatically extract the tags for you.

You can then get rid of all the regex and setup field alises if you need the fields to be different names to the tags.

0 Karma