Splunk Search

Best way to manage extra field from raw log

samlinsongguo
Communicator

I imported some custom log for file auditing. each log message is very long, it has 7 type of messages. To normalize /extra useful field from the raw log, I wrote 7 separate regex to fully extra every line of the log file. so props.conf file end up like this.

My question is : Is this a right/good way to manage field extraction in this situation, or I should write an app to manage this imperatively. Will this causing any performance issue?
Thanks

[customlog]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
TIME_FORMAT = %y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX = TimeCreated SystemTime=
category = Custom
 disabled = false
pulldown_type = 1
SHOULD_LINEMERGE = false
TZ = Australia/Canberra
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$1 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+" \w+=\"(?<UnixUID>\d+)\" \w+=\"(?<UnixGID>\d+)\" \w+=\"(?<UnixIsLocal>\w+)\"><\/\w+><\w+\s\w+=\"\w+">(?<SubjectUserSid>[\w\-\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>[\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectServer>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectType>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<HandleID>[\w\d\;]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectName>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<AccessList>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<AccessMask>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<DesiredAccess>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?(?=<)(?<Attributes>)|(?<Attribute>[^<]+))<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$2 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<AuthPackageName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$3 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$4 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+" \w+=\"(?<UnixUID>\d+)\" \w+=\"(?<UnixGID>\d+)\" \w+=\"(?<UnixIsLocal>\w+)\"><\/\w+><\w+\s\w+=\"\w+">(?<SubjectUserSid>[\w\-\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>[\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\$\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectServer>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectType>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<HandleID>[\w\d\;]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectName>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<InformationRequested>[^<]+)<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$5 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\w\$\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<AuthenticationPackageName>[\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$6 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><[^<]+<\/\w+><[\w\"\=\s]+>(?<SubjectUserSid>[\w\-]+)<\/\w+><[\w\"\=\s]+>(?<SubjectUserIsLocal>\w+)<\/\w+><[\w\"\=\s]+>(?<subjectDomainName>\w+)<\/\w+><[\w\"\=\s]+>(?<TargetUserName>[\w\_]+)<\/\w+><[\w\"\=\s]+>(?<ObjectServer>\w+)<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>(?<OldPath>[^<]+)<\/\w+><[\w\"\=\s]+>(?<NewPath>[^<]+)<\/\w+><[\w\"\=\s]+>(?<Attributes>)<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$7 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><[^<]+<\/\w+><[\w\"\=\s]+>(?<SubjectUserSid>[\w\-]+)<\/\w+><[\w\"\=\s]+>(?<SubjectUserIsLocal>\w+)<\/\w+><[\w\"\=\s]+>(?<subjectDomainName>\w+)<\/\w+><[\w\"\=\s]+>(?<TargetUserName>[\w\_]+)<\/\w+><[\w\"\=\s]+>(?<ObjectServer>\w+)<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>(?<ObjectName>[^<]+)<\/\w+><[\w\"\=\s]+>(?<WriteOffset>\d+)<\/\w+><[\w\"\=\s]+>(?<WriteCount>\d+)
Tags (2)
0 Karma

peterchenadded
Path Finder

Wow, probably better to try and convert the message into a proper XML message and have splunk automatically extract the tags for you.

You can then get rid of all the regex and setup field alises if you need the fields to be different names to the tags.

0 Karma
Get Updates on the Splunk Community!

Splunk App Dev Community Updates – What’s New and What’s Next

Welcome to your go-to roundup of everything happening in the Splunk App Dev Community! Whether you're building ...

The Latest Cisco Integrations With Splunk Platform!

Join us for an exciting tech talk where we’ll explore the latest integrations in Cisco &#43; Splunk! We’ve ...

Enterprise Security Content Update (ESCU) | New Releases

In April, the Splunk Threat Research Team had 2 releases of new security content via the Enterprise Security ...