Splunk Search

Best way to manage extra field from raw log

samlinsongguo
Communicator

I imported some custom log for file auditing. each log message is very long, it has 7 type of messages. To normalize /extra useful field from the raw log, I wrote 7 separate regex to fully extra every line of the log file. so props.conf file end up like this.

My question is : Is this a right/good way to manage field extraction in this situation, or I should write an app to manage this imperatively. Will this causing any performance issue?
Thanks

[customlog]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
TIME_FORMAT = %y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX = TimeCreated SystemTime=
category = Custom
 disabled = false
pulldown_type = 1
SHOULD_LINEMERGE = false
TZ = Australia/Canberra
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$1 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+" \w+=\"(?<UnixUID>\d+)\" \w+=\"(?<UnixGID>\d+)\" \w+=\"(?<UnixIsLocal>\w+)\"><\/\w+><\w+\s\w+=\"\w+">(?<SubjectUserSid>[\w\-\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>[\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectServer>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectType>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<HandleID>[\w\d\;]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectName>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<AccessList>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<AccessMask>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<DesiredAccess>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?(?=<)(?<Attributes>)|(?<Attribute>[^<]+))<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$2 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<AuthPackageName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$3 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$4 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+" \w+=\"(?<UnixUID>\d+)\" \w+=\"(?<UnixGID>\d+)\" \w+=\"(?<UnixIsLocal>\w+)\"><\/\w+><\w+\s\w+=\"\w+">(?<SubjectUserSid>[\w\-\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>[\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\$\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectServer>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectType>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<HandleID>[\w\d\;]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectName>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<InformationRequested>[^<]+)<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$5 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\w\$\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<AuthenticationPackageName>[\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$6 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><[^<]+<\/\w+><[\w\"\=\s]+>(?<SubjectUserSid>[\w\-]+)<\/\w+><[\w\"\=\s]+>(?<SubjectUserIsLocal>\w+)<\/\w+><[\w\"\=\s]+>(?<subjectDomainName>\w+)<\/\w+><[\w\"\=\s]+>(?<TargetUserName>[\w\_]+)<\/\w+><[\w\"\=\s]+>(?<ObjectServer>\w+)<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>(?<OldPath>[^<]+)<\/\w+><[\w\"\=\s]+>(?<NewPath>[^<]+)<\/\w+><[\w\"\=\s]+>(?<Attributes>)<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$7 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><[^<]+<\/\w+><[\w\"\=\s]+>(?<SubjectUserSid>[\w\-]+)<\/\w+><[\w\"\=\s]+>(?<SubjectUserIsLocal>\w+)<\/\w+><[\w\"\=\s]+>(?<subjectDomainName>\w+)<\/\w+><[\w\"\=\s]+>(?<TargetUserName>[\w\_]+)<\/\w+><[\w\"\=\s]+>(?<ObjectServer>\w+)<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>(?<ObjectName>[^<]+)<\/\w+><[\w\"\=\s]+>(?<WriteOffset>\d+)<\/\w+><[\w\"\=\s]+>(?<WriteCount>\d+)
Tags (2)
0 Karma

peterchenadded
Path Finder

Wow, probably better to try and convert the message into a proper XML message and have splunk automatically extract the tags for you.

You can then get rid of all the regex and setup field alises if you need the fields to be different names to the tags.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...