Splunk Search

Best way to manage extra field from raw log

samlinsongguo
Communicator

I imported some custom log for file auditing. each log message is very long, it has 7 type of messages. To normalize /extra useful field from the raw log, I wrote 7 separate regex to fully extra every line of the log file. so props.conf file end up like this.

My question is : Is this a right/good way to manage field extraction in this situation, or I should write an app to manage this imperatively. Will this causing any performance issue?
Thanks

[customlog]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
TIME_FORMAT = %y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX = TimeCreated SystemTime=
category = Custom
 disabled = false
pulldown_type = 1
SHOULD_LINEMERGE = false
TZ = Australia/Canberra
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$1 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+" \w+=\"(?<UnixUID>\d+)\" \w+=\"(?<UnixGID>\d+)\" \w+=\"(?<UnixIsLocal>\w+)\"><\/\w+><\w+\s\w+=\"\w+">(?<SubjectUserSid>[\w\-\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>[\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectServer>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectType>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<HandleID>[\w\d\;]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectName>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<AccessList>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<AccessMask>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<DesiredAccess>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?(?=<)(?<Attributes>)|(?<Attribute>[^<]+))<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$2 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<AuthPackageName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$3 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\s\w\$]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$4 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+" \w+=\"(?<UnixUID>\d+)\" \w+=\"(?<UnixGID>\d+)\" \w+=\"(?<UnixIsLocal>\w+)\"><\/\w+><\w+\s\w+=\"\w+">(?<SubjectUserSid>[\w\-\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>[\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\$\w\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectServer>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectType>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<HandleID>[\w\d\;]+)<\/\w+><\w+\s\w+=\"\w+">(?<ObjectName>[^<]+)<\/\w+><\w+\s\w+=\"\w+">(?<InformationRequested>[^<]+)<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$5 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><\w+\s\w+=\"\w+">(?<IPPort>\d+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserSid>[\-\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserName>[\w\$\s]+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetUserIsLocal>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<TargetDomainName>\w+)<\/\w+><\w+\s\w+=\"\w+">(?<AuthenticationPackageName>[\w\d]+)<\/\w+><\w+\s\w+=\"\w+">(?<LogonType>\d+)
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$6 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><[^<]+<\/\w+><[\w\"\=\s]+>(?<SubjectUserSid>[\w\-]+)<\/\w+><[\w\"\=\s]+>(?<SubjectUserIsLocal>\w+)<\/\w+><[\w\"\=\s]+>(?<subjectDomainName>\w+)<\/\w+><[\w\"\=\s]+>(?<TargetUserName>[\w\_]+)<\/\w+><[\w\"\=\s]+>(?<ObjectServer>\w+)<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>(?<OldPath>[^<]+)<\/\w+><[\w\"\=\s]+>(?<NewPath>[^<]+)<\/\w+><[\w\"\=\s]+>(?<Attributes>)<\/\w+><\/\w+><\/\w+>
EXTRACT-Log Field for \\\\cdc-prod-nfs1\\log$7 = "(?<ProviderName>[\w\-]+)" \w+="{(?<Guid>[\-\w\d]+)}"\/><\w+>(?<EventID>[\d]+)<\/\w+><\w+>(?<EventName>[\w\s]+)<\/\w+><\w+>(?<Version>[\w\.]+)<\/\w+><\w+>(?<Source>\w+)<\/\w+><\w+>(?<Level>\d+)<\/\w+><\w+>(?<Opcode>\d+)<\/\w+><\w+>(?<Keywords>[\w\d]+)<\/\w+><\w+>(?<Result>\w+\s\w+)<\/\w+><\w+\s\w+="(?<CreatTime>[^"]+)\"\/><\w+\/><\w+>(?<Channel>\w+)<\/\w+><\w+>(?<Computer>[\w\-\/]+)<\/\w+><\w+>(?<ComputerUUID>[\w\-\d\/]+)<\/\w+><\w+\/><\/\w+><\w+><[\w\"=\s]+>(?<IPAddress>[\d\.]+)<\/\w+><[^<]+<\/\w+><[\w\"\=\s]+>(?<SubjectUserSid>[\w\-]+)<\/\w+><[\w\"\=\s]+>(?<SubjectUserIsLocal>\w+)<\/\w+><[\w\"\=\s]+>(?<subjectDomainName>\w+)<\/\w+><[\w\"\=\s]+>(?<TargetUserName>[\w\_]+)<\/\w+><[\w\"\=\s]+>(?<ObjectServer>\w+)<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>[\d\w\;]+<\/\w+><[\w\"\=\s]+>(?<ObjectName>[^<]+)<\/\w+><[\w\"\=\s]+>(?<WriteOffset>\d+)<\/\w+><[\w\"\=\s]+>(?<WriteCount>\d+)
Tags (2)
0 Karma

peterchenadded
Path Finder

Wow, probably better to try and convert the message into a proper XML message and have splunk automatically extract the tags for you.

You can then get rid of all the regex and setup field alises if you need the fields to be different names to the tags.

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...