topic Re: Need help with Regex in Splunk Search

Need help with Regex

bhargavi — Wed, 17 Nov 2021 19:57:32 GMT

Hello all, kindly help with Regex..

I am seeing the below messages in splunkd logs. Though values are actually being extracted properly, below messages are annoying and I want to get rid of those. Need urgent help to construct a better Regex to avoid these messages. I have increased MATCHLIMIT in transforms.conf, but still seeing these messages.

11-17-2021 17:12:34.927 +1100 ERROR Regex - Failed in pcre_exec: Error PCRE_ERROR_MATCHLIMIT for regex
11-17-2021 17:12:34.927 +1100 WARN regexExtractionProcessor - Regular expression for stanza security_index exceeded configured PCRE match limit. One or more fields might not have their values extracted, which can lead to incorrect search results. Fix the regular expression to improve search performance or increase the MATCH_LIMIT in props.conf to include the missing field extractions.

Part of my regex looks like this. I have repeated the same regex corresponding to different VM and file combinations. Can someone please help with a better Regex.

My Sample event is as below. There are more than 10 type of VM's (like ops,db,sync,etc), more than 300 VM's and 250 different files and I made a Single regex. I need to create regex considering below two criteria:

1) The VM's : "xyz001vm002400-ops-vm01","xyz002vm002400-db-vm01", etc.
2) Log file beginning with a class keyword : "class":"/var/log/syslog.log , "class":"/opt/cat/audit.log , etc.

Sample Event 1:

{"VNFType":"xyz","VNFs":"xyz001vm002400","VNFCType":"ops","VNFCs":"xyz001vm002400-ops-vm01","event":{"log_message":"2021-11-18T02:45:01.085777+10:00, xyz001vm002400, xyz001vm002400-ops-vm01, ops, ops, info, cron, xyz001vm002400-ops-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz001vm002400, xyz001vm002400-ops-vm01, ops, ops, info, cron, xyz001vm002400-ops-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz001vm002400, xyz001vm002400-ops-vm01, ops, ops, info, cron, xyz001vm002400-ops-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz001vm002400, xyz001vm002400-ops-vm01, ops, ops, info, cron, xyz001vm002400-ops-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz001vm002400, xyz001vm002400-ops-vm01, ops, ops, info, cron, xyz001vm002400-ops-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz001vm002400, xyz001vm002400-ops-vm01, ops, ops, info, cron, xyz001vm002400-ops-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n","class":"/var/log/syslog.log","log_event_time_stamp":"2021-11-18T02:45:08+10:00"}}

Sample Event 2:

{"VNFType":"xyz","VNFs":"xyz002vm002400","VNFCType":"db","VNFCs":"xyz002vm002400-db-vm01","event":{"log_message":"2021-11-18T02:45:01.085777+10:00, xyz002vm002400, xyz002vm002400-db-vm01, db, db, info, cron, xyz002vm002400-db-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz002vm002400, xyz002vm002400-db-vm01, db, db, info, cron, xyz002vm002400-db-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz002vm002400, xyz002vm002400-db-vm01, db, db, info, cron, xyz002vm002400-db-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz002vm002400, xyz002vm002400-db-vm01, db, db, info, cron, xyz002vm002400-db-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz002vm002400, xyz002vm002400-db-vm01, db, db, info, cron, xyz002vm002400-db-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz002vm002400, xyz002vm002400-db-vm01, db, db, info, cron, xyz002vm002400-db-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n","class":"/opt/cat/audit.log","log_event_time_stamp":"2021-11-18T02:45:08+10:00"}}

@chrisyounger @harsmarvania57 - Any help is much appreciated.

Re: Need help with Regex

richgalloway — Wed, 17 Nov 2021 22:02:42 GMT

What is the goal of the regex? Is it to extract all VM names or just the one following "VNFs"? Is there a pattern to VM names (perhaps xxxnnnVMnnnnnn-xx-xxnn)?

Please share the transforms.conf settings.

Re: Need help with Regex

bhargavi — Thu, 18 Nov 2021 05:20:41 GMT

Hi @richgalloway

My goal to filter the events according to the VM types (like db, ops,telemetry,etc) and log file combinations and route it to different indexes.

For example, /opt/tomcat/server/cm.log might be coming from all type of VM's (around 12 VM types; VNFCs mentioned in bracket), but I want to filter the events only from few VM types, like opslb,ntfsync,telemetry,diagnostic and index them and ignore logs from other VM types.

VM name pattern would be like this as mentioned in the sample events : xyz002vm002400-db-vm01.xxxx.com.yy

A glimpse of transforms.conf

Sample Event 1: {"VNFType":"xyz","VNFs":"xyz002vm002400","VNFCType":"db","VNFCs":"xyz002vm002400-db-vm01.xxxx.com.yy","event":{"log_message":"2021-11-18T02:45:01.085777+10:00, xyz002vm002400, xyz002vm002400-db-vm01, db, db, info, cron, xyz002vm002400-db-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz002vm002400, xyz002vm002400-db-vm01.xxxx.com.yy, db, db, info, cron, xyz002vm002400-db-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz002vm002400, xyz002vm002400-db-vm01.xxxx.com.yy, db, db, info, cron, xyz002vm002400-db-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz002vm002400, xyz002vm002400-db-vm01, db, db, info, cron, xyz002vm002400-db-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz002vm002400, xyz002vm002400-db-vm01.xxxx.com.yy, db, db, info, cron, xyz002vm002400-db-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, xyz002vm002400, xyz002vm002400-db-vm01, db, db, info, cron, xyz002vm002400-db-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n","class":"/var/log/audit.log","log_event_time_stamp":"2021-11-18T02:45:08+10:00"}}

Sample Event 2:

{"VNFType":"xyz","VNFs":"abc002vm002400","VNFCType":"telemetry","VNFCs":"abc002vm002400-telemetry-vm01.xxxx.com.yy","event":{"log_message":"2021-11-18T02:45:01.085777+10:00, abc002vm002400, abc002vm002400-telemetry-vm01, telemetry, telemetry, info, cron, abc002vm002400-telemetry-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, abc002vm002400, abc002vm002400-telemetry-vm01.xxxx.com.yy, telemetry, telemetry, info, cron, abc002vm002400-telemetry-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, abc002vm002400, abc002vm002400-telemetry-vm01.xxxx.com.yy, telemetry, telemetry, info, cron, abc002vm002400-telemetry-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, abc002vm002400, abc002vm002400-telemetry-vm01, telemetry, telemetry, info, cron, abc002vm002400-telemetry-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, abc002vm002400, abc002vm002400-telemetry-vm01.xxxx.com.yy, telemetry, telemetry, info, cron, abc002vm002400-telemetry-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n2021-11-18T02:45:01.085777+10:00, abc002vm002400, abc002vm002400-telemetry-vm01.xxxx.com.yy, telemetry, telemetry, info, cron, abc002vm002400-telemetry-vm01, CROND[16311]:, (root) CMD (/usr/sbin/KillIdleSessions)\n","class":"/opt/log/local1.log","log_event_time_stamp":"2021-11-18T02:45:08+10:00"}}

Re: Need help with Regex

richgalloway — Thu, 18 Nov 2021 13:55:16 GMT

This regex will extract the VM name that follows "VNFCs" as well as the class name.

\VNFCs":"([^"]+).*class":"([^"]+)

To extract the specific VM and class names, this should work. Note the changes in how the escape character is used - slashes must be escaped, but most other special character do not.

Re: Need help with Regex

bhargavi — Thu, 18 Nov 2021 19:55:27 GMT

Thank you very much for the reply @richgalloway

I tried updating regex in the below pattern mentioned by you,but still receiving error messages in splunkd logs.

Re: Need help with Regex

richgalloway — Thu, 18 Nov 2021 20:51:27 GMT

I'm not sure what else to suggest. That regex is about 20x more efficient than the original. If Splunk still isn't happy then perhaps you should consider using multiple REGEX attributes, one for each pipe in the current regex.