How do I extract this field from my sample Meraki Flow syslog events to use in search?

[randombuffalo]

I am having trouble using a field that is in my log entries, but Splunk doesn't "auto-discover" it when I started indexing data. I have played around with using the "extract new fields", but can't get the field to work right. The field name is "pattern:" in the log entries. I would like to be able to search based on the value of "pattern:" in the entries. I believe the only values it can have in the logs are: 0, 1, allow all.

Here are some examples:

Dec 15 11:38:54 10.1.1.3 1 1481823534.895775102 HOST_NAME flows src=10.1.1.251 dst=23.23.23.23 mac=C4:71:FE:EE:EE:EE protocol=tcp sport=50814 dport=80 pattern: allow all

Dec 15 11:38:54 10.1.1.3 1 1481823535.012274548 HOST_NAME flows src=111.111.111.111 dst=111.111.111.111 protocol=tcp sport=10155 dport=443 pattern: 0 tcp  && dst port 443 && dst 111.111.111.111

Dec 15 11:41:07 10.1.1.3 1 1481823667.556427731 HOST_NAME flows src=111.111.111.111 dst=111.111.111.111 protocol=tcp sport=46078 dport=23 pattern: 1 all

Dec 15 11:42:00 10.1.1.3 1 1481823667.556427731 HOST_NAME flows src=111.111.111.111 dst=111.111.111.111 protocol=tcp sport=46078 dport=23 pattern: 1

As you can see the pattern: field can have text or a numeric value if that helps narrow down the issue. Any help would be greatly appreciated. I am new to Splunk and have looked around, but I might not even know the terminology to use to search properly.

[kumargaurav]

simply rexing out the field you need will work for you.You will have to just rex out your field from the field.

[somesoni2]

Give this a try

Your base search | rex "pattern: (?<pattern>(\d+|\w+\s*\w+|\w+))"

Run anywhere search with your sample data

| gentimes start=-1 | eval raw="Dec 15 11:38:54 10.1.1.3 1 1481823534.895775102 HOST_NAME flows src=10.1.1.251 dst=23.23.23.23 mac=C4:71:FE:EE:EE:EE protocol=tcp sport=50814 dport=80 pattern: allow all##Dec 15 11:38:54 10.1.1.3 1 1481823535.012274548 HOST_NAME flows src=111.111.111.111 dst=111.111.111.111 protocol=tcp sport=10155 dport=443 pattern: 0 tcp  && dst port 443 && dst 111.111.111.111##Dec 15 11:41:07 10.1.1.3 1 1481823667.556427731 HOST_NAME flows src=111.111.111.111 dst=111.111.111.111 protocol=tcp sport=46078 dport=23 pattern: 1 all##Dec 15 11:42:00 10.1.1.3 1 1481823667.556427731 HOST_NAME flows src=111.111.111.111 dst=111.111.111.111 protocol=tcp sport=46078 dport=23 pattern: 1" | table raw | makemv raw delim="##" | mvexpand raw | rename raw as _raw 
| rex "pattern: (?<pattern>(\d+|\w+\s*\w+|\w+))"

[randombuffalo]

Thanks for the input but I think I have my question answered above.

[tgow · Accepted solution]

Wondering if you start at the end then work backwards. Does this work:

pattern:\s?(?<pattern>[^$]+)

[randombuffalo]

Yes that worked! I added your code to my field extraction and it looks like it is picking it up correctly now! Thanks for the help.

[richgalloway]

Consider this REGEX as a starting place for a parser for your events.

\w{3} \d{1,2} \d\d:\d\d:\d\d (?<ip>[^ ]+) \d (?<field>[^ ]+) (?<host_name>[^ ]+) flows src=(?<src>[^ ]+) dst=(?<dst>[^ ]+) mac=(?<mac>[^ ]+) protocol=(?<protocol>[^ ]+) sport=(?<sport>[^ ]+) dport=(?<dport>[^ ]+) pattern: (?<pattern>.*)

[randombuffalo]

Rich, the other fields are picked up fine by Splunk. Would this REGEX mess up those existing extractions? This is what Splunk gave me when I tried to go through the extraction wizard:

 ^(?:[^=\n]*=){5}\d+\s+(?P<Pattern>[^ ]+)

But that doesn't seem to work and ends up including the wrong parts of the log entry as values. As shown in this picture:

[richgalloway]

Splunk will auto-discover fields in the name=value pattern, which most of your event uses. The 'pattern' field is different, which is why Splunk is not discovering it. Do you have the ability to change the logging?

[randombuffalo]

Rich, yes the other fields are being picked up fine. Unfortunately for this case, Meraki does not allow changing the output of the log entries. Not sure why they decided to have that one field formatted differently.