Splunk Search

How to extract field with it's value enclosed by square bracket

Stan
New Member

When parsing some customized log, the format it's like below

[timestamps] field name [value]
[00:46:38]  - Remain Queue      [          0 ]
[00:46:38]  - Remain Queue      [          2 ]

The only search term works for me is "Remain Queue" NOT 0. I've tried "Remain Queue">0 but no luck. How can I search/sort the remain queue count? How to train splurk to know Remain Queue is a field?

Further more, I've got another log entry looks like below. Is it possible to parse it if I want key-value pair become Connect=93?

[00:46:38]  - Connect           [     330931 /     330838 ]     [         93 ]

Thanks.

Tags (2)
0 Karma
1 Solution

Johnvey
Contributor

Define your field extraction via a regex:

^\[[^\]]+\]\s+\-\s*([\w ]+).+\[\s*([^\s]+)\s*\]

This is the format that you would add to props.conf, where the matched groups $1 and $2 will extract the field data you want.

You can also try this inline via the rex command (as to avoid editing conf files):

... | rex "^\[[^\]]+\]\s+\-\s*(?<action>[\w ]+).+\[\s*(?<value>[^\s]+)\s*\]" | search action="Remain Queue" value>=10

where the named extractions will generate a action and value field. The subsequent search command then uses the newly extracted fields. This regex works on both examples you provided:

[00:46:38]  - Remain Queue      [          0 ]
[00:46:38]  - Remain Queue      [          2 ]
[00:46:38]  - Connect           [     330931 /     330838 ]     [         93 ]

You can test out the regexes online at a regex testing page.

View solution in original post

Johnvey
Contributor

Define your field extraction via a regex:

^\[[^\]]+\]\s+\-\s*([\w ]+).+\[\s*([^\s]+)\s*\]

This is the format that you would add to props.conf, where the matched groups $1 and $2 will extract the field data you want.

You can also try this inline via the rex command (as to avoid editing conf files):

... | rex "^\[[^\]]+\]\s+\-\s*(?<action>[\w ]+).+\[\s*(?<value>[^\s]+)\s*\]" | search action="Remain Queue" value>=10

where the named extractions will generate a action and value field. The subsequent search command then uses the newly extracted fields. This regex works on both examples you provided:

[00:46:38]  - Remain Queue      [          0 ]
[00:46:38]  - Remain Queue      [          2 ]
[00:46:38]  - Connect           [     330931 /     330838 ]     [         93 ]

You can test out the regexes online at a regex testing page.

Johnvey
Contributor

Ah, you need to add another search or where command after the rex because the field will only come into being after it, like: * | rex ... | where action="Remain Queue" value>10. I've updated the example.

0 Karma

Stan
New Member

The other question is there're so many props.conf files in splunk folder. Which one should I modify. And which section and what key should I add in?
eg.
[SECTION NAME]
KEY = ^[[^]]+]\s+-\s*([\w ]+).+[\s*([^\s]+)\s*]

0 Karma

Stan
New Member

Thanks for the awesome solution. But I am too new to get this work. I tried to search with '"Remain Queue" > 10 | rex "^[[^]]+]\s+-\s*(?[\w ]+).+[\s*(?[^\s]+)\s*]" but no luck. Even remove double quotes from Remain Queue still doesn't work. Why?

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...