topic Regex for items in single quotes in Splunk Search

Regex for items in single quotes

aramakrishnan — Mon, 01 May 2017 21:30:00 GMT

I have a bunch of logs which look like the following.

2017-05-01 18:36:16,885 UTC - DEBUG - testpod_service.segment_processor - [BTESTC4277:37223] returning requests="[{'status': 'request_automat_eventab', 'priority': -2868.565, 'confidence': 1.0, 'db': {'dur_sec': 2868.565, 'n_hours': 2}, 'onset': 66377515, 'event_name': 'CYCLE', 'offset': 66951227

For every segment that looks like 'field':'value'I want Splunk to treat the value in quotes BEFORE the : as the field name and the value in quotes AFTER the : as the value for the field (similar treatment to field="value", which is Splunk's default).

How do I achieve this through a regex treatment? For example, in the example above, I'd like my field and value pairs to be as follows:

status = request_automat_eventab
priority = -2868.565
confidence = 1.0
dur_sec=2868.565
n_hours=2

...and so on

Thanks in advance!

Re: Regex for items in single quotes

DalJeanis — Tue, 02 May 2017 00:22:43 GMT

Are you asking about at index time or at search time?

Re: Regex for items in single quotes

adonio — Tue, 02 May 2017 00:25:29 GMT

use the field extractor from GUIi? click an event -> actions -> extract fields -> new tab pops -> follow the prompts -> save -> search your fields
read more here:
http://docs.splunk.com/Documentation/Splunk/6.5.3/Knowledge/ExtractfieldsinteractivelywithIFX

Re: Regex for items in single quotes

kmorris_splunk — Tue, 29 Sep 2020 13:54:26 GMT

If the field / value format is consistent, you can use <_KEY_1> and <_VAL_1> as your field names in your capture groups to dynamically extract all field value pairs. In your props.conf, under the sourcetype for these events add a field extraction like the following:

[yoursourcetypehere]
EXTRACT-keysandvals = \"\[{\'(?<_KEY_1>\S+)\': \'(?<_VAL_1>\S+)\'

You may have to play with the regex a bit, but this should work. It will take everything in single quotes before the colon as a field name and everything after the colon in single quotes as the value for that field. You can use this any time you have a consistent format for the pairs.

I had a customer example where the field was in brackets followed by the value:

[fieldname1]fieldvalue1[fieldname2]fieldvalue2

Hope this helps.

Re: Regex for items in single quotes

kmorris_splunk — Tue, 29 Sep 2020 13:54:29 GMT

You can use the _KEY_1 and _VAL_1 field names in your regex capture groups to get all of the key value pairs dynamically. For example, in your props.conf, under the sourcetype for these events:

[yoursourcetype]
EXTRACT-keysandvalues = \"\[{\'(?<_KEY_1>\S+)\': \'(?<_VAL_1>\S+)\'

This will take every value in single quotes before the colon as the field name and the value in single quotes after the colon as the value for that field. This will work for any format you can match in the extractions regex. As long as the key value pairs are consistent, it will grab them all.

Hope this helps.