topic Re: Why is the KV pair extraction with custom delimiters not working? in Getting Data In

Why is the KV pair extraction with custom delimiters not working?

splunked38 — Mon, 24 Jun 2019 14:03:22 GMT

Hello all,

I'm trying to get extraction to work on a dynamic key value log.

I've tried the following without any success (open to other suggestions away from this).

Ideally the output should be:

Thread=5\=/blah/blah
Method=GET
URI=/
Protocol=HTTP/1.1
IP=1.2.3.4
Port=54809
Referer=https://referrer
field=value
.
.
.
field=value

props.conf

[testsourcetype_log]
CHARSET=UTF-8
KV_MODE=none
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
category=Testing
description=Test KV log sourcetype
disabled=false
pulldown_type=true
REPORT-kv=kv_extraction
EXTRACT-status=^(\d{4}\-\d{2}\-\d{2}T\d{2}:\d{2}:\d{2})\s\[(?<status>\w+)

transforms.conf

[kv_extraction]
DELIMS = "]", ":"
MV_ADD=true

log snip:

2019-03-01T09:42:01 [status] [Thread: 5=/blah/blah] [Method: GET] [URI: /blah/blah]  [Protocol: HTTP/1.1] [IP: 1.2.3.4] [Port: 54809] [Referer: https://referrer] [..] ... [..] text string here

References:
https://www.splunk.com/blog/2008/02/12/delimiter-based-key-value-pair-extraction.html
https://answers.splunk.com/answers/170826/set-delimiter.html

Thanks in advance

UPDATE 6/25:
I've tried combinations from @FrankVl, @VatsalJagani, @woodcock but it seems none of them work.

Naturally, I've restarted splunk after each change. Here is the output from btool to show that I'm not going insane

/opt/splunk/bin/splunk cmd btool props list
[testsourcetype_log]
ADD_EXTRA_TIME_FIELDS = True
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG = /etc/datetime.xml
DEPTH_LIMIT = 1000
HEADER_MODE =
KV_MODE = none
LEARN_MODEL = true
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MATCH_LIMIT = 100000
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = false
TRANSFORMS =
TRANSFORMS-kv = kv_extraction
TRUNCATE = 10000
category = Testing
description = Test KV log sourcetype
detect_trailing_nulls = false
disabled = false
maxDist = 100
priority =
pulldown_type = true
sourcetype =

/opt/splunk/bin/splunk cmd btool transforms list
[kv_extraction]
CAN_OPTIMIZE = True
CLEAN_KEYS = True
DEFAULT_VALUE =
DEPTH_LIMIT = 1000
DEST_KEY =
FORMAT = $1::$2
KEEP_EMPTY_VALS = False
LOOKAHEAD = 4096
MATCH_LIMIT = 100000
MV_ADD = true
REGEX = \[([^:[]+):\s+([^\]]+)]
SOURCE_KEY = _raw
WRITE_META = False

Updated props.conf

[testsourcetype_log]
CHARSET=UTF-8
KV_MODE=none
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
category=Testing
description=Test KV log sourcetype
disabled=false
pulldown_type=true
TRANSFORMS-kv=kv_extraction

updated transforms.conf

[kv_extraction]
REGEX = \[([^:[]+):\s+([^\]]+)]
FORMAT = $1::$2
MV_ADD=true

UPDATE 6/27:

Using a clean splunk docker image, I:

recreated indexers, inputs, props, transforms on the docker instance (external volume)
stripped those files to a bare minimum
renamed the sourcetype (to be sure that Splunk is reading props/transforms)
moved the configs from being inside an app to system/local/*.conf
checked the knowledge object existence via the gui (new/renamed transform is listed)
checked the knowledge object permissions (global)
and restarted after each change

nada, log is being ingested but no new fields created (except for the value of thread that is field: 5 value: /blah/blah/)

current config:

$ cat system/local/inputs.conf
[default]
host = 7278c011e1e0

[monitor:///opt/splunk/var/log/testlogs/*.log]
disabled=false
sourcetype=blahblah
index = testindex

$ cat system/local/props.conf
[blahblah]
CHARSET=UTF-8
KV_MODE=none
NO_BINARY_CHECK=true
SHOULD_LINEMERGE=false
category=Testing
description=Test KV log sourcetype access
disabled=false
pulldown_type=true
TRANSFORMS-blahkv=blahkvextraction
#TRANSFORMS-replace_source = replacedefaultsource2

$ cat system/local/transforms.conf
[blahkvextraction]
FORMAT = $1::$2
MV_ADD = 1
#REGEX = \[([^:[]+):\s+([^\]]+)]
REGEX = \[([^:\]]+):\s+([^\]]+)\]

BTW: for @FrankVl, @VatsalJagani, @woodcock, thanks. I have used iterations of each of your code and strongly believe that it works. I've done variations of the below to prove that your solutions work and it does (I get one instance of field1=Thread, field2= value😞

index=testindex sourcetype=blahblah
| rex field=_raw "\[(?<field1>[^:\]]+):\s+(?<field2>[^\]]+)\]"

Re: Why is the KV pair extraction with custom delimiters not working?

VatsalJagani — Mon, 24 Jun 2019 14:13:50 GMT

Can you try: DELIMS = "] [", ": "?

Re: Why is the KV pair extraction with custom delimiters not working?

FrankVl — Mon, 24 Jun 2019 14:37:17 GMT

Really curious if that works, because I don't think DELIMS in transforms.conf is intended to contain multi-character delimiter strings. Each character is interpreted on its own as a delimeter and especially with the space occuring in both delimiters, I kind of expect this will fail.

Re: Why is the KV pair extraction with custom delimiters not working?

VatsalJagani — Mon, 24 Jun 2019 14:40:07 GMT

Yeah right my bad, it will not work. Instead use REGEX from transforms as suggested in answer.

Re: Why is the KV pair extraction with custom delimiters not working?

FrankVl — Mon, 24 Jun 2019 14:42:30 GMT

If you can't get it working with the DELIMS suggestion from @VatsalJagani then try it using REGEX:

[kv_extraction]
REGEX = \[([^:\]]+):\s+([^\]]+)\]
FORMAT = $1::$2
MV_ADD=true

See: https://regex101.com/r/PDHjzk/1
Note: this assumes fieldnames do not contain : or ] and field values do not contain ].

Re: Why is the KV pair extraction with custom delimiters not working?

VatsalJagani — Mon, 24 Jun 2019 14:43:54 GMT

Hello @splunked38,

Please give a shot to below transforms instead of DELIMS.

[kv_extraction]
REGEX = \[([^:\[]+):\s+([^\]]+)\]
FORMAT = $1::$2

Hope this helps!!!

Re: Why is the KV pair extraction with custom delimiters not working?

splunked38 — Mon, 24 Jun 2019 18:38:27 GMT

I tried @VatsalJagani's example but it didn't work (then again, it could other things as well)

Re: Why is the KV pair extraction with custom delimiters not working?

woodcock — Mon, 24 Jun 2019 19:50:45 GMT

Try this in transforms.conf:

[kv_extraction]
REGEX = \[([^:[]+):\s+([^\]]+)]
FORMAT = $1::$2
MV_ADD = true

Re: Why is the KV pair extraction with custom delimiters not working?

VatsalJagani — Tue, 25 Jun 2019 03:44:30 GMT

Try transform that I've given in my answer if that is not working then comment further issues that you are having and we'll debug issues further.
Also, what do you mean by "it could other things as well", please describe.

Re: Why is the KV pair extraction with custom delimiters not working?

FrankVl — Tue, 25 Jun 2019 06:54:48 GMT

You're missing the : between key and value. Also: using *? is typically not the best performing construct and is best avoided if it is possible to just use a more specific regex that doesn't require backtracking.

Re: Why is the KV pair extraction with custom delimiters not working?

VatsalJagani — Tue, 25 Jun 2019 07:06:33 GMT

Yeah agree with you @FrankVI. Here is the well performing regex \[([^:]+):\s*([^\]]+)\].

Re: Why is the KV pair extraction with custom delimiters not working?

FrankVl — Tue, 25 Jun 2019 07:20:34 GMT

With still one small mistake (I made that myself as well initially). Have a look at what this regex does with the provided sample event: https://regex101.com/r/NtE461/1

That is why I added a \] in the character class of the first capture group. @woodcock solved it by adding a [ in there, which has roughly the same effect.

Re: Why is the KV pair extraction with custom delimiters not working?

VatsalJagani — Tue, 25 Jun 2019 07:25:26 GMT

Yeah right, to be on safer side I always add \ before all special characters in regex. In our new regex is using only 87 steps to complete with sample event.

Re: Why is the KV pair extraction with custom delimiters not working?

FrankVl — Tue, 25 Jun 2019 07:41:04 GMT

My point was that it trips over the [status] part when using your regex. It takes status] [Thread as the first fieldname.

Re: Why is the KV pair extraction with custom delimiters not working?

VatsalJagani — Tue, 25 Jun 2019 08:32:56 GMT

This regex is working for me - \[([^:\[]+):\s+([^\]]+)\].

Re: Why is the KV pair extraction with custom delimiters not working?

FrankVl — Tue, 25 Jun 2019 08:41:44 GMT

Yes, adding the \[ to the first capture group's negative character set solves it indeed 🙂

Re: Why is the KV pair extraction with custom delimiters not working?

FrankVl — Tue, 25 Jun 2019 11:29:36 GMT

Commenting on the update:
Have you deployed the config on the searchhead(s)? Did you confirm from the GUI (settings - fields ->...) that the TRANSFORMS-kv is indeed enabled and shared Global?

Re: Why is the KV pair extraction with custom delimiters not working?

woodcock — Tue, 25 Jun 2019 15:14:13 GMT

Mine definitely works on your sample data set; see here:
https://regex101.com/r/s0ACKL/1

Re: Why is the KV pair extraction with custom delimiters not working?

woodcock — Tue, 25 Jun 2019 15:14:55 GMT

Mine definitely works on your sample data set; see here:
https://regex101.com/r/s0ACKL/1

Re: Why is the KV pair extraction with custom delimiters not working?

splunked38 — Wed, 26 Jun 2019 10:44:00 GMT

I initially installed this on a standalone SH, btool output that I posted says that the config is loaded.

Having said that, the props/transforms is part of an app not system wide. so it may be the problem.

I'm currently standing up a clean SH/indexer instance to retest and check the perms. I'll report back shortly.