Select the contents of the key-values fields from the string

[mbabakov]

Hi!

There is a log with such records:

Dec 17 10:08:38 10.52.137.1 Apr  3 22:46:57 2012 930-RTR-944 %%10SSH/6/SSH_LOGIN(l): -DevIP=10.52.137.1; STEL user monitor (IP: 192.168.181.94) logged in successfully.
Dec 17 10:08:38 10.52.137.1 Apr  3 22:46:57 2012 930-RTR-944 %%10SC/6/SC_AAA_SUCCESS(l): -DevIP=10.52.137.1-AAAType=ACCOUNT-AAAScheme= local-Service=login-UserName=monitor@system; AAA is successful.
Dec 17 10:08:38 10.52.137.1 Apr  3 22:46:57 2012 930-RTR-944 %%10SC/6/SC_AAA_SUCCESS(l): -DevIP=10.52.137.1-AAAType=AUTHEN-AAAScheme= hwtacacs-scheme tacacs-Service=login-UserName=monitor@system; AAA is successful.
Dec 17 10:08:13 10.98.171.65 Jan 20 00:00:17 2011 MSR954-RTR-LTE-5686 %%10CELLULAR/5/CELLULAR: -DevIP=10.98.171.65;   Controller Cellular1/0: The network connection switched to 3G.
Dec 17 10:08:04 10.199.69.26 May 23 21:50:30 2012 930-RTR-14815 %%10SSH/4/TrapLogoff(t):  1.3.6.1.4.1.25506.2.22.1.3.0.4 SSH user logoff trap information

It is necessary to parse the fields between the characters "-" and ";". For example:

-DevIP = 10.52.137.1;
-DevIP = 10.52.137.1-AAAType = ACCOUNT-AAAScheme = local-Service = login-UserName = monitor@system;
-DevIP = 10.52.137.1-AAAType = AUTHEN-AAAScheme = hwtacacs-scheme tacacs-Service = login-UserName = monitor@system;

The difficulty is that the symbol "-" is used as a separator between key-value pairs, which is also present in the value: "AAAScheme = hwtacacs-scheme tacacs".

I used EVAL to replace "-" with "|":

EVAL-cmd_params_src = replace (cmd_params_src, "- (\ p {Lu})", "| \ 1")

And I see:

cmd_params_src = "DevIP = 10.52.137.1 | AAAType = AUTHEN | AAAScheme = hwtacacs-scheme tacacs | Service = login | UserName = monitor @ system"

But fields are not translated to key value.

props.conf

[hp_routers]
EVAL-vendor = "HP"
KV_MODE = none

REPORT-fields_general = extract_hp_route_general
REPORT-fields_cmd_parameters = extract_hp_route_cmd_parameters
REPORT-fields_cmd_message = extract_hp_route_cmd_message
EVAL-cmd_params_src = replace(cmd_params_src, "-(\p{Lu})", "|\1")
REPORT-field_params = extracet_field_from_params

transforms.conf

[extract_hp_route_general]
REGEX = ^(?P<Date>\w+\s+\d+\s+\d+:\d+:\d+)\s+(?P<device_ip>[^ ]+)\s+(?P<device_time>\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)[^ \n]* (?P<hostname>[^ ]+)[^ \n]* \%\%10(?<cmd_grp>\w+)\/$

[extract_hp_route_cmd_parameters]
REGEX = (?:\s-)(?<cmd_params_src>.+)(?:;)
disabled = 0

[extract_hp_route_cmd_message]
REGEX = (?:;\s+)(?<cmd_msg>.+)$

[extracet_field_from_params]
REGEX = (\w+)=(\s?[.\-\s@\w]+)
FORMAT = $1::$2
SOURCE_KEY = cmd_params_src
#DELIMS = "|", "="
REPEAT_MATCH = True
CLEAN_KEYS = 1

What am I doing wrong, how to fix the situation?

regards
Michael

[woodcock · Accepted solution]

To set this up to be automatic, like this:

In props.conf:

[hp_routers]
KV_MODE = none
REPORT-HP_KVPs = HP_KVPs

In transforms.conf:

[HP_KVPs]
REGEX = ([^-=]+)=([^=]*?)(?=$|-[^-=]+=)
FORMAT = $1::$2
REPEAT_MATCH = true

The key is to do your own KVP decoding and to use the RegEx positive lookahead feature to ensure that no keys (only values) can contain hyphens.

[mbabakov]

Thank you!

[woodcock]

Like this:

| makeresults 
| eval raw="Dec 17 10:08:38 10.52.137.1 Apr  3 22:46:57 2012 930-RTR-944 %%10SSH/6/SSH_LOGIN(l): -DevIP=10.52.137.1; STEL user monitor (IP: 192.168.181.94) logged in successfully.:::Dec 17 10:08:38 10.52.137.1 Apr  3 22:46:57 2012 930-RTR-944 %%10SC/6/SC_AAA_SUCCESS(l): -DevIP=10.52.137.1-AAAType=ACCOUNT-AAAScheme= local-Service=login-UserName=monitor@system; AAA is successful.:::Dec 17 10:08:38 10.52.137.1 Apr  3 22:46:57 2012 930-RTR-944 %%10SC/6/SC_AAA_SUCCESS(l): -DevIP=10.52.137.1-AAAType=AUTHEN-AAAScheme= hwtacacs-scheme tacacs-Service=login-UserName=monitor@system; AAA is successful.:::Dec 17 10:08:13 10.98.171.65 Jan 20 00:00:17 2011 MSR954-RTR-LTE-5686 %%10CELLULAR/5/CELLULAR: -DevIP=10.98.171.65;   Controller Cellular1/0: The network connection switched to 3G.:::Dec 17 10:08:04 10.199.69.26 May 23 21:50:30 2012 930-RTR-14815 %%10SSH/4/TrapLogoff(t):  1.3.6.1.4.1.25506.2.22.1.3.0.4 SSH user logoff trap information" 
| makemv delim=":::" raw 
| mvexpand raw 
| rename raw AS _raw
| eval _time = strptime(_raw, "%b %d %H:%M:%S")

| rename COMMENT AS "Everything above generates sample events; everything below is your solution"

| rex max_match=0 "(?<key>[^-=]+)=(?<value>[^=]*?)(?=$|-[^-=]+=)"
| eval _raw=mvzip(key, value, "=")
| fields - key value
| kv

[woodcock]

The key is to do your own KVP decoding and to use the RegEx positive lookahead feature to ensure that no keys (only values) can contain hyphens.

[mbabakov]

Thanks for your solution!
I wanted the analysis to go at the indexing stage. But in the end agreed to parse the search stage. 🙂

[woodcock]

Right, I forgot that. See my other answer.

[woodcock]

I think you clicked Accept on the wrong one, right?

[mbabakov]

Sorry, yes. Corrected

[gcusello]

Hi mbabakov,
can you use "DevIP" that is after "-"?
If yes, you can try (see on regex101.com at https://regex101.com/r/2cwhUx/1 )

-(?<my_field>DevIP[^;]*)

If instead you're not sure about the word "DevIP" but you're sure that there's the equal after the first word, you could use something like this

-(?<my_field>[^\=]*\=[^;]*)

Bye.
Giuseppe

[mbabakov]

Sorry, but I need to work out all possible lines between "-" and ";". For example:
-DevIP = 10.52.137.1;
-DevIP = 10.52.137.1-AAAType = ACCOUNT-AAAScheme = local-Service = login-UserName = monitor@system;
-DevIP = 10.52.137.1-AAAType = AUTHEN-AAAScheme = hwtacacs-scheme tacacs-Service = login-UserName = monitor@system;

A special case of "-DevIP = 10.52.137.1;", this is very simple. 🙂

[gcusello]

infact, as you can see at https://regex101.com/r/2cwhUx/1 using this regex, you take all between "-" and the first ";".
If you have more ";", you can use a similar regex.
Bye.
Giuseppe

[gcusello]

Sorry for an additional information:
if you have "-DevIP = 10.52.137.1-AAAType = ACCOUNT-AAAScheme = local-Service = login-UserName = monitor@system;" do you want separately each field?
something like

DevIP = 10.52.137.1
AAAType = ACCOUNT
AAAScheme = local
Service = login
UserName = monitor@system

if this is your requirement, see https://regex101.com/r/2cwhUx/2 and try

-(?<my_field>\w*\=[^;-]*)

Bye.
Giuseppe

[mbabakov]

Yes, I need to separate each field separately. And I do not know how many of these fields will be in different records and how they will be called.
Therefore, I need a construction of the form (? <_ KEY_1> [^ =] [\ p {Lu} \ p {Lt}]. +) = (? <_ VAL_1> [^ |]. +). But, when using the separator "-" between the pairs, the regular expression becomes very complicated, since the "-" symbol can be used in the Value field.
I used EVAL to replace "-" with "|", but I don’t know how in this case to force splunk to parse all key-value pairs.

For example https://regex101.com/r/tlaRxc/1.

[mbabakov]

To clarify, I am interested in the analysis of all possible fields in rows in the log, for example:
-DevIP = 10.52.137.1;
-DevIP = 10.52.137.1-AAAType = ACCOUNT-AAAScheme = local-Service = login-UserName = monitor@system;
-DevIP = 10.52.137.1-AAAType = AUTHEN-AAAScheme = hwtacacs-scheme tacacs-Service = login-UserName = monitor@system;

Those. for the first line is DevIP. For the second, these are DevIP, AAAType, AAAScheme, Service, UserName, etc.