- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![mbabakov mbabakov](https://community.splunk.com/legacyfs/online/avatars/549848.jpg)
Hi!
There is a log with such records:
Dec 17 10:08:38 10.52.137.1 Apr 3 22:46:57 2012 930-RTR-944 %%10SSH/6/SSH_LOGIN(l): -DevIP=10.52.137.1; STEL user monitor (IP: 192.168.181.94) logged in successfully.
Dec 17 10:08:38 10.52.137.1 Apr 3 22:46:57 2012 930-RTR-944 %%10SC/6/SC_AAA_SUCCESS(l): -DevIP=10.52.137.1-AAAType=ACCOUNT-AAAScheme= local-Service=login-UserName=monitor@system; AAA is successful.
Dec 17 10:08:38 10.52.137.1 Apr 3 22:46:57 2012 930-RTR-944 %%10SC/6/SC_AAA_SUCCESS(l): -DevIP=10.52.137.1-AAAType=AUTHEN-AAAScheme= hwtacacs-scheme tacacs-Service=login-UserName=monitor@system; AAA is successful.
Dec 17 10:08:13 10.98.171.65 Jan 20 00:00:17 2011 MSR954-RTR-LTE-5686 %%10CELLULAR/5/CELLULAR: -DevIP=10.98.171.65; Controller Cellular1/0: The network connection switched to 3G.
Dec 17 10:08:04 10.199.69.26 May 23 21:50:30 2012 930-RTR-14815 %%10SSH/4/TrapLogoff(t): 1.3.6.1.4.1.25506.2.22.1.3.0.4 SSH user logoff trap information
It is necessary to parse the fields between the characters "-" and ";". For example:
-DevIP = 10.52.137.1;
-DevIP = 10.52.137.1-AAAType = ACCOUNT-AAAScheme = local-Service = login-UserName = monitor@system;
-DevIP = 10.52.137.1-AAAType = AUTHEN-AAAScheme = hwtacacs-scheme tacacs-Service = login-UserName = monitor@system;
The difficulty is that the symbol "-" is used as a separator between key-value pairs, which is also present in the value: "AAAScheme = hwtacacs-scheme tacacs".
I used EVAL to replace "-" with "|":
EVAL-cmd_params_src = replace (cmd_params_src, "- (\ p {Lu})", "| \ 1")
And I see:
cmd_params_src = "DevIP = 10.52.137.1 | AAAType = AUTHEN | AAAScheme = hwtacacs-scheme tacacs | Service = login | UserName = monitor @ system"
But fields are not translated to key value.
props.conf
[hp_routers]
EVAL-vendor = "HP"
KV_MODE = none
REPORT-fields_general = extract_hp_route_general
REPORT-fields_cmd_parameters = extract_hp_route_cmd_parameters
REPORT-fields_cmd_message = extract_hp_route_cmd_message
EVAL-cmd_params_src = replace(cmd_params_src, "-(\p{Lu})", "|\1")
REPORT-field_params = extracet_field_from_params
transforms.conf
[extract_hp_route_general]
REGEX = ^(?P<Date>\w+\s+\d+\s+\d+:\d+:\d+)\s+(?P<device_ip>[^ ]+)\s+(?P<device_time>\w+\s+\d+\s+\d+:\d+:\d+\s+\d+)[^ \n]* (?P<hostname>[^ ]+)[^ \n]* \%\%10(?<cmd_grp>\w+)\/$
[extract_hp_route_cmd_parameters]
REGEX = (?:\s-)(?<cmd_params_src>.+)(?:;)
disabled = 0
[extract_hp_route_cmd_message]
REGEX = (?:;\s+)(?<cmd_msg>.+)$
[extracet_field_from_params]
REGEX = (\w+)=(\s?[.\-\s@\w]+)
FORMAT = $1::$2
SOURCE_KEY = cmd_params_src
#DELIMS = "|", "="
REPEAT_MATCH = True
CLEAN_KEYS = 1
What am I doing wrong, how to fix the situation?
regards
Michael
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![woodcock woodcock](https://community.splunk.com/legacyfs/online/avatars/1493.jpg)
To set this up to be automatic, like this:
In props.conf:
[hp_routers]
KV_MODE = none
REPORT-HP_KVPs = HP_KVPs
In transforms.conf:
[HP_KVPs]
REGEX = ([^-=]+)=([^=]*?)(?=$|-[^-=]+=)
FORMAT = $1::$2
REPEAT_MATCH = true
The key is to do your own KVP decoding and to use the RegEx positive lookahead feature to ensure that no keys (only values) can contain hyphens.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![woodcock woodcock](https://community.splunk.com/legacyfs/online/avatars/1493.jpg)
To set this up to be automatic, like this:
In props.conf:
[hp_routers]
KV_MODE = none
REPORT-HP_KVPs = HP_KVPs
In transforms.conf:
[HP_KVPs]
REGEX = ([^-=]+)=([^=]*?)(?=$|-[^-=]+=)
FORMAT = $1::$2
REPEAT_MATCH = true
The key is to do your own KVP decoding and to use the RegEx positive lookahead feature to ensure that no keys (only values) can contain hyphens.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![mbabakov mbabakov](https://community.splunk.com/legacyfs/online/avatars/549848.jpg)
Thank you!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![woodcock woodcock](https://community.splunk.com/legacyfs/online/avatars/1493.jpg)
Like this:
| makeresults
| eval raw="Dec 17 10:08:38 10.52.137.1 Apr 3 22:46:57 2012 930-RTR-944 %%10SSH/6/SSH_LOGIN(l): -DevIP=10.52.137.1; STEL user monitor (IP: 192.168.181.94) logged in successfully.:::Dec 17 10:08:38 10.52.137.1 Apr 3 22:46:57 2012 930-RTR-944 %%10SC/6/SC_AAA_SUCCESS(l): -DevIP=10.52.137.1-AAAType=ACCOUNT-AAAScheme= local-Service=login-UserName=monitor@system; AAA is successful.:::Dec 17 10:08:38 10.52.137.1 Apr 3 22:46:57 2012 930-RTR-944 %%10SC/6/SC_AAA_SUCCESS(l): -DevIP=10.52.137.1-AAAType=AUTHEN-AAAScheme= hwtacacs-scheme tacacs-Service=login-UserName=monitor@system; AAA is successful.:::Dec 17 10:08:13 10.98.171.65 Jan 20 00:00:17 2011 MSR954-RTR-LTE-5686 %%10CELLULAR/5/CELLULAR: -DevIP=10.98.171.65; Controller Cellular1/0: The network connection switched to 3G.:::Dec 17 10:08:04 10.199.69.26 May 23 21:50:30 2012 930-RTR-14815 %%10SSH/4/TrapLogoff(t): 1.3.6.1.4.1.25506.2.22.1.3.0.4 SSH user logoff trap information"
| makemv delim=":::" raw
| mvexpand raw
| rename raw AS _raw
| eval _time = strptime(_raw, "%b %d %H:%M:%S")
| rename COMMENT AS "Everything above generates sample events; everything below is your solution"
| rex max_match=0 "(?<key>[^-=]+)=(?<value>[^=]*?)(?=$|-[^-=]+=)"
| eval _raw=mvzip(key, value, "=")
| fields - key value
| kv
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![woodcock woodcock](https://community.splunk.com/legacyfs/online/avatars/1493.jpg)
The key is to do your own KVP decoding and to use the RegEx positive lookahead
feature to ensure that no keys (only values) can contain hyphens.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![mbabakov mbabakov](https://community.splunk.com/legacyfs/online/avatars/549848.jpg)
Thanks for your solution!
I wanted the analysis to go at the indexing stage. But in the end agreed to parse the search stage. 🙂
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![woodcock woodcock](https://community.splunk.com/legacyfs/online/avatars/1493.jpg)
Right, I forgot that. See my other answer.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![woodcock woodcock](https://community.splunk.com/legacyfs/online/avatars/1493.jpg)
I think you clicked Accept
on the wrong one, right?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![mbabakov mbabakov](https://community.splunk.com/legacyfs/online/avatars/549848.jpg)
Sorry, yes. Corrected
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![gcusello gcusello](https://community.splunk.com/legacyfs/online/avatars/553812.jpg)
![SplunkTrust SplunkTrust](/html/@E48BE65924041B382F8C3220FF058B38/rank_icons/splunk-trust-16.png)
Hi mbabakov,
can you use "DevIP" that is after "-"?
If yes, you can try (see on regex101.com at https://regex101.com/r/2cwhUx/1 )
-(?<my_field>DevIP[^;]*)
If instead you're not sure about the word "DevIP" but you're sure that there's the equal after the first word, you could use something like this
-(?<my_field>[^\=]*\=[^;]*)
Bye.
Giuseppe
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![mbabakov mbabakov](https://community.splunk.com/legacyfs/online/avatars/549848.jpg)
Sorry, but I need to work out all possible lines between "-" and ";". For example:
-DevIP = 10.52.137.1;
-DevIP = 10.52.137.1-AAAType = ACCOUNT-AAAScheme = local-Service = login-UserName = monitor@system;
-DevIP = 10.52.137.1-AAAType = AUTHEN-AAAScheme = hwtacacs-scheme tacacs-Service = login-UserName = monitor@system;
A special case of "-DevIP = 10.52.137.1;", this is very simple. 🙂
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![gcusello gcusello](https://community.splunk.com/legacyfs/online/avatars/553812.jpg)
![SplunkTrust SplunkTrust](/html/@E48BE65924041B382F8C3220FF058B38/rank_icons/splunk-trust-16.png)
infact, as you can see at https://regex101.com/r/2cwhUx/1 using this regex, you take all between "-" and the first ";".
If you have more ";", you can use a similar regex.
Bye.
Giuseppe
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![gcusello gcusello](https://community.splunk.com/legacyfs/online/avatars/553812.jpg)
![SplunkTrust SplunkTrust](/html/@E48BE65924041B382F8C3220FF058B38/rank_icons/splunk-trust-16.png)
Sorry for an additional information:
if you have "-DevIP = 10.52.137.1-AAAType = ACCOUNT-AAAScheme = local-Service = login-UserName = monitor@system;" do you want separately each field?
something like
DevIP = 10.52.137.1
AAAType = ACCOUNT
AAAScheme = local
Service = login
UserName = monitor@system
if this is your requirement, see https://regex101.com/r/2cwhUx/2 and try
-(?<my_field>\w*\=[^;-]*)
Bye.
Giuseppe
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![mbabakov mbabakov](https://community.splunk.com/legacyfs/online/avatars/549848.jpg)
Yes, I need to separate each field separately. And I do not know how many of these fields will be in different records and how they will be called.
Therefore, I need a construction of the form (? <_ KEY_1> [^ =] [\ p {Lu} \ p {Lt}]. +) = (? <_ VAL_1> [^ |]. +). But, when using the separator "-" between the pairs, the regular expression becomes very complicated, since the "-" symbol can be used in the Value field.
I used EVAL to replace "-" with "|", but I don’t know how in this case to force splunk to parse all key-value pairs.
For example https://regex101.com/r/tlaRxc/1.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
![mbabakov mbabakov](https://community.splunk.com/legacyfs/online/avatars/549848.jpg)
To clarify, I am interested in the analysis of all possible fields in rows in the log, for example:
-DevIP = 10.52.137.1;
-DevIP = 10.52.137.1-AAAType = ACCOUNT-AAAScheme = local-Service = login-UserName = monitor@system;
-DevIP = 10.52.137.1-AAAType = AUTHEN-AAAScheme = hwtacacs-scheme tacacs-Service = login-UserName = monitor@system;
Those. for the first line is DevIP. For the second, these are DevIP, AAAType, AAAScheme, Service, UserName, etc.
![](/skins/images/89D5ADE867CBAF0B5A525B7E23D83D7E/responsive_peak/images/icon_anonymous_message.png)