topic Re: Automatic field discovery / key value pairs with := in Getting Data In

Automatic field discovery / key value pairs with :=

adiwhy — Fri, 17 Jan 2020 08:06:25 GMT

Below are two line example of the data being indexed.

2020-01-17 15:40:53; 192.168.0.69; 192.168.0.69; Trap Service; abcde;
MessageType:= SNMPv2-MIB:authenticationFailure;
Message:= SNMP Trap
Received Time:1/17/2020 3:40:53 PM
Source:192.168.0.69(192.168.0.69)
Community:abcde
Variable Bindings
sysUpTime:= 30 days 3 hours 56 minutes 30.81 seconds (260619081)
snmpTrapOID:= SNMPv2-MIB:authenticationFailure (1.3.6.1.6.3.1.1.5.5)
snmpTrapEnterprise:= LIEBERT-GP-REGISTRATION-MIB:liebertGlobalProducts (1.3.6.1.4.1.476.1.42);
--ENDOFTRAP--

2020-01-17 15:40:52; 192.168.1.6; 192.168.1.6; Trap Service; abcde;
MessageType:= LIEBERT-GP-AGENT-MIB:lgpAgentHeartbeat;
Message:= SNMP Trap
Received Time:1/17/2020 3:40:52 PM
Source:192.168.1.6(192.168.1.6)
Community:abcde
Variable Bindings
sysUpTime:= 41 days 23 hours 8 minutes 4.90 seconds (362568490)
lgpConditionsPresent:= 0
lgpAgentConnectedDeviceCount:= 1
snmpTrapOID:= LIEBERT-GP-AGENT-MIB:lgpAgentHeartbeat (1.3.6.1.4.1.476.1.42.2.3.0.7)
sysUpTime:= 41 days 23 hours 8 minutes 4.90 seconds (362568490)
experimental.1057.1.0:= 192.168.1.6
snmpTrapEnterprise:= LIEBERT-GP-REGISTRATION-MIB:lgpAgentNotifications (1.3.6.1.4.1.476.1.42.2.3);
--ENDOFTRAP--

I would like to have all string with bold font above (before ":=") automatically discovered as fields. And all strings after := become its field's value.

I did below on transform & props but still not working. any help is appreciated.
transforms.conf :

[trap_plaintransform]
REGEX=\[(?!(?:headerName|headerValue))([^\s\=]+)\:\=([^\]]+)\]
FORMAT=$1::$2

[trap_transform]
REGEX= \[headerName\=(\w+)\],\s\[headerValue=([^\]]+)\]
FORMAT= $1::$2

props.conf:

[s_trap]
DATETIME_CONFIG =
MAX_TIMESTAMP_LOOKAHEAD = 32
MUST_BREAK_AFTER = \-\-ENDOFTRAP\-\-
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S
category = Miscellaneous
disabled = false
pulldown_type = true
SHOULD_LINEMERGE = true
TRANSFORMS-sw_trap_host = sw_trap_host
BREAK_ONLY_BEFORE =
KV_MODE = none
REPORT-a= trap_transform, trap_plaintransform

Re: Automatic field discovery / key value pairs with :=

richgalloway — Fri, 17 Jan 2020 14:25:24 GMT

Part of the problem lies with the REGEX strings. They both expect to find text within square brackets, but there are no square brackets in the sample events.

Re: Automatic field discovery / key value pairs with :=

maciep — Sat, 18 Jan 2020 14:45:14 GMT

what about this regex with $1::$2 ??

https://regex101.com/r/fXTb3W/1

Re: Automatic field discovery / key value pairs with :=

woodcock — Sat, 18 Jan 2020 19:13:00 GMT

Try this:

In your props.conf:

[YourSourcetypeNameHere]
REPORT-YourSourcetypeNameHere_OID_KVPs = YourSourcetypeNameHere_OID_KVPs

In your transforms.conf:

[YourSourcetypeNameHere_OID_KVPs]
REGEX = ^|[\r\n](?<key>[^:]+):=?\s*(?<value>.*$)
FORMAT = $1::$2
REPEAT_MATCH = true

Re: Automatic field discovery / key value pairs with :=

adiwhy — Tue, 21 Jan 2020 02:39:42 GMT

yeah right.. i found it already how to do that after struggling on the docs 😄

Re: Automatic field discovery / key value pairs with :=

adiwhy — Tue, 21 Jan 2020 03:06:47 GMT

weird.. i tried that regex in the regex tester it is working, but it is not working when i put on the transforms.conf

Re: Automatic field discovery / key value pairs with :=

adiwhy — Tue, 21 Jan 2020 03:13:19 GMT

somehow after reading docs, i tried mine below is working

transforms.conf

[sw_trap_transform]
REGEX= ([\w|\.]+)\:\=\s([\w|\.|\s|\(|\)|\-|\d|\:|\[|\]|\;]+)$
FORMAT= $1::$2

[sw_trap_transform2]
REGEX = ([\w|\.]+)\:\=\s([^\n\r].*)\;$
FORMAT= $1::$2

props.conf

[sw_trap]
KV_MODE = none
REPORT-a = sw_trap_transform, sw_trap_transform2

Re: Automatic field discovery / key value pairs with :=

richgalloway — Tue, 21 Jan 2020 12:28:24 GMT

@adiwhy If your problem is resolved then please accept an answer to help future readers.