Getting Data In

Automatic field discovery / key value pairs with :=

adiwhy
Explorer

Below are two line example of the data being indexed.

2020-01-17 15:40:53; 192.168.0.69; 192.168.0.69; Trap Service; abcde;
MessageType:= SNMPv2-MIB:authenticationFailure;
Message:= SNMP Trap
Received Time:1/17/2020 3:40:53 PM
Source:192.168.0.69(192.168.0.69)
Community:abcde
Variable Bindings
sysUpTime:= 30 days 3 hours 56 minutes 30.81 seconds (260619081)
snmpTrapOID:= SNMPv2-MIB:authenticationFailure (1.3.6.1.6.3.1.1.5.5)
snmpTrapEnterprise:= LIEBERT-GP-REGISTRATION-MIB:liebertGlobalProducts (1.3.6.1.4.1.476.1.42);
--ENDOFTRAP--

2020-01-17 15:40:52; 192.168.1.6; 192.168.1.6; Trap Service; abcde;
MessageType:= LIEBERT-GP-AGENT-MIB:lgpAgentHeartbeat;
Message:= SNMP Trap
Received Time:1/17/2020 3:40:52 PM
Source:192.168.1.6(192.168.1.6)
Community:abcde
Variable Bindings
sysUpTime:= 41 days 23 hours 8 minutes 4.90 seconds (362568490)
lgpConditionsPresent:= 0
lgpAgentConnectedDeviceCount:= 1
snmpTrapOID:= LIEBERT-GP-AGENT-MIB:lgpAgentHeartbeat (1.3.6.1.4.1.476.1.42.2.3.0.7)
sysUpTime:= 41 days 23 hours 8 minutes 4.90 seconds (362568490)
experimental.1057.1.0:= 192.168.1.6
snmpTrapEnterprise:= LIEBERT-GP-REGISTRATION-MIB:lgpAgentNotifications (1.3.6.1.4.1.476.1.42.2.3);
--ENDOFTRAP--

I would like to have all string with bold font above (before ":=") automatically discovered as fields. And all strings after := become its field's value.

I did below on transform & props but still not working. any help is appreciated.
transforms.conf :

[trap_plaintransform]
REGEX=\[(?!(?:headerName|headerValue))([^\s\=]+)\:\=([^\]]+)\]
FORMAT=$1::$2

[trap_transform]
REGEX= \[headerName\=(\w+)\],\s\[headerValue=([^\]]+)\]
FORMAT= $1::$2

props.conf:

[s_trap]
DATETIME_CONFIG =
MAX_TIMESTAMP_LOOKAHEAD = 32
MUST_BREAK_AFTER = \-\-ENDOFTRAP\-\-
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S
category = Miscellaneous
disabled = false
pulldown_type = true
SHOULD_LINEMERGE = true
TRANSFORMS-sw_trap_host = sw_trap_host
BREAK_ONLY_BEFORE =
KV_MODE = none
REPORT-a= trap_transform, trap_plaintransform
0 Karma
1 Solution

adiwhy
Explorer

somehow after reading docs, i tried mine below is working

transforms.conf

[sw_trap_transform]
REGEX= ([\w|\.]+)\:\=\s([\w|\.|\s|\(|\)|\-|\d|\:|\[|\]|\;]+)$
FORMAT= $1::$2

[sw_trap_transform2]
REGEX = ([\w|\.]+)\:\=\s([^\n\r].*)\;$
FORMAT= $1::$2

props.conf

[sw_trap]
KV_MODE = none
REPORT-a = sw_trap_transform, sw_trap_transform2

View solution in original post

0 Karma

adiwhy
Explorer

somehow after reading docs, i tried mine below is working

transforms.conf

[sw_trap_transform]
REGEX= ([\w|\.]+)\:\=\s([\w|\.|\s|\(|\)|\-|\d|\:|\[|\]|\;]+)$
FORMAT= $1::$2

[sw_trap_transform2]
REGEX = ([\w|\.]+)\:\=\s([^\n\r].*)\;$
FORMAT= $1::$2

props.conf

[sw_trap]
KV_MODE = none
REPORT-a = sw_trap_transform, sw_trap_transform2
0 Karma

richgalloway
SplunkTrust
SplunkTrust

@adiwhy If your problem is resolved then please accept an answer to help future readers.

---
If this reply helps you, Karma would be appreciated.
0 Karma

woodcock
Esteemed Legend

Try this:

In your props.conf:

[YourSourcetypeNameHere]
REPORT-YourSourcetypeNameHere_OID_KVPs = YourSourcetypeNameHere_OID_KVPs

In your transforms.conf:

[YourSourcetypeNameHere_OID_KVPs]
REGEX = ^|[\r\n](?<key>[^:]+):=?\s*(?<value>.*$)
FORMAT = $1::$2
REPEAT_MATCH = true

adiwhy
Explorer

weird.. i tried that regex in the regex tester it is working, but it is not working when i put on the transforms.conf

0 Karma

maciep
Champion

what about this regex with $1::$2 ??

https://regex101.com/r/fXTb3W/1

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Part of the problem lies with the REGEX strings. They both expect to find text within square brackets, but there are no square brackets in the sample events.

---
If this reply helps you, Karma would be appreciated.
0 Karma

adiwhy
Explorer

yeah right.. i found it already how to do that after struggling on the docs 😄

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...