Extract Field Name and Value from Data Source using Delimiter based KV extraction

jspigler2010 — Wed, 03 Jul 2019 19:39:39 GMT

I'm looking to dynamically extract both the field name and the associated value from a data source. Essentially, the field name is a session variable on F5 and these variables can be added and removed at will. So statically trying to regex the field extraction and field name would be impossible. Here is a data sample

hostname="ip-10-1-1-49.us-gov-west-1.compute.internal",errdefs_msgno="01490007:6:",partition_name="Common",session_id="c3da4e31",Access_Profile="/Common/testPolicy1",Partition="Common",Session_Id="c3da4e31",Session_Variable_Name="session.ssl.cert.end",Session_Variable_Value="Nov 26 22:14:03 2020 GMT"
hostname="ip-10-1-1-49.us-gov-west-1.compute.internal",errdefs_msgno="01490007:6:",partition_name="Common",session_id="c3da4e31",Access_Profile="/Common/testPolicy1",Partition="Common",Session_Id="c3da4e31",Session_Variable_Name="session.ssl.cert.exist",Session_Variable_Value="1"
hostname="ip-10-1-1-49.us-gov-west-1.compute.internal",errdefs_msgno="01490007:6:",partition_name="Common",session_id="c3da4e31",Access_Profile="/Common/testPolicy1",Partition="Common",Session_Id="c3da4e31",Session_Variable_Name="session.ssl.cert.issuer",Session_Variable_Value="DC=local, DC=cloudmegalodon, CN=cloudmegalodon-CMDC1-CA"
hostname="ip-10-1-1-49.us-gov-west-1.compute.internal",errdefs_msgno="01490007:6:",partition_name="Common",session_id="c3da4e31",Access_Profile="/Common/testPolicy1",Partition="Common",Session_Id="c3da4e31",Session_Variable_Name="session.ssl.cert.serial",Session_Variable_Value="59:00:00:00:09:21:54:f7:30:0b:fd:f7:9f:00:00:00:00:00:09"
hostname="ip-10-1-1-49.us-gov-west-1.compute.internal",errdefs_msgno="01490007:6:",partition_name="Common",session_id="c3da4e31",Access_Profile="/Common/testPolicy1",Partition="Common",Session_Id="c3da4e31",Session_Variable_Name="session.ssl.cert.start",Session_Variable_Value="Nov 27 22:14:03 2018 GMT"
hostname="ip-10-1-1-49.us-gov-west-1.compute.internal",errdefs_msgno="01490007:6:",partition_name="Common",session_id="c3da4e31",Access_Profile="/Common/testPolicy1",Partition="Common",Session_Id="c3da4e31",Session_Variable_Name="session.ssl.cert.subject",Session_Variable_Value="DC=local, DC=cloudmegalodon, OU=OrgUsers, CN=TINA F. OLSON.1468013579"
hostname="ip-10-1-1-49.us-gov-west-1.compute.internal",errdefs_msgno="01490007:6:",partition_name="Common",session_id="c3da4e31",Access_Profile="/Common/testPolicy1",Partition="Common",Session_Id="c3da4e31",Session_Variable_Name="session.ssl.cert.valid",Session_Variable_Value="0"

I want to take for example Session_Variable_Name="session.ssl.cert.serial",Session_Variable_Value="59:00:00:00:09:21:54:f7:30:0b:fd:f7:9f:00:00:00:00:00:09"
and make session.ssl.cert.serial the field name and 59:00:00:00:09:21:54:f7:30:0b:fd:f7:9f:00:00:00:00:00:09 the field value.

I have not been able to get this to work with kv pairdelim and kvdelim. Maybe there is a better way to do this. Open to any and all ideas!

Here is the search I'm working with now

index="test_f5" sourcetype="f5_syslog_splunk" | kv pairdelim="Session_Variable_Name\r\n" kvdelim="\",Session_Variable_Value=\""

Re: Extract Field Name and Value from Data Source using Delimiter based KV extraction

harsmarvania57 — Thu, 04 Jul 2019 08:21:19 GMT

Hi,

You can achieve this using regular expression in props and transforms to extract field and value at search time.

On Search Head
props.conf

[yourSourceType]
REPORT-test = extract_session_kv

transforms.conf

[extract_session_kv]
CLEAN_KEYS = 0
FORMAT = $1::$2
REGEX = Session_Variable_Name\=\"([^"]+)\"\,Session_Variable_Value\=\"([^"]+)\"

Re: Extract Field Name and Value from Data Source using Delimiter based KV extraction

jspigler2010 — Mon, 08 Jul 2019 16:55:43 GMT

Works great! Thanks harsmarvania!

topic Re: Extract Field Name and Value from Data Source using Delimiter based KV extraction in Splunk Search

Extract Field Name and Value from Data Source using Delimiter based KV extraction

Re: Extract Field Name and Value from Data Source using Delimiter based KV extraction

Re: Extract Field Name and Value from Data Source using Delimiter based KV extraction