Getting Data In

Configuring Transforms.conf for log with multiple delimiters

ezajac
Path Finder

I have a new log that I need to define a sourcetype for. There are several different ways the fields are delimited.

The formatting is defined in a Log4Net.config file is below. Is there an easy way to configure this in the transforms.conf file?


2012-10-17 12:06:43,546 [53] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyData203.125
2012-10-17 12:06:43,656 [6] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyData203.125
2012-10-17 12:06:43,765 [53] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyNameAddress156.25
2012-10-17 12:06:43,828 [53] INFO HTTPPages [(null)] - PROD\AOSHOGAYHTTPPagehttp://plcontactview.prodlb.travp.net/plcontactview/PLContactSearch.aspx?txtClientFunction=InsuredSe...671.875

GET

2012-10-17 12:06:43,875 [6] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyNameAddress203.125
2012-10-17 12:06:43,921 [6] INFO HTTPPages [(null)] - PROD\BROBERT4HTTPPagehttp://plcontactview.prodlb.travp.net/plcontactview/PLContactSearch.aspx?txtClientFunction=PhoneSear...640.625
398POST

2012-10-17 12:06:44,234 [6] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyData250
2012-10-17 12:06:44,265 [41] INFO ServiceCalls [(null)] - ServiceCallPreferenceManagementServiceHook.GetPreferences218.75
2012-10-17 12:06:44,328 [16] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyData234.375
2012-10-17 12:06:44,328 [53] INFO ServiceCalls [(null)] - ServiceCallPreferenceManagementServiceHook.GetPreferences156.25
2012-10-17 12:06:44,421 [41] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetClaimData156.25
2012-10-17 12:06:44,437 [41] INFO HTTPPages [(null)] - PROD\BROBERT4HTTPPagehttp://plcontactview.prodlb.travp.net/plcontactview/PLBillingTreeView.aspx421.875
GET

2012-10-17 12:06:44,437 [6] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyNameAddress171.875
2012-10-17 12:06:44,531 [41] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetProducerData46.875
2012-10-17 12:06:44,531 [53] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetClaimData203.125
2012-10-17 12:06:44,546 [16] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyNameAddress187.5

Tags (1)
0 Karma

davecroto
Splunk Employee
Splunk Employee

Set a common, single sourcetype for the 2 type of events you show above. Use a tranforms and props stanza like I did here... I called my sourcetype log4net. Now, all the events from that source with either the word ServiceCalls or HTTPPages will be sourcetyped as log4net. Then just refer to the new sourcetype in the props.conf file for the field extractions.

tranforms.conf

[log4net]
REGEX = (ServiceCalls|HTTPPages)
FORMAT = sourcetype::log4net
DEST_KEY = MetaData:Sourcetype

props.conf

[source::…/bla/bla/log4nettype.log]
TRANSFORMS-log4net =log4net

[log4net]
EXTRACT-log4net = (?i)\[(?\d+)\]\s+(?\w+)\s+(?\w+)\s+\[\((?\w+)\)\]\s+\-(?.*)

0 Karma

davecroto
Splunk Employee
Splunk Employee

Noted that the "backslashes" in the Extract needed to be escaped to show up on this post. Let me know if this worked for you....

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...