Configuring Transforms.conf for log with multiple delimiters

[ezajac]

I have a new log that I need to define a sourcetype for. There are several different ways the fields are delimited.

The formatting is defined in a Log4Net.config file is below. Is there an easy way to configure this in the transforms.conf file?

2012-10-17 12:06:43,546 [53] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyData203.125
2012-10-17 12:06:43,656 [6] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyData203.125
2012-10-17 12:06:43,765 [53] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyNameAddress156.25
2012-10-17 12:06:43,828 [53] INFO HTTPPages [(null)] - PROD\AOSHOGAYHTTPPagehttp://plcontactview.prodlb.travp.net/plcontactview/PLContactSearch.aspx?txtClientFunction=InsuredSe...671.875

GET

2012-10-17 12:06:43,875 [6] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyNameAddress203.125
2012-10-17 12:06:43,921 [6] INFO HTTPPages [(null)] - PROD\BROBERT4HTTPPagehttp://plcontactview.prodlb.travp.net/plcontactview/PLContactSearch.aspx?txtClientFunction=PhoneSear...640.625

398POST

2012-10-17 12:06:44,234 [6] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyData250
2012-10-17 12:06:44,265 [41] INFO ServiceCalls [(null)] - ServiceCallPreferenceManagementServiceHook.GetPreferences218.75
2012-10-17 12:06:44,328 [16] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyData234.375
2012-10-17 12:06:44,328 [53] INFO ServiceCalls [(null)] - ServiceCallPreferenceManagementServiceHook.GetPreferences156.25
2012-10-17 12:06:44,421 [41] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetClaimData156.25
2012-10-17 12:06:44,437 [41] INFO HTTPPages [(null)] - PROD\BROBERT4HTTPPagehttp://plcontactview.prodlb.travp.net/plcontactview/PLBillingTreeView.aspx421.875

GET

2012-10-17 12:06:44,437 [6] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyNameAddress171.875
2012-10-17 12:06:44,531 [41] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetProducerData46.875
2012-10-17 12:06:44,531 [53] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetClaimData203.125
2012-10-17 12:06:44,546 [16] INFO ServiceCalls [(null)] - ServiceCallCRDBServiceHook.GetPolicyNameAddress187.5

[davecroto]

Set a common, single sourcetype for the 2 type of events you show above. Use a tranforms and props stanza like I did here... I called my sourcetype log4net. Now, all the events from that source with either the word ServiceCalls or HTTPPages will be sourcetyped as log4net. Then just refer to the new sourcetype in the props.conf file for the field extractions.

tranforms.conf

[log4net]
REGEX = (ServiceCalls|HTTPPages)
FORMAT = sourcetype::log4net
DEST_KEY = MetaData:Sourcetype

props.conf

[source::…/bla/bla/log4nettype.log]
TRANSFORMS-log4net =log4net

[log4net]
EXTRACT-log4net = (?i)\[(?\d+)\]\s+(?\w+)\s+(?\w+)\s+\[\((?\w+)\)\]\s+\-(?.*)

[davecroto]

Noted that the "backslashes" in the Extract needed to be escaped to show up on this post. Let me know if this worked for you....