Merging related rows and allowing them to be searchable

[scottjreynolds]

We have a logfile that logs the following two lines per logical unit of work completed by the application server. In the report and dashboard that we'd like to produce, we somehow would like to merge the FROM data to the TO format so that it's searchable. We've thought about writing an external script to process the log and load the formatted data into splunk as an additional event source, but we'd ideally prefer to utilize splunk to merge the data and store it as a new event.

FROM:

[00:00:42.839] {Correlation_Id=7437c4c7-acad-6fec-bc4d-470905a6642fd, Referrer=null, name=null, first_name=John, last_name=Doe, contact_time=Evening, callOptIn=null, email=jdoe@domain.com, phone=123-123-1234, phone2=, fax=null, address_1_city=New York, address_1_state=NY, address_1_street1=New York St, address_1_street2=null, address_1_zip=12258, ip_address=xxx.xxx.xxx.xxx, affiliate_id=1234, lid=null, link_id=null, custom_subaff=null, custom_campaign=null, config_id=1, type=Complete, import=null, redirect=-1, source=null, distributionCap=1, licenseKey=, agent_affiliate_id=null, searchignite_client_id=null, searchignite_token=null, how_hear=null, licenseNumbers=, captiveCompanyIDs=1}

[00:00:42.852] {FAILURE=>, Correlation_Id=7437c4c7-acad-6fec-bc4d-470905a6642fd, Problem="(BatchedLeadValidator)=Affiliate has exceeded his monthly limit}"

TO:

2012/05/06 00:00:42.839 name=FAILURE event_id=7437c4c7-acad-6fec-bc4d-470905a6642fd end_time="2012/05/06 00:00:42.852" error="(BatchedLeadValidator)=Affiliate has exceeded his monthly limit" Referrer=null first_name=John last_name=Die contact_time=Evening callOptIn=null email=jdoe@domain.com...and so on.

Any suggestions on how to approach this would be great.

Thanks!

[bjalex80]

The field on the TO event is named slightly different so you can do something like this "| eval Correlation_Id=coalesce(Correlation_Id,event_id) | transaction Correlation_ID"

[mikelanghorst]

Not sure about merging the data to a new event, but "| transaction Correlation_Id" (without the quotes) will group the 2 events together allowing you to search/report on them as a single event.