Consume Free-Form text

[kmattern]

I have a help desk database in SQL Server that I want to export to log type files and have Splunk consume. I'm not having any trouble getting the data into Splunk but I can't seem to get Splunk to understand where the boundaries for each record/event is. I have defined my output as follows:

TicketNum=000001

CustName=Bob Smith

CallDate=2011-11-01

Status=Closed

CallDesc=Mr. Smith had trouble accessing his hydro accelerator while in mimsy mode.

CallResolution=Told Mr. Smith that he had to be sure his vorbis was in gear

Of course the CallDesc and CallResolution fields can be quite long. They contain copies of emails, comments and more. I have been careful to separate them with only line feeds. The only carriage return/linefeed is at the end of each record/event. There are 14 fields in each record/event.

When I run a search on the raw data many of the records/events run together and they do not necessarily break at the end of a record/event. Yet others do. I have set the DELIMS="\n" in transforms.conf but it doesn't seem to help.

Does anyone know how I can break these records/events out properly?

Thanks

[tgow · Accepted solution]

You will need to instruct Splunk that this is a mulit-line event and also tell it where the line breakers are. Assuming that the TicketNum field is where a new event starts try this in your $SPLUNK_HOME/etc/system/local/props.conf:

[yoursourcetype]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = ^TicketNum

[_d_]

Try using DELIMS="([\r\n])+" as there may be carriage returns and/or new lines.

Hope this helps

> please upvote and accept answer if you find it useful - thanks!

[kmattern]

That didn't seem to do anything different. Maybe part of the problem is that in the free-form text theare are usually a number of dates. Emails are copied comppletely into these records and that includes the date and time of the email. I moved all of my date fields to the top of the event but that didn't seem to help either.