How do I configure Splunk to break events properly based on my sample data?

[vrmandadi]

I have the below sample data and I want to break the events at the request message qualifier field

Request Message Qualifier: CMO_REQ Inquiry Type : DEP Bank: PB_ABA1 Amount: 12500   # $125.00 which is less than $500.00. Response Message Qualifier: CMO_RES Instrument Type: CHK Payment Guarantee: G Request Message Qualifier: CMO_REQ
Inquiry Type : DEP Bank: PB_ABA1 Amount: 150000 # $1500.00 is more than $500 Response Message Qualifier: CMO_RES
Instrument Type: CHK Payment Guarantee: G.................................

I want to break the above event at Request Message Qualifier i.e. the events starts with Request Message Qualifier and end at Payment Guarantee .

I have also attached the sample file.

Please help on this.Thanks in advance

[skoelpin]

Go into your $Splunk_Home\etc\system\local\props.conf on the indexer and insert this stanza

[source::<YOUR SOURCE>]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = Request\sMessage\sQualifier\:

Restart the Splunk service

http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Indexmulti-lineevents

[vrmandadi]

I have used the following when indexing the data

Sourcetype=system Defaults

breaktype as regex

Pattern: Request Message Qualifier

timestamp : Current time

The data look like this in splunk for each event

I want request message qualifier,inquiry type,bank,account etc in interesting fields ..Is that possible

[lguinn2]

Updated to include timestamp

Because "Request Message Qualifier" appears repeatedly in your data, I think you will be better off if you use the "Payment Guarantee" to separate events.
In props.conf on the indexer (or wherever you are parsing the data), do this

[yoursourcetypehere]
SHOULD_LINEMERGE=true
MUST_BREAK_AFTER = ^Payment Guarantee\:
BREAK_ONLY_BEFORE_DATE = false
DATETIME_CONFIG = CURRENT

Since there is no timestamp in the data, this tells Splunk to use the current time as the timestamp. Making this explicit will stop Splunk from searching for a timestamp where there is none.

[lguinn2]

Your original question did not include how to get the fields; that's really a different question. But you could add this to props.conf

TRANSFORMS-ef1=extract_message_fields

Then create the transforms.conf file

[extract_message_fields]
REGEX  =^(?m-s)(?<_KEY_1>\w+):(?<_VAL_1>\w+)$
MV_ADD = true
CLEAN_KEYS=true

This should extract all the fields in the event. However, only the most common fields will show up in the interesting fields list.

[skoelpin]

Agreed, timestamps should be in your events. Go to your props.conf and add this stanza to get timestamping

[sourcetypeName]
TIME_PREFIX = ^
TIME_FORMAT = %s%3n

If you timestamp your events you will then need to modify your linebreaking regex to match the timestamp

[vrmandadi]

I have used the following when indexing the data

Sourcetype=system Defaults

breaktype as regex

Pattern: Request Message Qualifier

timestamp : Current time

The data look like this in splunk for each event

The above image shows how data looks in splunk ,but i want request message qualifier,inquiry type,bank,account etc in interesting fields