Getting Data In
Highlighted

How do I configure Splunk to break events properly based on my sample data?

Builder

I have the below sample data and I want to break the events at the request message qualifier field

Request Message Qualifier: CMO_REQ Inquiry Type : DEP Bank: PB_ABA1 Amount: 12500   # $125.00 which is less than $500.00. Response Message Qualifier: CMO_RES Instrument Type: CHK Payment Guarantee: G Request Message Qualifier: CMO_REQ
Inquiry Type : DEP Bank: PB_ABA1 Amount: 150000 # $1500.00 is more than $500 Response Message Qualifier: CMO_RES
Instrument Type: CHK Payment Guarantee: G.................................

I want to break the above event at Request Message Qualifier i.e. the events starts with Request Message Qualifier and end at Payment Guarantee .

I have also attached the sample file.

Please help on this.Thanks in advance

0 Karma
Highlighted

Re: How do I configure Splunk to break events properly based on my sample data?

Legend

Updated to include timestamp

Because "Request Message Qualifier" appears repeatedly in your data, I think you will be better off if you use the "Payment Guarantee" to separate events.
In props.conf on the indexer (or wherever you are parsing the data), do this

[yoursourcetypehere]
SHOULD_LINEMERGE=true
MUST_BREAK_AFTER = ^Payment Guarantee\:
BREAK_ONLY_BEFORE_DATE = false
DATETIME_CONFIG = CURRENT

Since there is no timestamp in the data, this tells Splunk to use the current time as the timestamp. Making this explicit will stop Splunk from searching for a timestamp where there is none.

Highlighted

Re: How do I configure Splunk to break events properly based on my sample data?

SplunkTrust
SplunkTrust

Agreed, timestamps should be in your events. Go to your props.conf and add this stanza to get timestamping

[sourcetypeName]
TIME_PREFIX = ^
TIME_FORMAT = %s%3n

If you timestamp your events you will then need to modify your linebreaking regex to match the timestamp

0 Karma
Highlighted

Re: How do I configure Splunk to break events properly based on my sample data?

Builder

I have used the following when indexing the data

Sourcetype=system Defaults

breaktype as regex

Pattern: Request Message Qualifier

timestamp : Current time

The data look like this in splunk for each event
alt text

The above image shows how data looks in splunk ,but i want request message qualifier,inquiry type,bank,account etc in interesting fields

0 Karma
Highlighted

Re: How do I configure Splunk to break events properly based on my sample data?

Legend

Your original question did not include how to get the fields; that's really a different question. But you could add this to props.conf

TRANSFORMS-ef1=extract_message_fields

Then create the transforms.conf file

[extract_message_fields]
REGEX  =^(?m-s)(?<_KEY_1>\w+):(?<_VAL_1>\w+)$
MV_ADD = true
CLEAN_KEYS=true

This should extract all the fields in the event. However, only the most common fields will show up in the interesting fields list.

0 Karma
Highlighted

Re: How do I configure Splunk to break events properly based on my sample data?

SplunkTrust
SplunkTrust

Go into your $Splunk_Home\etc\system\local\props.conf on the indexer and insert this stanza

[source::<YOUR SOURCE>]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = Request\sMessage\sQualifier\:

Restart the Splunk service

http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Indexmulti-lineevents

0 Karma
Highlighted

Re: How do I configure Splunk to break events properly based on my sample data?

Builder

I have used the following when indexing the data

Sourcetype=system Defaults

breaktype as regex

Pattern: Request Message Qualifier

timestamp : Current time

The data look like this in splunk for each event

alt text

I want request message qualifier,inquiry type,bank,account etc in interesting fields ..Is that possible

0 Karma