topic Re: How can I count group separators and field separators in an event or log file? in Splunk Search

How can I count group separators and field separators in an event or log file?

nitesh218ss — Wed, 22 Apr 2015 06:15:07 GMT

I have one log file where the Msg part contains many things like transaction id, error code, marchend name, etc
But these things are based on a group separator and field separator, so if i want any value like transaction id, I need to count which GS (group separator) or inside GS which FS (field separator) this value is present.

I pasted here a small part of my log file. If you open it in notepad++ you are able to see FS and GS, but via splunk or normal notepad, you are not able to see this FS GS.

my log file is:
ProgID ThreadID InternalSeq PortalSeq LogTime Msg

PWRECV(26104:10.120.11.117:1) 26104 1 1 2015-03-27 19:22:17:506 QRLNV12015032719221700001mmm09NPGBP00000000000000000000004444333322221111=1215 X00PWSERVER002

<<PWSEND(29924:10.120.11.117:1) 29924 2 2 2015-03-27 19:22:17:541 ARLNV12015032719221700001mmm09NPGBP0000000000000000000000XXXX XXXX XXXX 1111=1215NNNNNN0 NN1 V VISA Visa Credit BMS 2567326121249872 0000 B020150327192217PWSERVER56000000292200000350026999999990099999999002124987210000001000999900N

PWRECV(26104:10.120.11.117:2) 26104 3 3 2015-03-27 19:22:19:782 QRLNL12015032719221900001mmm09NPGBP00000004500000000000004444333322221111=1215 21249872 AdminPGBPX001PWSERVER00GBP2

<<PWSEND(31116:10.120.11.117:2) 31116 4 4 2015-03-27 19:22:24:019 ARLNL12015032719221900001mmm09NPGBP0000000450000000000000XXXX XXXX XXXX 1111=1215 0 NN1 A03BF96 OVISA Visa Credit BMS 2567326121249872 0040AUTH CODE:03BF96 0000 AdminPGBPO020150327192219AUTH CODE:03BF96PWSERVER56000000292310AC0000000GBP

PWRECV(26104:10.120.11.117:3) 26104 5 5 2015-03-27 19:22:26:591 QRL E042015032719222600001mmm09560000002923NP03BF96O450000002

<<PWSEND(29924:10.120.11.117:3) 29924 6 6 2015-03-27 19:22:26:640 ARL E042015032719222600001mmm09Y0000

So i want to know how i can read values base on GS or FS position in splunk.

Re: How can I count group separators and field separators in an event or log file?

bjoernjensen — Wed, 22 Apr 2015 08:02:39 GMT

You could use the FIELD_DELIMITER option in your props.conf to seperate fields by FS (file sepeator). I would recommend to edit the props.conf directly in np++ to make sure the right char is in place.
http://docs.splunk.com/Documentation/Splunk/6.2.2/Admin/Propsconf

Other then that: use regex knowing that \x1D matches GS and \x1C matches FS:
... | rex field=_raw "(?:[^\x1d]*\x1d){6}(?:[^\x1c]*\x1c){10}(?<myfield>[^\x1c\x1d]*)"

This would evalute on the field _raw and within there go after the 6th occurance of GS and within there after the 10th occurance of FS and take everything until the next GS/FS into the field myfield. This of course presupposes that the values are ALWAYS on the same "position".

Re: How can I count group separators and field separators in an event or log file?

nitesh218ss — Wed, 22 Apr 2015 09:30:10 GMT

For FIELD_DELIMITER i already taken is tab because if you see in header(ProgID ThreadID InternalSeq PortalSeq LogTime Msg)
every field is separated by tab but in Msg field is also separate in may part by GS of FS .so field inside msg i want to take base on GS and FS

Re: How can I count group separators and field separators in an event or log file?

bjoernjensen — Wed, 22 Apr 2015 10:04:54 GMT

I had to edit my post above (angle brackets).

Add an EXTRACT field extraction stanza to props.conf:
http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles#Add_an_EXTRACT_field_extraction_stanza_to_props.conf

This would result in something like this:
[sourcetypeStanza]
...
EXTRACT-myfield = (?:[^\x1d]*\x1d){6}(?:[^\x1c]*\x1c){10}([^\x1c\x1d]*) in Msg

Re: How can I count group separators and field separators in an event or log file?

nitesh218ss — Wed, 22 Apr 2015 10:06:14 GMT

i use this rex field=_raw "(?:[^\x1d]*\x1d){6}(?:[^\x1c]*\x1c){10}(?[^\x1c\x1d]*)"
but i got error:

Error in 'rex' command: Encountered the following error while compiling the regex '(?:[^\x1d]*\x1d){1}(?:[^\x1c]*\x1c){2}(?[^\x1c\x1d]*)': Regex: unrecognized character after (? or (?-

Re: How can I count group separators and field separators in an event or log file?

nitesh218ss — Wed, 22 Apr 2015 11:14:48 GMT

i use this line in props.conf but myfield not coming in field

[field serp]
FIELD_DELIMITER = tab
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = false
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
EXTRACT-myfield = (?:[^\x1d]*\x1d){1}(?:[^\x1c]*\x1c){2}([^\x1c\x1d]*) in Msg

Re: How can I count group separators and field separators in an event or log file?

nitesh218ss — Wed, 22 Apr 2015 11:29:43 GMT

When i use this search with EXTRACT-myfield = (?:[^x1d]x1d){1}(?:[^x1c]x1c){2}([^x1cx1d]) in Msg in props.conf. After that I ran this search:

index=myrr | rex field=_raw "(?:[^x1d]x1d){6}(?:[^x1c]x1c){10}(?[^x1cx1d])"

Then i got my field thank you