LINE_BREAKER not work with if i add FIELD_DELIMITER in props.conf

[nitesh218ss]

I create regularExp. for line break which work correctly but if i add FIELD_DELIMITER = | with that then line break not work.

props.conf
[uk1] When work properly
LINE_BREAKER = ([\r\n]+)\d{2}:\d{2}:\d{2}.\d{3}|[^|]+|[^|]+|[^|]+|[^|]+|[^|]+|
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
disabled = false

[uk12] when not work with field delimiter
FIELD_DELIMITER = |
FIELD_NAMES = timee,audit,serviceType,processId,serviceName,indicator,status1,status2,status3,status4,status5,status6
LINE_BREAKER = ([\r\n]+)\d{2}:\d{2}:\d{2}.\d{3}|[^|]+|[^|]+|[^|]+|[^|]+|[^|]+|
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
disabled = false
pulldown_type = true

my log file line is start sample
13:47:17.624|1303659|2|32| |O|Creating new re

My log file sample
<start>
13:47:17.624|1303659|2|32| |O|Creating new reply object to send response
13:47:17.624|8100229|2|32| |O|1|13:47:16.202|<NUA>101008</NUA><Request>1221606590018F2F61179807908**********************************50000115031813462282682610(EMV_DATA)104800601323000020000E0B8C851DA9B2F</Request>
TT: 62 (ms) 13:47:16.249|1221606590018F2F61079807908**********************************500015031813462282682610(EMV_DATA)104800601323000020000E0B8C8
13:47:17.609|122160659001812000AUTH CODE:91127801503(EMV_DATA)00080000100112345678901234512341002GP15077000000BC1
13:47:17.624|<Response><Data>122160659001813002911278100001AUTH CODE:911278090A437769C37E915303000080000100112345678901234512341002GP15077000000BC1DD2D26A9</Data><Offline>0</Offline><UTID>001303659</UTID><Client Key>43</Client Key><POSMessageFormat>1</POSMessageFormat><THOG Time>62</THOG Time><Operator Time>1360</Operator Time></Response>
13:47:17.624|8100229|2|32| |O|getLogMsg returned: [229^101008^17^ihost-uatlive^1^2^5^S^12^11^13^0018^231^0^15^D^18^22160659^21^150318^22^134600^24^TtWlS34ZjBo=^25^0^26^5000^199^5000^29^XTKueLHMJ9pDGvfSQn0xXI6sjf7niXAV^252^0119^181^1^274^1^275^0^45^43^71^00^80^79807908^82^20150318^83^134716^84^ASH001303659^23^911278^30^92^70^476173900^126^1^147^0^148^1^149^0^160^0^16^0^195^2^176^826^40^826^177^^178^22^179^10^188^^180^0^112^19Ifc1SHTID5k5lzxvfcU/E3O1GW+yWMFYHl+H9QlSALfWyz0rELv8UX7rdiLsIfTwMjR/DJPDrsKKBlNlQr4vurBvytdUbb7VvL1tk+r8o6VAXue3/3/PBxITFU6B8VlEF1ghoVwe8VVpWdI43HUKYPYVeDhPzQpqn5Efruc9OQY=^112^21104800601323000020000E0B8C8^19^3^137^F2F6^151^1^150^10000^146^000800^112^2090A437769C37E9153030^112^2200100112345678901234512341002GP15077000000BC1^95^AUTH CODE:911278^196^1^94^0^119^0^134^0^33^0^253^5C413DD4331BA4E6436E2E96FDA471D3^107^^32^0000^]
13:47:17.624|8100229|1|81| |O|Received 360 byte response from IP Gateway

<end>

[rsennett_splunk · Accepted solution]

(for those of you watching from home... Nitesh has sent me his data and my answer reflects a larger sample than he has provided int he question)

Nitesh…

You have twelve fields in your "FIELDS" statement and yet the data doesn't reflect that...
In order to use the DELIMITER you would have structured data. The field list implies that each field is represented all the time. Where a pipe always delimited the same number of fields. Otherwise Splunk ignores it.
It certainly doesn't have anything to do with how the lines break since that is an index time action and the DELIMITER in props is a search time action.
One has nothing to do with the other.

You only have seven fields that are consistent.
Extract them from your data that looks like this:

13:01:26.052|0000000|6|31| |O|Accepted connection from gateway
13:01:26.052|0000001|6|31| |O|About to lock connection array
13:01:26.052|0000001|6|31| |O|Connection array locked for GetConnection
13:01:26.052|0000001|6|31| |O|Connection array unlocked having found an available connection
13:01:26.052|0000001|6|31| |O|Sending to Client Key 43(GP B24 UAT), Connect Type 1, on connection 3 [10.172.1.81:3520 -> 64.69.206.9:7000] (LockId = 000000120150318130126052)
13:01:27.661|0000001|6|31| |O|Received response data from operator 43(GP B24 UAT) [125660715320412000AUTH CODE:00365382E38A3F3C886BE83030000800]
13:01:27.661|0000001|6|31| |O|Locking master connection semaphore to release connection
13:01:27.661|0000001|6|31| |O|Connection released.

like this:

EXTRACT-fields =     (?<time>\d{2}:\d{2}:\d{2}\.\d{3})\|(?<audit>[^\|]+)\|(?<serviceType>[^\|]+)\|(?<processId>[^\|]+)\|(?<serviceName>[^\|]+)\|(?<indicator>[^\|]+)\|(?P<message>.+)

You may do this with Splunk running.

Then add the following to your search:

Index=yours sourcetype=yours|extract reload=true

Note there are no spaces in "reload=true"

Now you will see your fields.

Now… use the following search:

…|stats count by punct|sort –count

You will see that there are 63 unique patterns to your data. None of them represent 12 fields.

Now run this:

…|stats count by punct message|stats list(message) as Message  by punct

Now you can see what each individual, unique structure holds.

Within those individual patters, there are a number of delimiters so you could then use the message field to continue to subdivide each one based on it's unique patter with Regex.

Now, if you want to you can add fields extractions like this:

EXTRACT-parse_request = \<Request\>(?<request>[^\<\/Reqest\>]+) in message

[nitesh218ss]

i understand properly thanks sir