https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235877#M70063 <P>So - if a double quote exists, there is a pair of them around the message body? If no double quotes, than no "message" ? Just trying to picture it in my head. You can do a conditional pipe in the middle of the regex. Can you describe what would constitute a "message" or at least, what would precede a message? I think I can do it; just need a bit of detail first. </P> Tue, 22 Nov 2016 22:58:41 GMT sshelly_splunk 2016-11-22T22:58:41Z https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235872#M70058 <P>I am trying to extract fields out of events that are tab-delimited unless there are quotes around them. For example,</P> <PRE><CODE>CMSFFMHC 11/21/16 5:19 PM "This is an error message at some point in the code" 16891349 USERNAME 4 function 1234567890 </CODE></PRE> <P>The format is:</P> <PRE><CODE>string -tab- date/time -tab- message -tab- error_code -tab- username -tab- transaction_code -tab- function -tab- transaction_id </CODE></PRE> <P>Sometimes "message" contains a tab character. When it does, it's enclosed in quotes ( "message" ). I'm trying to write a regular expression to extract the message field that would:<BR /> - if there is a quotation mark before message, ignore it<BR /> - capture everything until a tab followed by eight digits<BR /> - if there is a quotation mark at the end of the message, ignore it</P> <P>Further examples:</P> <PRE><CODE>ABCDEFG 11/21/16 11:14 PM The request channel timed out while waiting for a reply after 00:00:17.9375000. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a long 27332747 HOST12345 5 Function1 12345678901234567890123456 ABCDEFG 11/21/16 11:13 PM The request channel timed out while waiting for a reply after 00:00:18. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeo 16964220 HOST23456 5 Function2 23456789012345678901234567 ABCDEFG 11/21/16 5:19 PM " The operationFFMContactLookupfailed. Error Code:[OSB-380000] Reason :[[OSB-381304]Exception in HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.net.SocketTimeoutException java.net.SocketTimeoutException at weblogic.net.http.Sock" 16891349 HOST34567 4 Function3 34567890123456789012345678 ABCDEFG 11/21/16 4:06 PM " The operationFFMContactLookupfailed. Error Code:[OSB-380000] Reason :[[OSB-381304]Exception in HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.net.SocketTimeoutException java.net.SocketTimeoutException at weblogic.net.http.Sock" 16865750 HOST45678 4 Function4 45678901234567890123456789 </CODE></PRE> <P>Here is my current field extraction in props.conf:</P> <P>EXTRACT-message=^\w+\t[^\t]+\t(?P[^\t]+)<BR /> The problem is this keeps the quotation marks and stops at the tab character in the midst of the message.</P> <P>Can someone more versed in PCRE please lend me a hand?</P> <P>Thanks!</P> Tue, 22 Nov 2016 17:11:00 GMT https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235872#M70058 jwalthour 2016-11-22T17:11:00Z https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235873#M70059 <P>Can u post at least 2 examples of the event data with "messages" fields ?</P> Tue, 22 Nov 2016 17:36:12 GMT https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235873#M70059 sshelly_splunk 2016-11-22T17:36:12Z https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235874#M70060 <P>Sorry - just re-read your post. This should suffice, I think:<BR /> <CODE>"(?P&lt;message&gt;.+)"</CODE><BR /> this will extract everything between double quote marks and assign the field named "message". (remove the single quote marks)<BR /> If you can post the contents of your props.conf and a few more sample events, we can review and make sure it will work.</P> Tue, 22 Nov 2016 17:41:11 GMT https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235874#M70060 sshelly_splunk 2016-11-22T17:41:11Z https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235875#M70061 <P>There's only one double quote in your events above. Can u edit maybe? As well, if you could include the props.conf entries and/or transforms that come into play, that would help. If you're doing a tab separated extract, you might need to get more specific using transforms since tabs can appear within field values.</P> Tue, 22 Nov 2016 18:22:16 GMT https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235875#M70061 sshelly_splunk 2016-11-22T18:22:16Z https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235876#M70062 <P>Thank you, sshelly, but your suggestion doesn't work. The regex should only consider the quotations if they are there, which most of the time they are not. More importantly, it needs to stop at the next tab, unless the tab is part of the message when it is enclosed in quotation marks.</P> Tue, 22 Nov 2016 18:53:13 GMT https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235876#M70062 jwalthour 2016-11-22T18:53:13Z https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235877#M70063 <P>So - if a double quote exists, there is a pair of them around the message body? If no double quotes, than no "message" ? Just trying to picture it in my head. You can do a conditional pipe in the middle of the regex. Can you describe what would constitute a "message" or at least, what would precede a message? I think I can do it; just need a bit of detail first. </P> Tue, 22 Nov 2016 22:58:41 GMT https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235877#M70063 sshelly_splunk 2016-11-22T22:58:41Z https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235878#M70064 <P><CODE>\"(?P&lt;message&gt;[^\"]+)</CODE> should give you a good start.<BR /> Sometimes the rex can be funny with quote marks, you may end up having to add in multiples e.g. \" or \\"</P> Tue, 22 Nov 2016 23:02:45 GMT https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235878#M70064 mrgibbon 2016-11-22T23:02:45Z https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235879#M70065 <P>How about something like this?</P> <PRE><CODE>^[^\t]+\t[^\t]+\t\"?(?&lt;message&gt;[^\"]+)\"?\t\d{8}\t </CODE></PRE> Wed, 23 Nov 2016 01:35:38 GMT https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235879#M70065 maciep 2016-11-23T01:35:38Z https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235880#M70066 <P>See if this regex suffices which will captureall the cases you mentioned:</P> <PRE><CODE>^(?&lt;first&gt;[^\t]+)\t(?&lt;second&gt;[^\t]+)(\t|\t")(?&lt;message&gt;(.*))("\t|\t)\d{8}\t </CODE></PRE> <P>See extraction at work <A href="https://regex101.com/r/4hyefK/1">here</A></P> Wed, 23 Nov 2016 05:31:04 GMT https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235880#M70066 gokadroid 2016-11-23T05:31:04Z https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235881#M70067 <P>Like this:</P> <PRE><CODE>| rex "(?ms)^(?:[^\t]*\t){2}\"?(?&lt;message&gt;.+)\"?\t(?:[^\t]*\t){4}[^\t]*$" </CODE></PRE> Sat, 11 Mar 2017 21:09:59 GMT https://community.splunk.com/t5/Splunk-Search/How-to-write-a-regular-expression-to-extract-a-long-string-until/m-p/235881#M70067 woodcock 2017-03-11T21:09:59Z