topic Ignore data after CRLF in Getting Data In

Ignore data after CRLF

donemery — Wed, 20 Jun 2018 17:58:55 GMT

When running a search the _raw field returns results that typically end with the ] character. On a rare occasion the search has a CRLF and additional lines of data. I am trying to figure a way to ignore anything beyond the last ] or after a CRLF.

Example 1:
Jun 14 13:00:00 -07:00 HOSTNAME condition is inside brackets

Example 2:
Jun 14 13:00:00 -07:00 HOSTNAME [condition is inside brackets]
Jun 14 13:00:00 -07:00 HOSTNAME [unexpected results here]

My goal in example 2 is to only return the results of the first line.

Thanks

Re: Ignore data after CRLF

DalJeanis — Thu, 21 Jun 2018 00:14:30 GMT

I've posted this as a little regex challenge in the #regex channel of the splunk slack channel. you should get a way to do this shortly.

the general form of this is

| rex mode=sed field=_raw "s/thing you want to change/thing you want to change it to/g"

...so in between the first two /, you need to match CRLF and then everything to the end of the event, and then put nothing between the second and third /.

Re: Ignore data after CRLF

chrisyounger — Thu, 21 Jun 2018 00:19:22 GMT

Give this a try:

| rex "\](?:\s*\((?<results_normal>[^\)]*)|[^\[]+\[(?<result_unusual>[^\]]*))" | table results_normal result_unusual _raw

Re: Ignore data after CRLF

chrisyounger — Thu, 21 Jun 2018 00:27:58 GMT

Actually this might be closer to what you need if you just want to delete everything after the first line:

| rex mode=sed field=_raw "s/\r?\n[\s\S]*//"

Re: Ignore data after CRLF

chrisyounger — Thu, 21 Jun 2018 00:33:34 GMT

Just in case that one doesn't work, give this a try too:
| rex mode=sed field=_raw "s/][\s\S]*/]/"

Re: Ignore data after CRLF

donemery — Thu, 21 Jun 2018 14:40:50 GMT

Both options worked for me. Thanks so much for your help!

Re: Ignore data after CRLF

woodcock — Sat, 30 Jun 2018 23:42:54 GMT

You are not going to like this answer but the problem is not in your searching, it is in your forwarding. These events are not properly line-broken. Somebody has either redefined LINE_BREAKER or turned on SHOULD_LINEMERGE = true. You need to find the props.conf settings on the Indexers or HF for this sourcetype and fix the line-breaking. Then you will not have this problem AND the data will be correct for everybody, without mis-joined events.

Re: Ignore data after CRLF

donemery — Fri, 20 Jul 2018 15:07:55 GMT

Thanks for your help! We were working on fixing the configuration issue as well and fortunately it is resolved now.