transforms.conf regex only n characters of a line

[nareshinsvu]

Hi Experts,

I want to filter for a line with a string. But display only first n characters. Note: My input has other lines with a different patterns which should be displayed in full (3rd line in below example)

Regex tried: (?:^.{0,55})(search_me)

test lines from log:

2019-02-20_14:51:27.041 [https-openssl-apr-443-exec-51] DEBUG search_me
2019-02-20_14:55:27.041 [https-openssl-apr-443-exec-51] DEBUG search_me
,responseTime="30secs"
2019-02-20_14:57:27.041 [https-openssl-apr-443-exec-51] DEBUG search_me

Output needed:

2019-02-20_14:51:27.041 [https-openssl-apr-443-exec-51]
2019-02-20_14:55:27.041 [https-openssl-apr-443-exec-51]
,responseTime="30secs"
2019-02-20_14:57:27.041 [https-openssl-apr-443-exec-51]

[whrg]

Hello @nareshinsvu,

Do you want to filter at index-time or search-time?

If at index-time: Check out SEDCMD in props.conf.
Here is the relevant documentation: https://docs.splunk.com/Documentation/Splunk/7.2.4/Data/Anonymizedata

If at search-time:

The eval substr function can get you the first 55 characters:

| makeresults count=1
| eval _raw="2019-02-20_14:51:27.041 [https-openssl-apr-443-exec-51] DEBUG search_me"
| eval _raw=substr(_raw,0,55)

You can also use the rex command. This will cut anything off after the ] character:

| makeresults count=1
| eval _raw="2019-02-20_14:51:27.041 [https-openssl-apr-443-exec-51] DEBUG search_me"
| rex mode=sed "s/\].*/]/g"

[nareshinsvu]

@whrg - yes, I need it at index-time. I somehow tried to copy the SED solution (to replace everything after ] with a ] ) from another blog. But i am still not getting the desired result.

Any tweaks to this SED code to make it work?

SEDCMD-remove_header = s/^]/].*?/g

[whrg]

Is your SEDCMD in your comment displayed correctly? If not, use the Code Sample (101010) formatting.
If is it displayed correctly: ^ refers to the beginning of the line, so ^] does not make any sense. Also, the replacement (in your case ].*?) should be a string and not a regex.

Did you test the regex which I posted above? Like this:

[yoursourcetype]
SEDCMD-remove_header = s/\].*/]/g

You need to put it on your heavy forwarder/indexer. And restart Splunk after making changes to configuration files.

[nareshinsvu]

Hi @whrg,

I am indexing data from a remote shared file. Not using any forwarder. I am getting below error after restarting post changes.As per my request, I also need to extract other lines from my log which doesn't have this pattern.

Invalid key in stanza [monitor://\\data$] in D:\Program Files\Splunk\etc\system\local\inputs.conf, line 23: SEDCMD-remove_header (value: (s/].*/]/g)).

[whrg]

Put the SEDCMD setting in props.conf instead of inputs.conf.

You need to specify the sourcetype in props.conf:

[yoursourcetype]
SEDCMD-remove_header = s/\].*/]/g

Alternatively, you can also refer to the host:

[host::YOURHOST]
SEDCMD-remove_header = s/\].*/]/g