topic Re: using SEDCMD to remove repeated lines in Getting Data In

using SEDCMD to remove repeated lines

twistedsixty4 — Tue, 25 Feb 2014 00:06:52 GMT

hey everyone,
Our server here generates a filestamp/header at midnight or on resets that start with a line of dashes(----), then enters a line of system and log information (this is useful), then ends with another line of dashes (----). I've been trying to use SEDCMD to delete the dashed lines (including the carriage return) but it doesnt seem to be working.

SEDCMD-StripBreaks = s/\n?----*//g

I've tried changing the name for the class, tried removing the "\n", double escaping the "\n", all of it just doesn't work. One thing that works is if I try it inline..

| rex mode=sed "s/\n?----*//g"

I could use some help getting my head around this..

Update:

so I've tried to do this with a different issue I'm having which is repeated spaces, if I use the rex sed mode it works just fine, but the second i add it to props it falls apart. here's the second one I've tried..

SEDCMD-SubSpaces = s/\s\s+/ /g

beside this i've tried changing how the SED is applied by giving it a source file instead of the sourcetype and it still doesnt work. here is the working rex for my second SEDCMD.

rex mode=sed "s/\s\s+/ /g"

again, any help is appreciated!

Re: using SEDCMD to remove repeated lines

dshpritz — Tue, 25 Feb 2014 21:35:56 GMT

From a first look, the problem may be the props.conf syntax.

You have:

SED-StripBreaks = s/\n?----*//g

You may want:

SEDCMD-StripBreaks = s/\n?----*//g

Additionally, if the headers are showing up as individual events, you may want to look into using a null queue routing:

Route and filter data

Re: using SEDCMD to remove repeated lines

twistedsixty4 — Tue, 25 Feb 2014 22:24:48 GMT

sorry that was a problem with my syntax copying over, i did have SEDCMD there, im working off an airgapped network so i cant copy/paste. i updated with new information.

Re: using SEDCMD to remove repeated lines

dshpritz — Tue, 25 Feb 2014 22:28:07 GMT

Can you provide an example of what the header like looks like?

Re: using SEDCMD to remove repeated lines

twistedsixty4 — Tue, 25 Feb 2014 22:33:55 GMT

its one line of dashes, maybe 30 or 40 dashes long, and its stamped throughout the log because it points to another file. the whole line is nothing but dashes, then a path to a file, then another line of dashes.

Re: using SEDCMD to remove repeated lines

dshpritz — Tue, 25 Feb 2014 22:37:36 GMT

Are these header lines showing up as individual events in Splunk?

Re: using SEDCMD to remove repeated lines

twistedsixty4 — Tue, 25 Feb 2014 22:43:22 GMT

no, since they have no timestamp splunk sees them as part of the log before it, which is fine.

Re: using SEDCMD to remove repeated lines

Lowell — Tue, 25 Feb 2014 22:48:43 GMT

I'm assuming the content you want to strip looks something like this? And that you want to remove the solid lines and keep the line in the middle.

------------------------------------------------
some other log file ...
------------------------------------------------

Then something like this should work:

[your-source-type]
SEDCMD-StripBreaks = s/----+[\r\n]*//g

This will handle different EOL combinations (CR/CRLF/LF) and instead of consuming the leading EOL, it will remove the end. (Unless you have lines with random trailing dashes, that you would like to keep, this should work fine.)

Just to be clear, this is index-time setting which means (1) existing events already indexed will NOT be updated, and (2) you must restart Splunk for this setting to take effect.

Re: using SEDCMD to remove repeated lines

twistedsixty4 — Tue, 25 Feb 2014 23:33:32 GMT

this worked! but your points at the end helped the most, I was stopping and restarting my server, but wasn't cleaning it, now that i think about it it makes a lot of sense to do that. thanks for your help!

Re: using SEDCMD to remove repeated lines

maria1991 — Wed, 29 Dec 2021 13:42:46 GMT

What if I need to remove the line in between the ---- lines as well?

Please suggest @Lowell and @dshpritz

Re: using SEDCMD to remove repeated lines

dshpritz — Wed, 29 Dec 2021 23:15:41 GMT

Can you provide an example?

Re: using SEDCMD to remove repeated lines

maria1991 — Thu, 30 Dec 2021 07:34:42 GMT

-----------------------------------------------------------------

some text

-----------------------------------------------------------------

These lines won't have any timestamps, need to complete ignore them from indexing.

Re: using SEDCMD to remove repeated lines

splunKR1 — Fri, 07 Jan 2022 11:21:35 GMT

Hi @dshpritz

-----------------------------------------------------------------

some text

-----------------------------------------------------------------

These lines won't have any timestamps, need to complete ignore them from indexing.

Re: using SEDCMD to remove repeated lines

dshpritz — Sat, 08 Jan 2022 03:18:27 GMT

I think something like this would work:

SEDCMD-turnthismotherout = s/(?:^-+[\n\r$])(?:.+?[\r\n])(?:^-+[\n\r$])//g