topic Re: Trim whitespace in indexed files in Getting Data In

Trim whitespace in indexed files

oscargarcia — Fri, 15 Apr 2011 16:20:23 GMT

Hi,

We are indexing a substantial number of XML files. These files have between 30% and 50% of white space that can be trimmed with no side effects on the real content of the file.

I was wondering wether it was possible to filter these files for removing white space (really simple regex to apply), before indexing. Can this be done on the UniversalForwarder? On the indexer?

Our aim is reducing the amount of daily indexed data as you can imagine...

Many thanks

Re: Trim whitespace in indexed files

dwaddle — Fri, 15 Apr 2011 16:34:08 GMT

You should be able to do this with a SEDCMD. (But the regex might get complicated). See the docs at http://www.splunk.com/base/Documentation/4.2/Data/Anonymizedatawithsed for info on how to configure this.

If you are using Universal or Light forwarder, the SEDCMD needs to be configured at the indexer. Your whitespace will cross the wire, but will be filtered at the indexer before it writes to the index.

Re: Trim whitespace in indexed files

Stephen_Sorkin — Fri, 15 Apr 2011 16:35:28 GMT

You can use the SEDCMD configuration in props.conf to replace whitespace.

http://www.splunk.com/base/Documentation/4.2/Data/Anonymizedatawithsed

Re: Trim whitespace in indexed files

bojanz — Fri, 15 Apr 2011 18:30:25 GMT

As said previously, SEDCMD is the way to go. Something like this in props.conf on the indexer:

[sourcetype]
SEDCMD-repws = s/\s+/ /g

This will match on one or more whitespace characters and replace it with one space.

Re: Trim whitespace in indexed files

gkanapathy — Sat, 16 Apr 2011 03:36:31 GMT

Although, you might want something like: s/(\s)\s*/\1/g which is more likely to help preserve a line break. (While stripping off indents at the start of a line.)