Getting Data In

Trim whitespace in indexed files

Path Finder

Hi,

We are indexing a substantial number of XML files. These files have between 30% and 50% of white space that can be trimmed with no side effects on the real content of the file.

I was wondering wether it was possible to filter these files for removing white space (really simple regex to apply), before indexing. Can this be done on the UniversalForwarder? On the indexer?

Our aim is reducing the amount of daily indexed data as you can imagine...

Many thanks

0 Karma
1 Solution

Communicator

As said previously, SEDCMD is the way to go. Something like this in props.conf on the indexer:

[sourcetype]
SEDCMD-repws = s/\s+/ /g

This will match on one or more whitespace characters and replace it with one space.

View solution in original post

Communicator

As said previously, SEDCMD is the way to go. Something like this in props.conf on the indexer:

[sourcetype]
SEDCMD-repws = s/\s+/ /g

This will match on one or more whitespace characters and replace it with one space.

View solution in original post

Splunk Employee
Splunk Employee

Although, you might want something like: s/(\s)\s*/\1/g which is more likely to help preserve a line break. (While stripping off indents at the start of a line.)

0 Karma

Splunk Employee
Splunk Employee

You can use the SEDCMD configuration in props.conf to replace whitespace.

http://www.splunk.com/base/Documentation/4.2/Data/Anonymizedatawithsed

0 Karma

SplunkTrust
SplunkTrust

You should be able to do this with a SEDCMD. (But the regex might get complicated). See the docs at http://www.splunk.com/base/Documentation/4.2/Data/Anonymizedatawithsed for info on how to configure this.

If you are using Universal or Light forwarder, the SEDCMD needs to be configured at the indexer. Your whitespace will cross the wire, but will be filtered at the indexer before it writes to the index.

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!