Getting Data In

Trim whitespace in indexed files

oscargarcia
Path Finder

Hi,

We are indexing a substantial number of XML files. These files have between 30% and 50% of white space that can be trimmed with no side effects on the real content of the file.

I was wondering wether it was possible to filter these files for removing white space (really simple regex to apply), before indexing. Can this be done on the UniversalForwarder? On the indexer?

Our aim is reducing the amount of daily indexed data as you can imagine...

Many thanks

0 Karma
1 Solution

bojanz
Communicator

As said previously, SEDCMD is the way to go. Something like this in props.conf on the indexer:

[sourcetype]
SEDCMD-repws = s/\s+/ /g

This will match on one or more whitespace characters and replace it with one space.

View solution in original post

bojanz
Communicator

As said previously, SEDCMD is the way to go. Something like this in props.conf on the indexer:

[sourcetype]
SEDCMD-repws = s/\s+/ /g

This will match on one or more whitespace characters and replace it with one space.

gkanapathy
Splunk Employee
Splunk Employee

Although, you might want something like: s/(\s)\s*/\1/g which is more likely to help preserve a line break. (While stripping off indents at the start of a line.)

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

You can use the SEDCMD configuration in props.conf to replace whitespace.

http://www.splunk.com/base/Documentation/4.2/Data/Anonymizedatawithsed

0 Karma

dwaddle
SplunkTrust
SplunkTrust

You should be able to do this with a SEDCMD. (But the regex might get complicated). See the docs at http://www.splunk.com/base/Documentation/4.2/Data/Anonymizedatawithsed for info on how to configure this.

If you are using Universal or Light forwarder, the SEDCMD needs to be configured at the indexer. Your whitespace will cross the wire, but will be filtered at the indexer before it writes to the index.

0 Karma
Get Updates on the Splunk Community!

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...

DevSecOps: Why You Should Care and How To Get Started

 WATCH NOW In this Tech Talk we will talk about what people mean by DevSecOps and deep dive into the different ...