Getting Data In

Trim whitespace in indexed files

oscargarcia
Path Finder

Hi,

We are indexing a substantial number of XML files. These files have between 30% and 50% of white space that can be trimmed with no side effects on the real content of the file.

I was wondering wether it was possible to filter these files for removing white space (really simple regex to apply), before indexing. Can this be done on the UniversalForwarder? On the indexer?

Our aim is reducing the amount of daily indexed data as you can imagine...

Many thanks

0 Karma
1 Solution

bojanz
Communicator

As said previously, SEDCMD is the way to go. Something like this in props.conf on the indexer:

[sourcetype]
SEDCMD-repws = s/\s+/ /g

This will match on one or more whitespace characters and replace it with one space.

View solution in original post

bojanz
Communicator

As said previously, SEDCMD is the way to go. Something like this in props.conf on the indexer:

[sourcetype]
SEDCMD-repws = s/\s+/ /g

This will match on one or more whitespace characters and replace it with one space.

gkanapathy
Splunk Employee
Splunk Employee

Although, you might want something like: s/(\s)\s*/\1/g which is more likely to help preserve a line break. (While stripping off indents at the start of a line.)

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

You can use the SEDCMD configuration in props.conf to replace whitespace.

http://www.splunk.com/base/Documentation/4.2/Data/Anonymizedatawithsed

0 Karma

dwaddle
SplunkTrust
SplunkTrust

You should be able to do this with a SEDCMD. (But the regex might get complicated). See the docs at http://www.splunk.com/base/Documentation/4.2/Data/Anonymizedatawithsed for info on how to configure this.

If you are using Universal or Light forwarder, the SEDCMD needs to be configured at the indexer. Your whitespace will cross the wire, but will be filtered at the indexer before it writes to the index.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...