Getting Data In

Is it possible to prevent indexing part of a line in a log file?

machiel
Path Finder

I know it is possible to skip lines in an input, however, I have the case where I want to skip part of a line.

For example, I have an inputs.conf stanza like the following:

[monitor://C:\temp\example.log]
...

And I have the following log file, example.log:

time/fieldb/fieldc 13:50,200,300
time/fieldb/fieldc 14:00,210,310
time/fieldb/fieldc 14:10,223,305
time/fieldb/fieldc 14:20,215,307
...

I want to only index the part after the space, due to having the index size as small as possible.
Is it possible to somehow skip the "time/fieldb/fieldc"-part from being indexed?

0 Karma

somesoni2
Revered Legend

It is possible to remove/replace certain part of the data before it's indexed. It's generally used for masking of sensitive data. I would throw a caution before going into details that it add overhead to indexer/heavy forwarder as Splunk now has to to additional processing of each event.

You can use SEDCMD script in props.conf OR add a transforms.conf to achieve the same. See this for more details on data masking
http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/Anonymizedata

0 Karma

machiel
Path Finder

Do you have an example explaining how to use the SEDCMD here? That would help me a lot.

0 Karma

somesoni2
Revered Legend

In your props.conf on INdexer/Heavy Forwarder (if any), add this to your sourcetype.

props.conf (on Indexer/Heavy Forwarder)

[YourSourceType]
..other settings..
SEDCMD-removeheader = s/^(\S+\s+)(.*)/\2/g
0 Karma
Get Updates on the Splunk Community!

Infographic provides the TL;DR for the 2024 Splunk Career Impact Report

We’ve been buzzing with excitement about the recent validation of Splunk Education! The 2024 Splunk Career ...

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...