Getting Data In

index only part of a log?

gregwilliams
Path Finder

Is there a way to index only part of a log? These logs are custom log files from a windows server. There are thousands of files, each file belonging to an AD user, and the files are updated when a user logs onto/off of a domain machine. I don't want the entire log to be index because there is a lot of junk in there that I don't need or want to index. For example this is the structure:

Date | Time | user | macaddress | IP | Function | computer name | AD Display Name | share directory | then a bunch of other garbage | etc.

Basically I need everything but share directory, AD Display name, macaddress, and the other garbage.

The windows server that the files are sitting on is separate from the splunk server.

1 Solution

lguinn2
Legend

Here is the easiest and most efficient way to do it. On the Splunk server, put the following in

$SPLUNK_HOME/etc/system/local/props.conf

[source::nameoffiles]
TRUNCATE = 100

TRUNCATE says "only index the first 100 characters of each event." The 100 is arbitrary, but if you can figure out a reasonable value for your situation, it will work well. And if you sometimes get a a little bit of the "garbage" in there, you can just ignore it.

For the "nameoffiles", you must specify the input files that should be affected. You can use wildcards here, but be careful. And you do need to specify the full path name, etc. Take a look in the manual for more info about props.conf.
You could specify the same thing, but using the sourcetype of the data instead - if the data all has the same sourcetype:

[yoursourcetypename]
TRUNCATE = 100

There are other ways to do this, but they require more processing, etc.

View solution in original post

gregwilliams
Path Finder

Thanks! The fields are fixed length so this works perfectly for what I'm trying to do.

0 Karma

lguinn2
Legend

Here is the easiest and most efficient way to do it. On the Splunk server, put the following in

$SPLUNK_HOME/etc/system/local/props.conf

[source::nameoffiles]
TRUNCATE = 100

TRUNCATE says "only index the first 100 characters of each event." The 100 is arbitrary, but if you can figure out a reasonable value for your situation, it will work well. And if you sometimes get a a little bit of the "garbage" in there, you can just ignore it.

For the "nameoffiles", you must specify the input files that should be affected. You can use wildcards here, but be careful. And you do need to specify the full path name, etc. Take a look in the manual for more info about props.conf.
You could specify the same thing, but using the sourcetype of the data instead - if the data all has the same sourcetype:

[yoursourcetypename]
TRUNCATE = 100

There are other ways to do this, but they require more processing, etc.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...