topic Re: index only part of a log? in Getting Data In

index only part of a log?

gregwilliams — Wed, 07 Mar 2012 23:02:01 GMT

Is there a way to index only part of a log? These logs are custom log files from a windows server. There are thousands of files, each file belonging to an AD user, and the files are updated when a user logs onto/off of a domain machine. I don't want the entire log to be index because there is a lot of junk in there that I don't need or want to index. For example this is the structure:

Basically I need everything but share directory, AD Display name, macaddress, and the other garbage.

The windows server that the files are sitting on is separate from the splunk server.

Re: index only part of a log?

lguinn2 — Thu, 08 Mar 2012 02:25:26 GMT

Here is the easiest and most efficient way to do it. On the Splunk server, put the following in

$SPLUNK_HOME/etc/system/local/props.conf

[source::nameoffiles]
TRUNCATE = 100

TRUNCATE says "only index the first 100 characters of each event." The 100 is arbitrary, but if you can figure out a reasonable value for your situation, it will work well. And if you sometimes get a a little bit of the "garbage" in there, you can just ignore it.

For the "nameoffiles", you must specify the input files that should be affected. You can use wildcards here, but be careful. And you do need to specify the full path name, etc. Take a look in the manual for more info about props.conf.
You could specify the same thing, but using the sourcetype of the data instead - if the data all has the same sourcetype:

[yoursourcetypename]
TRUNCATE = 100

There are other ways to do this, but they require more processing, etc.

Re: index only part of a log?

gregwilliams — Thu, 08 Mar 2012 16:43:02 GMT

Thanks! The fields are fixed length so this works perfectly for what I'm trying to do.