topic Re: Editing logs at index time in Splunk Search

Editing logs at index time

mawwx3 — Mon, 28 Jun 2010 22:56:59 GMT

I have splunk indexing a local file that is being continuously written to and I need the first word in each event to not be indexed so that bluecoat will extract the proper fields.

Here is an example log:

Something 2010-06-25 22:09:45 194 123.123.123.123 - - - OBSERVED "none" 0 TUNNELED unknown - ssl 122.122.122.122 443 - - 123.12.12.123 8676 833 - none - - medium *.fubar.com "Sweet Deals"

I need the "Something" to not be indexed or not show up because it is offsetting the field extractions for the bluecoat app.

Any comments are appreciated.

Thanks,

Re: Editing logs at index time

gkanapathy — Mon, 28 Jun 2010 23:16:29 GMT

You can use the SEDCMD rules in props.conf to do this, something like SEDCMD-something = s/^\S+ //, although you might consider instead setting up a slightly different search-time field extraction for these events to offset/ignore the first field.

http://www.splunk.com/base/Documentation/4.1.3/Admin/Anonymizedatawithsed http://www.splunk.com/base/Documentation/4.1.3/Admin/Propsconf

Re: Editing logs at index time

gkanapathy — Mon, 28 Jun 2010 23:17:47 GMT

I would recommend the search-time extraction change over using SEDCMD in general.

Re: Editing logs at index time

mawwx3 — Tue, 29 Jun 2010 00:25:01 GMT

How would one go about setting up a search-time field extraction to offset or ignore the first field?

Re: Editing logs at index time

Lowell — Tue, 29 Jun 2010 02:38:09 GMT

You would have to find which field extractions were broken and then simply add something like "^\S+ " to the front of them. These will be in props.conf, transforms.conf and possibly in searches in the form of rex commands. (So how did you come to the conclusion that your problem was because of and extra field in the front of your data?)