topic Re: Skip lines/events during log rotation in Getting Data In

Is it possible to Skip lines/events during log rotation?

fatsug — Mon, 18 Jul 2022 14:22:11 GMT

Hello community

We are ingesting sftp log. The logfile rotates once every 24h. "headers" are set in the new file every rotation which gets indexed.

Unlike every other event indexed, the "linecount" for this event is 2 instead of 1 so they are pretty easy to spot.

#Date: Mon Jan 10 00:00:00 CEST 2020 
#Fields: date time ip port .........

I've seen examples regarding skipping header lines in CSV files, though this is a textfile. It is not a huge issue though still something which is a bit irritating.

Is it possible to skip these lines so they are not forwarded/indexed? How would I go about accomplishing this?

Thank you in advace

Re: Skip lines/events during log rotation

JacekF — Mon, 18 Jul 2022 11:17:23 GMT

You can configure Splunk to drop events (send them to nullQueue), based on regex. You can find details in the Splunk documentation:
Route and filter data - Splunk Documentation

Re: Skip lines/events during log rotation

fatsug — Mon, 18 Jul 2022 11:32:54 GMT

Hi and thanks

Hm, so basically I could do something like:

props.conf

[source::/my/source/here]
TRANSFORMS-null= setnull

transforms.conf

[setnull]
REGEX = ^#[a-zA-Z]+: 
DEST_KEY = queue
FORMAT = nullQueue

In the same files where I define field extraction? Currently this TA lives on the search heads and the universal forwarder collecting the log. Do I need this TA anywhere else or would that be enough?

Thank you again

Re: Skip lines/events during log rotation

JacekF — Mon, 18 Jul 2022 11:41:43 GMT

As per documentation: "Although similar to forwarder-based routing, queue routing can be performed by an indexer, as well as a heavy forwarder." Which means that you need to create a TA and deploy it to the indexer(s) or heavy forwarder (if your are using it).

The transforms file should be ok, if you are sure that events you want to keep, will not match provided REGEX.

Re: Skip lines/events during log rotation

fatsug — Mon, 18 Jul 2022 11:54:03 GMT

Ah, the "caveat" at the end...

So yeah, I need to deploy the TA to the indexers to "skip" these header events once per 24h. Not sure I understand the manual here 100% though. Is it enough if this config is present on indexers and heavy forwarders, or should I push this to universal forwarder and search heads as well?

Regarding the regex, no events should ever start with # for this source, so that should be OK.

Thank you again! Fantastic feedback

Re: Skip lines/events during log rotation

JacekF — Mon, 18 Jul 2022 11:56:59 GMT

You only need those props and transforms conf files on indexers/heavy forwarders.

Re: Skip lines/events during log rotation

fatsug — Mon, 18 Jul 2022 11:58:53 GMT

Fantastic!

I'll mark the initial reply as the solution and se if I can get the configuration deployed. Fingers crossed