topic Re: Removing Garbled logs by linecount before indexing in Getting Data In

Removing Garbled logs by linecount before indexing

twistedsixty4 — Mon, 24 Feb 2014 18:31:24 GMT

Hey everyone,
So I have a script that generates status logs for a few network devices that my Splunk forwarder grabs, but every now and again something screws it up and it garbles the entry. The garbled entry is always a smaller linecount, this accounts for roughly 1% of my logs from the script. Is there a way to tell Splunk in the props file to throw away the entry if it's less than 23 lines long? Thanks!

Re: Removing Garbled logs by linecount before indexing

yannK — Mon, 24 Feb 2014 18:37:59 GMT

Not if you monitor the log file with a [monitor]
and the nullQueue filtering will only work on a line per line basis.

If you were using a scripted input, then you could add some logic when you generate the logs to drop the invalid ones.

Re: Removing Garbled logs by linecount before indexing

somesoni2 — Mon, 24 Feb 2014 18:58:01 GMT

yes, something like if result count < 23 don't write anything (to file or to script output)

Re: Removing Garbled logs by linecount before indexing

twistedsixty4 — Mon, 24 Feb 2014 22:54:34 GMT

thanks for your help, i've adjusted my script, oh well.