topic Re: Netapp XML audit data makes the file monitor stop reading before end of file. Is there a workaround? in Dashboards & Visualizations

Netapp XML audit data makes the file monitor stop reading before end of file. Is there a workaround?

jplumsdaine22 — Thu, 28 Jan 2016 16:54:57 GMT

We are trying to index NetApp XML audit logs. The look like this

<Events xmlns=blah>
<Event>stuff</Event>
<Event>stuff</Event>
<Event>stuff</Event>
<Event>stuff</Event>
</Events>

Unfortunately, new events are INSERTED BEFORE the final tag.
So the next time it reads the file, the value of scrc will be different and Splunk reindexes the entire file. The error message is "Checksum for seekptr didn't match, will re-read entire file"

This is expected behaviour from Splunk, but I'm wondering if anyone has managed to work around it? One method that comes to mind is can the seekptr be told to ignore a regex? If it ignored the final element, then there would be no scrc mismatch.

Any ideas?

Re: Netapp XML audit data makes the file monitor stop reading before end of file. Is there a workaround?

geraldomagella — Fri, 13 May 2016 19:45:36 GMT

Hello! Have you fixed it?

Are you monitoring the files directly? How about exclude the "filerXYZ-last.xml" and getting only the one that was already rotated? That could fix your issue, right?
You could set it to rotate every 5 minutes (or something like that) and keep only the 10, 15 files (loglimit).
If you need the cDot commands, I can help you with this.

Re: Netapp XML audit data makes the file monitor stop reading before end of file. Is there a workaround?

kapanig — Fri, 20 May 2016 17:39:42 GMT

Did you ever fix this? I have tried (to no avail):

INPUTS (only monitoring last .xml file)

[ontap]
initCrcLength = 2048
multiline_event_extra_waittime = true
disabled = 0
sourcetype = ontap
index = ontap

PROPS:

[ontap]
SHOULD_LINEMERGE = false
KV_MODE = xml
LINE_BREAKER = ()
MUST_BREAK_AFTER = \
TRANSFORMS-t1 = remove_header_footer

TRANSFORMS:

[remove_header_footer]
REGEX=^<(\/|)Events(\s|>)
DEST_KEY = queue
FORMAT = nullQueue

Still getting

WatchedFile - Checksum for seekptr didn't match, will re-read entire file=(.xml file)
WatchedFile - Will begin reading at offset=0 for file= (.xml file)

Re: Netapp XML audit data makes the file monitor stop reading before end of file. Is there a workaround?

kapanig — Fri, 20 May 2016 17:41:05 GMT

CORRECTION:
LINE_BREAKER = (<Event>)
MUST_BREAK_AFTER = \</Event\>

Re: Netapp XML audit data makes the file monitor stop reading before end of file. Is there a workaround?

jplumsdaine22 — Wed, 25 May 2016 14:34:48 GMT

Hi,

We did in fact end up reading the rotated file. Works fine, but we miss being able to get real time info.

Re: Netapp XML audit data makes the file monitor stop reading before end of file. Is there a workaround?

jplumsdaine22 — Wed, 25 May 2016 14:35:32 GMT

We ended up reading the rotated log files instead of the live file, as there is no way to manipulate seekptr