topic Re: Reindex entire file when file is updated. in Getting Data In

Reindex entire file when file is updated.

Lucas_K — Thu, 23 May 2013 06:02:56 GMT

I have already read this older thread on the subject -> : http://splunk-base.splunk.com/answers/5426/entire-file-contents-as-a-single-event

What i'd like to know is if there is a way to reindex the entire file upon change regardless of change type.

Using the method in the link above if you remove or add to the file anywhere inside it (apart from the end) the entire file will be indexed as a separate event (what I want). If I append a single entry to it only that event will show up.

My line breaks are fine (entire file is being indexed as a single event). Its only these additions that seem to break what I am trying to achieve.

Re: Reindex entire file when file is updated.

dwaddle — Thu, 23 May 2013 18:54:17 GMT

You may be able to do this using a props.conf setting similar to this:

[source::/path/to/file]
CHECK_METHOD = entire_md5

Unlike most props.conf settings, this needs to be done on the forwarder itself.

(Or as pyro says, "mhhhhhf mhhhhhhhf mhhf mhffff")

Re: Reindex entire file when file is updated.

Lucas_K — Fri, 24 May 2013 00:32:55 GMT

"entire_md5" ... wow. Never seen that one before.

Re: Reindex entire file when file is updated.

Lucas_K — Sun, 26 May 2013 23:24:04 GMT

just got around to trying out "entire_md5". No dice. 😞

Appended events still show up on their own. For changes in locations anywhere else in the file its fine.

Re: Reindex entire file when file is updated.

Lucas_K — Mon, 28 Sep 2020 14:09:30 GMT

well a month on and i'm not closer to getting this to work. As the only other way i've found to do this is fschange and that is a depreciated method that I'd rather not create an entire app around.

I'm finding all the monitor options relate to how a change is detected and not what constitutes the new event inside (via seekptr).

So ... check_method = entire_md5|modtime doesn't actually get the results im after. What i'm really trying to do is somehow set the seekptr to 0. ie. the monitor shouldn't know where it was upto in the file ... thus reindex the entire thing.

If anything in the monitored file is updated EXCEPT for the something including a change on the last line then it works. If its JUST the last line then it doesn't 😞

Its not consistent in its behaviour.

Re: Reindex entire file when file is updated.

Ayn — Mon, 24 Jun 2013 07:19:52 GMT

Well to be fair you ARE bending the functionality a bit. It might be a better idea to do this as some kind of scripted input. That way you're in full control of the solution and can tailor it more to your needs.

Re: Reindex entire file when file is updated.

Lucas_K — Mon, 24 Jun 2013 09:18:30 GMT

We'll i could but that could add system dependencies that I was trying to avoid. I also don't get the close to real time monitoring with a scripted input.

I mean the functionality is right there already inside the inputs.conf. And it is already correctly watching and updating what changes are occurring in files in close to realtime.

I'll give the scripted input and see how much I need to wind it up to get what we are after.

I might also have a poke around the pci app and i'm sure I saw a similar feature.

Re: Reindex entire file when file is updated.

marcus_doron — Tue, 01 Nov 2016 14:29:37 GMT

Hi Lucas,
I wonder whether you found a solution for this issue ?

Thanks,

Doron

Re: Reindex entire file when file is updated.

Lucas_K — Tue, 01 Nov 2016 22:16:44 GMT

No, You could have a look inside the configuration audit app on splunkbase and see how they made the input TA.

Re: Reindex entire file when file is updated.

miteshp250283 — Wed, 02 Nov 2016 04:45:37 GMT

Have you tried crcSalt= in inputs.conf? Is that something you are looking for?

Here is the link to Docs Search results (http://docs.splunk.com/Special:SplunkSearch/docs?q=crcSalt) that might give more info about it's use and limitations.

Hope this helps.