Reindex entire file when file is updated.

[Lucas_K]

I have already read this older thread on the subject -> : http://splunk-base.splunk.com/answers/5426/entire-file-contents-as-a-single-event

What i'd like to know is if there is a way to reindex the entire file upon change regardless of change type.

Using the method in the link above if you remove or add to the file anywhere inside it (apart from the end) the entire file will be indexed as a separate event (what I want). If I append a single entry to it only that event will show up.

My line breaks are fine (entire file is being indexed as a single event). Its only these additions that seem to break what I am trying to achieve.

[Lucas_K]

well a month on and i'm not closer to getting this to work. As the only other way i've found to do this is fschange and that is a depreciated method that I'd rather not create an entire app around.

I'm finding all the monitor options relate to how a change is detected and not what constitutes the new event inside (via seekptr).

So ... check_method = entire_md5|modtime doesn't actually get the results im after. What i'm really trying to do is somehow set the seekptr to 0. ie. the monitor shouldn't know where it was upto in the file ... thus reindex the entire thing.

If anything in the monitored file is updated EXCEPT for the something including a change on the last line then it works. If its JUST the last line then it doesn't 😞

Its not consistent in its behaviour.

[marcus_doron]

Hi Lucas,
I wonder whether you found a solution for this issue ?

Thanks,

Doron

[Lucas_K]

No, You could have a look inside the configuration audit app on splunkbase and see how they made the input TA.

[miteshp250283]

Have you tried crcSalt= in inputs.conf? Is that something you are looking for?

Here is the link to Docs Search results (http://docs.splunk.com/Special:SplunkSearch/docs?q=crcSalt) that might give more info about it's use and limitations.

Hope this helps.

[Lucas_K]

We'll i could but that could add system dependencies that I was trying to avoid. I also don't get the close to real time monitoring with a scripted input.

I mean the functionality is right there already inside the inputs.conf. And it is already correctly watching and updating what changes are occurring in files in close to realtime.

I'll give the scripted input and see how much I need to wind it up to get what we are after.

I might also have a poke around the pci app and i'm sure I saw a similar feature.

[Ayn]

Well to be fair you ARE bending the functionality a bit. It might be a better idea to do this as some kind of scripted input. That way you're in full control of the solution and can tailor it more to your needs.

[dwaddle]

You may be able to do this using a props.conf setting similar to this:

[source::/path/to/file]
CHECK_METHOD = entire_md5

Unlike most props.conf settings, this needs to be done on the forwarder itself.

(Or as pyro says, "mhhhhhf mhhhhhhhf mhhf mhffff")

[Lucas_K]

just got around to trying out "entire_md5". No dice. 😞

Appended events still show up on their own. For changes in locations anywhere else in the file its fine.

[Lucas_K]

"entire_md5" ... wow. Never seen that one before.