Getting Data In

Reindex entire file when file is updated.

Lucas_K
Motivator

I have already read this older thread on the subject -> : http://splunk-base.splunk.com/answers/5426/entire-file-contents-as-a-single-event

What i'd like to know is if there is a way to reindex the entire file upon change regardless of change type.

Using the method in the link above if you remove or add to the file anywhere inside it (apart from the end) the entire file will be indexed as a separate event (what I want). If I append a single entry to it only that event will show up.

My line breaks are fine (entire file is being indexed as a single event). Its only these additions that seem to break what I am trying to achieve.

Lucas_K
Motivator

well a month on and i'm not closer to getting this to work. As the only other way i've found to do this is fschange and that is a depreciated method that I'd rather not create an entire app around.

I'm finding all the monitor options relate to how a change is detected and not what constitutes the new event inside (via seekptr).

So ... check_method = entire_md5|modtime doesn't actually get the results im after. What i'm really trying to do is somehow set the seekptr to 0. ie. the monitor shouldn't know where it was upto in the file ... thus reindex the entire thing.

If anything in the monitored file is updated EXCEPT for the something including a change on the last line then it works. If its JUST the last line then it doesn't 😞

Its not consistent in its behaviour.

0 Karma

marcus_doron
New Member

Hi Lucas,
I wonder whether you found a solution for this issue ?

Thanks,

Doron

0 Karma

Lucas_K
Motivator

No, You could have a look inside the configuration audit app on splunkbase and see how they made the input TA.

0 Karma

miteshp250283
Path Finder

Have you tried crcSalt= in inputs.conf? Is that something you are looking for?

Here is the link to Docs Search results (http://docs.splunk.com/Special:SplunkSearch/docs?q=crcSalt) that might give more info about it's use and limitations.

Hope this helps.

0 Karma

Lucas_K
Motivator

We'll i could but that could add system dependencies that I was trying to avoid. I also don't get the close to real time monitoring with a scripted input.

I mean the functionality is right there already inside the inputs.conf. And it is already correctly watching and updating what changes are occurring in files in close to realtime.

I'll give the scripted input and see how much I need to wind it up to get what we are after.

I might also have a poke around the pci app and i'm sure I saw a similar feature.

0 Karma

Ayn
Legend

Well to be fair you ARE bending the functionality a bit. It might be a better idea to do this as some kind of scripted input. That way you're in full control of the solution and can tailor it more to your needs.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

You may be able to do this using a props.conf setting similar to this:

[source::/path/to/file]
CHECK_METHOD = entire_md5

Unlike most props.conf settings, this needs to be done on the forwarder itself.

(Or as pyro says, "mhhhhhf mhhhhhhhf mhhf mhffff")

Lucas_K
Motivator

just got around to trying out "entire_md5". No dice. 😞

Appended events still show up on their own. For changes in locations anywhere else in the file its fine.

0 Karma

Lucas_K
Motivator

"entire_md5" ... wow. Never seen that one before.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!