topic Re: Splunk Reindexes File that gets a new first line when closed in Getting Data In

Splunk Reindexes File that gets a new first line when closed

TiagoMatos — Sat, 23 Apr 2016 20:16:22 GMT

Hello,

My problem is simple to explain: I have an app that generates logs that are written whenever a new action is performed.

The problem is, when the session is over, the first line of that log is changed to include the close time of the session, which makes splunk REINDEX everything on the log.

Any ideas?

Thanks

Re: Splunk Reindexes File that gets a new first line when closed

martin_mueller — Sat, 23 Apr 2016 21:25:46 GMT

From Splunk's point of view, that's intended behaviour. If you're tailing a log file and its start changes, it's considered to be a new log file.

Have your application move the file to a different place after completion, and monitor that different place with Splunk.

Re: Splunk Reindexes File that gets a new first line when closed

bentleymi — Sun, 24 Apr 2016 14:02:51 GMT

If the events are being written in near real time you could just use DATETIME_CONFIG=NONE in your props so that Splunk will use the time the file is written as the event date.

Re: Splunk Reindexes File that gets a new first line when closed

martin_mueller — Sun, 24 Apr 2016 14:07:32 GMT

Timestamp extraction won't influence reindexing files that have their beginning changed later on.

Re: Splunk Reindexes File that gets a new first line when closed

jkat54 — Sun, 24 Apr 2016 14:12:00 GMT

Yeah I'm trying to edit he could try to decrease Crc init to a lower value possibly too.

Re: Splunk Reindexes File that gets a new first line when closed

martin_mueller — Sun, 24 Apr 2016 14:24:03 GMT

I doubt that would work. From Splunk's point of view, a file is unchanged if the start CRC, the end CRC, and the size doesn't change. Adding a session end to the start would inevitably change the size. If the size changes in any context other than appending to the end, Splunk will reindex.

Re: Splunk Reindexes File that gets a new first line when closed

TiagoMatos — Sun, 24 Apr 2016 14:27:02 GMT

Firstly, Thanks for the suggestion. It would avoid a real time monitor of the logo content though.. because splunk would only see the file when it would be full and closed... Nota an óption for our case

Re: Splunk Reindexes File that gets a new first line when closed

jkat54 — Sun, 24 Apr 2016 14:27:30 GMT

Can you show us what the first line of the file looks like before and after the process is completed?
Maybe we can use props to delete events when they start with one or the other.

Re: Splunk Reindexes File that gets a new first line when closed

TiagoMatos — Sun, 24 Apr 2016 14:28:38 GMT

I agree with you, I feel my hands are tied on this because I cant avoid it to check the beggining of the file

Re: Splunk Reindexes File that gets a new first line when closed

martin_mueller — Sun, 24 Apr 2016 14:55:43 GMT

Mkay, here's another option: Have the application append the session end time to the end of the log file.

Re: Splunk Reindexes File that gets a new first line when closed

mattymo — Sun, 24 Apr 2016 16:15:27 GMT

Your hands are far from tied here, heck they aren't even dirty yet! 😉

martin_mueller's suggestion of moving this file's data to a different location for monitoring is the "best" option here without changing your app's log behavior and does not stop you from maintaining a "realtime" view of the data.

Your file is fundamentally not fit for direct monitor by splunk without dealing with re-index of data/dedup of duplicate events (which is a completely viable option if license/filesize allow). So now what???

Some factors on how to attack this include:

What operating system are you dealing with? *nix? windows?
What Admin/scripting skills are available to you?
How long are these files open and how many do you have at any one time? How big are they?
How often are events written to this file?

I would begin with the option of tailing the writes to this file to another file. Depending on the specifics of your app, it isn't all that challenging to write a small script to maintain a copy of this file at all times. This allows you to control the behavior of the data while maintaining a "realtime" look at this file.

Heck, even a cron job that simply checks this file at a tiny interval and does a cp myUglyFile.log >> myBeautifulSplunkFormattedLog.log then cleans up dupe lines before splunk ingestion could work....

It will take some solutioneering, but this is totaly doable..many data sources require a bit of massaging to help splunk spend its time doing what it does best

Re: Splunk Reindexes File that gets a new first line when closed

TiagoMatos — Sun, 24 Apr 2016 16:23:12 GMT

Hi mmodestino,

Of course I can use scripts and lots of other stuff to get around this. The objective though is to keep everything on splunk side! And this is the point here, it looks like for now I have to change things other than splunk conf files.

So I am using a Splunk HF on WINDOWS to process these files and sends them to an Splunk Indexer on CentOS. I can do explore administrational and scripting skills to solve this.

The files can be opened for hours, can be written every second or be left unchanged for minutes.

Everything can be done right? This is just something I would like to do ONLY on Splunk, without scripts!

Thank you for your suggestions

P.S.: copying thousands of files with many Gb is not a viable option. Neither cutting them

Re: Splunk Reindexes File that gets a new first line when closed

mattymo — Sun, 24 Apr 2016 17:34:47 GMT

Fair enough...I have heard many people say the same thing...eventually every admin figures out the difference between what you CAN do in splunk and what you SHOULD do...

Before you look to bend splunk to the behaviour of some other app, then i would recommend studying how splunk monitors a file:

http://docs.splunk.com/Documentation/Splunk/6.4.0/Data/Howlogfilerotationishandled

And then I would suggest seeing if you can shrink the crc check on this input to a length that would keep splunks ability to check for unique files, but not check far enough to see what ur app writes to the top of the file on session close

can u share the first 256 bytes of a new file and a closed file??

At the end of the day this is not a logfile and saying "i want to do it on the splunk side" when talking about something splunk doesnt natively do, is going to require more than some conf file tweaks.

Re: Splunk Reindexes File that gets a new first line when closed

TiagoMatos — Sun, 24 Apr 2016 17:42:42 GMT

I'm definitly gonna check that link.

As you mentioned, the minimum that splunk uses to check if a file has been indexed or not is 256 bytes. The problem is that my app writes a close date on the first line of the log, at the time it actually closes an app session.

Open file first line: 2021 2016-04-24 18:28:54 0000-00-00 00:00:00 +0100 00000cf0 001 003f 0001 09

Closed file first line: 2021 2016-04-24 18:28:54 2016-04-24 18:29:31 +0100 00000cf0 001 003f 0001

Re: Splunk Reindexes File that gets a new first line when closed

Richfez — Sun, 24 Apr 2016 18:29:06 GMT

Could you use a crc length of, like, 23? I was eyeballing it, you'll have to count the right number properly! That'll let it use the FIRST date in it as the check for when it's a new file, ignoring the newly written closing log file time, when that happens.

Re: Splunk Reindexes File that gets a new first line when closed

mattymo — Sun, 24 Apr 2016 18:30:42 GMT

ah yeah, sucks that it's so tight to the beginning of the file...

Pretty much guarantees you can't use splunk's ability to monitor the file in "realtime"..

Also be weary that if you are monitoring a directory on the HF that has thousands of files ( like var/log/*) it will have to keep track of all the files and can impact performance.

I speak from experience and is why i told you the tale of moving data around, cause once splunk indexes you will want to get rid of those files if you can...

Re: Splunk Reindexes File that gets a new first line when closed

Richfez — Sun, 24 Apr 2016 18:32:27 GMT

I second this. And third it. All in favor?

Really, though, IMO this is inordinately strange behavior from a logger. There's obviously some business reason or some reason involving an inferior product being worked around that caused the logging to be created this way, and it could be at least worth an ask to see if that behavior can be fixed now. There are quite a few compelling reasons.

Re: Splunk Reindexes File that gets a new first line when closed

mattymo — Sun, 24 Apr 2016 19:03:36 GMT

initCRCLength has to be a minimum of 256

😞

Re: Splunk Reindexes File that gets a new first line when closed

Richfez — Sun, 24 Apr 2016 19:36:59 GMT

Does anyone know why there's an arbitrary lower limit?

It makes me suspicious when the smallest possible value for a variable is also the default, and that default/smallest value isn't zero. Suspicious of what? I'm not sure, but suspicious of something. 🙂

Why not let the user decide what they need? There are cases where having it bigger is useful, and we've just stumbled across a situation where being able to make it smaller would be vastly simpler, easier and better than any other solution (excepting "fix the logger").

Re: Splunk Reindexes File that gets a new first line when closed

mattymo — Sun, 24 Apr 2016 19:46:13 GMT

my thoughts exactly...will see what I can find out!