topic Ingesting files with updated first line in Getting Data In

Ingesting files with updated first line

erikgrasman — Fri, 15 Jun 2018 09:43:11 GMT

I got a file which get new log entries during the day, when a user logs out, the first line of the log is updated with the timestamp of the last event. New crc check will say this is a new file and will reingest, I'm aware that i can turn that off, so the file won't be reindexed, but the updated line (the first line of the logfile) should be ingested again as a new event.

For example at the beginning the file looks like this

2018-06-15 11:32 728638327652 INFO Log file opened Log file finished XXXX-XX-XX XX:XX some additional info
2018-06-15 11:32 728638327652 INFO Log line 2 some additional info
2018-06-15 11:32 728638327652 INFO Log line 3 some additional info
2018-06-15 11:32 728638327652 INFO Log line 4 some additional info
....

Then at the end of a session the following happens

2018-06-15 11:32 728638327652 INFO Log file opened Log file finished 2018-06-15 11:36 some additional info
2018-06-15 11:32 728638327652 INFO Log line 2 some additional info
2018-06-15 11:32 728638327652 INFO Log line 3 some additional info
2018-06-15 11:32 728638327652 INFO Log line 4 some additional info
...
2018-06-15 11:36 728638327652 INFO Log line 86 some additional info
2018-06-15 11:36 728638327652 INFO Log line 87 some additional info
2018-06-15 11:36 728638327652 INFO Log line 88 last line of the log file

I am aware that I can solve this in SPL

In case of double ingestion by dedupping
In case of ingesting the file without an updated first event by | eventstats latest(_time) by sessionid

But we're going to query this very often and the preferred solution is index all lines when written to file and after the first line is updated, index the first line again.

Re: Ingesting files with updated first line

richgalloway — Fri, 15 Jun 2018 13:22:38 GMT

Splunk either indexes the entire file each time it changes or it indexes new lines. There is no option to re-index parts of a file.

Re: Ingesting files with updated first line

somesoni2 — Fri, 15 Jun 2018 15:13:32 GMT

It's an odd type of logging that you do. Do you have control over how logging is done? Can you change it to log the timestamp change (for "opened Log file..") to the end of the time along with new lines to be added?

Re: Ingesting files with updated first line

erikgrasman — Fri, 15 Jun 2018 15:37:39 GMT

odd... at least indeed

I don't think we got very much control over the guys who are creating the logging, it's a company called oracle 🙂

https://docs.oracle.com/cd/E14004_01/books/OIRef/Using_Siebel_VB_and_Siebel_eScript10.html

Re: Ingesting files with updated first line

erikgrasman — Thu, 28 Jun 2018 06:27:59 GMT

Discussed this with Splunk Support / Development. Currently there is no out of the box solution (like a crc offset which starts after x amount of characters).
1
Thanks