Ingesting files with updated first line

[erikgrasman]

I got a file which get new log entries during the day, when a user logs out, the first line of the log is updated with the timestamp of the last event. New crc check will say this is a new file and will reingest, I'm aware that i can turn that off, so the file won't be reindexed, but the updated line (the first line of the logfile) should be ingested again as a new event.

For example at the beginning the file looks like this

2018-06-15 11:32 728638327652 INFO Log file opened Log file finished XXXX-XX-XX XX:XX some additional info
2018-06-15 11:32 728638327652 INFO Log line 2 some additional info
2018-06-15 11:32 728638327652 INFO Log line 3 some additional info
2018-06-15 11:32 728638327652 INFO Log line 4 some additional info
....

Then at the end of a session the following happens

2018-06-15 11:32 728638327652 INFO Log file opened Log file finished 2018-06-15 11:36 some additional info
2018-06-15 11:32 728638327652 INFO Log line 2 some additional info
2018-06-15 11:32 728638327652 INFO Log line 3 some additional info
2018-06-15 11:32 728638327652 INFO Log line 4 some additional info
...
2018-06-15 11:36 728638327652 INFO Log line 86 some additional info
2018-06-15 11:36 728638327652 INFO Log line 87 some additional info
2018-06-15 11:36 728638327652 INFO Log line 88 last line of the log file

I am aware that I can solve this in SPL

• In case of double ingestion by dedupping
• In case of ingesting the file without an updated first event by | eventstats latest(_time) by sessionid

But we're going to query this very often and the preferred solution is index all lines when written to file and after the first line is updated, index the first line again.

[erikgrasman]

Discussed this with Splunk Support / Development. Currently there is no out of the box solution (like a crc offset which starts after x amount of characters).
1
Thanks

[somesoni2]

It's an odd type of logging that you do. Do you have control over how logging is done? Can you change it to log the timestamp change (for "opened Log file..") to the end of the time along with new lines to be added?

[erikgrasman]

odd... at least indeed

I don't think we got very much control over the guys who are creating the logging, it's a company called oracle 🙂

https://docs.oracle.com/cd/E14004_01/books/OIRef/Using_Siebel_VB_and_Siebel_eScript10.html

[richgalloway]

Splunk either indexes the entire file each time it changes or it indexes new lines. There is no option to re-index parts of a file.