Getting Data In

Ingesting files with updated first line

erikgrasman
Engager

I got a file which get new log entries during the day, when a user logs out, the first line of the log is updated with the timestamp of the last event. New crc check will say this is a new file and will reingest, I'm aware that i can turn that off, so the file won't be reindexed, but the updated line (the first line of the logfile) should be ingested again as a new event.

For example at the beginning the file looks like this

2018-06-15 11:32 728638327652 INFO Log file opened Log file finished XXXX-XX-XX XX:XX some additional info
2018-06-15 11:32 728638327652 INFO Log line 2 some additional info
2018-06-15 11:32 728638327652 INFO Log line 3 some additional info
2018-06-15 11:32 728638327652 INFO Log line 4 some additional info
....

Then at the end of a session the following happens

2018-06-15 11:32 728638327652 INFO Log file opened Log file finished 2018-06-15 11:36 some additional info
2018-06-15 11:32 728638327652 INFO Log line 2 some additional info
2018-06-15 11:32 728638327652 INFO Log line 3 some additional info
2018-06-15 11:32 728638327652 INFO Log line 4 some additional info
...
2018-06-15 11:36 728638327652 INFO Log line 86 some additional info
2018-06-15 11:36 728638327652 INFO Log line 87 some additional info
2018-06-15 11:36 728638327652 INFO Log line 88 last line of the log file

I am aware that I can solve this in SPL

  • In case of double ingestion by dedupping
  • In case of ingesting the file without an updated first event by | eventstats latest(_time) by sessionid

But we're going to query this very often and the preferred solution is index all lines when written to file and after the first line is updated, index the first line again.

0 Karma

erikgrasman
Engager

Discussed this with Splunk Support / Development. Currently there is no out of the box solution (like a crc offset which starts after x amount of characters).
1
Thanks

0 Karma

somesoni2
SplunkTrust
SplunkTrust

It's an odd type of logging that you do. Do you have control over how logging is done? Can you change it to log the timestamp change (for "opened Log file..") to the end of the time along with new lines to be added?

0 Karma

erikgrasman
Engager

odd... at least indeed

I don't think we got very much control over the guys who are creating the logging, it's a company called oracle 🙂

https://docs.oracle.com/cd/E14004_01/books/OIRef/Using_Siebel_VB_and_Siebel_eScript10.html

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Splunk either indexes the entire file each time it changes or it indexes new lines. There is no option to re-index parts of a file.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Using the Splunk Threat Research Team’s Latest Security Content

REGISTER HERE Tech Talk | Security Edition Did you know the Splunk Threat Research Team regularly releases ...