Getting Data In

How do I index files which do not grow, just change?

chengka
Explorer

I currently write status files and collect them into Splunk using monitor statements and Universal Forwarders. Pretty vanilla stuff. Well, we are looking to rewrite our status scripts and I've been asked if we can just rewrite the status information into the same file every 3 minutes and save it. The file size would not just(much), just the contents would change. My first reaction was a faint yet. I thought forwarder would notice a CRC change and reindex from the beginning. This doesn't seem to be happening. If I restart the forwarder is throws and error and reindexes, and it seems once in a while it will catch a change, but not dependably. I assumed placing a timestamp at the head of the status information would assure a CRC change, but sadly we are not getting the desired results.

Ignoring the advisability of this method, should it work? Is there a timeout or polling interval which applies here, at which time the forwarder will recheck CRC?

Log from Forwarder on restart

2/19/15 
8:29:37.599 AM  
02-19-2015 09:29:37.599 -0600 INFO  WatchedFile - Checksum for seekptr didn't match, will re-read entire file='D:\SplunkData\Test.txt

Here is the contents of my test file. The only required change every 3 minutes is the time stamp, the rest of the text could be identical.

[02/19/15 10:25:21 -0600]  Node(xxxxxxxx) DataCenter(YYYY) Volume(/aaaa/bbbb) Size(32768000) Time(0.13843) Throughput(236.712) NewField(two)

bshuler_splunk
Splunk Employee
Splunk Employee

Option 1:
Put a timestamp in the filename:

filename-01-19-2015-13-03-00.txt

Be sure to script a clean up of the directory every week or so.

Option 2:
Use a scripted input. Have the script read out the file into splunk, then delete the file from the file system.

Scripted inputs are covered here:

http://docs.splunk.com/Documentation/Splunk/6.2.1/AdvancedDev/ScriptedInputsIntro

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...