Getting Data In

How do I index files which do not grow, just change?

chengka
Explorer

I currently write status files and collect them into Splunk using monitor statements and Universal Forwarders. Pretty vanilla stuff. Well, we are looking to rewrite our status scripts and I've been asked if we can just rewrite the status information into the same file every 3 minutes and save it. The file size would not just(much), just the contents would change. My first reaction was a faint yet. I thought forwarder would notice a CRC change and reindex from the beginning. This doesn't seem to be happening. If I restart the forwarder is throws and error and reindexes, and it seems once in a while it will catch a change, but not dependably. I assumed placing a timestamp at the head of the status information would assure a CRC change, but sadly we are not getting the desired results.

Ignoring the advisability of this method, should it work? Is there a timeout or polling interval which applies here, at which time the forwarder will recheck CRC?

Log from Forwarder on restart

2/19/15 
8:29:37.599 AM  
02-19-2015 09:29:37.599 -0600 INFO  WatchedFile - Checksum for seekptr didn't match, will re-read entire file='D:\SplunkData\Test.txt

Here is the contents of my test file. The only required change every 3 minutes is the time stamp, the rest of the text could be identical.

[02/19/15 10:25:21 -0600]  Node(xxxxxxxx) DataCenter(YYYY) Volume(/aaaa/bbbb) Size(32768000) Time(0.13843) Throughput(236.712) NewField(two)

bshuler_splunk
Splunk Employee
Splunk Employee

Option 1:
Put a timestamp in the filename:

filename-01-19-2015-13-03-00.txt

Be sure to script a clean up of the directory every week or so.

Option 2:
Use a scripted input. Have the script read out the file into splunk, then delete the file from the file system.

Scripted inputs are covered here:

http://docs.splunk.com/Documentation/Splunk/6.2.1/AdvancedDev/ScriptedInputsIntro

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...