Getting Data In

How do I index files which do not grow, just change?

chengka
Explorer

I currently write status files and collect them into Splunk using monitor statements and Universal Forwarders. Pretty vanilla stuff. Well, we are looking to rewrite our status scripts and I've been asked if we can just rewrite the status information into the same file every 3 minutes and save it. The file size would not just(much), just the contents would change. My first reaction was a faint yet. I thought forwarder would notice a CRC change and reindex from the beginning. This doesn't seem to be happening. If I restart the forwarder is throws and error and reindexes, and it seems once in a while it will catch a change, but not dependably. I assumed placing a timestamp at the head of the status information would assure a CRC change, but sadly we are not getting the desired results.

Ignoring the advisability of this method, should it work? Is there a timeout or polling interval which applies here, at which time the forwarder will recheck CRC?

Log from Forwarder on restart

2/19/15 
8:29:37.599 AM  
02-19-2015 09:29:37.599 -0600 INFO  WatchedFile - Checksum for seekptr didn't match, will re-read entire file='D:\SplunkData\Test.txt

Here is the contents of my test file. The only required change every 3 minutes is the time stamp, the rest of the text could be identical.

[02/19/15 10:25:21 -0600]  Node(xxxxxxxx) DataCenter(YYYY) Volume(/aaaa/bbbb) Size(32768000) Time(0.13843) Throughput(236.712) NewField(two)

bshuler_splunk
Splunk Employee
Splunk Employee

Option 1:
Put a timestamp in the filename:

filename-01-19-2015-13-03-00.txt

Be sure to script a clean up of the directory every week or so.

Option 2:
Use a scripted input. Have the script read out the file into splunk, then delete the file from the file system.

Scripted inputs are covered here:

http://docs.splunk.com/Documentation/Splunk/6.2.1/AdvancedDev/ScriptedInputsIntro

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...