topic Re: Ingesting file data in Getting Data In

Ingesting file data

bcross64 — Thu, 03 Oct 2013 18:44:48 GMT

I have a Powershell script that is writing data about sessions for an application to a file every 5 minutes so the file timestamp is changing, but Splunk only pulls in changed or new sessions. I was hoping Splunk would pull in all the data every 5 minutes so that I could get a good point-in-time count of the number of sessions and what they looked like at that time. Does anyone know if I can force Splunk to ingest the entire file when it changes? Here is the code I use to write a powershell object to a Splunk friendly file in case anyone is interested. "Id" is where the line break should be.

($obj | fl | Out-String).Trim() -replace "nrn","" -replace "rn","," -replace " :","=" -replace " +"," " -replace "(?m)^Id","nId" | Set-Content $file

Re: Ingesting file data

Ayn — Thu, 03 Oct 2013 19:03:48 GMT

Change the CHECK_METHOD for the file in props.conf. Any other than the default method will cause Splunk to index the entire file if it's changed. From the docs:

CHECK_METHOD = [endpoint_md5|entire_md5|modtime]
* Set CHECK_METHOD endpoint_md5 to have Splunk checksum of the first and last 256 bytes of a
  file. When it finds matches, Splunk lists the file as already indexed and indexes only new
  data, or ignores it if there is no new data.
* Set CHECK_METHOD = entire_md5 to use the checksum of the entire file.
* Set CHECK_METHOD = modtime to check only the modification time of the file.
* Settings other than endpoint_md5 cause Splunk to index the entire file for each detected
  change.
* Defaults to endpoint_md5.

So, you should do something like:

[source::<yoursource>]
CHECK_METHOD = modtime