topic Re: How do I set the inputs.conf file for continuous monitoring of an input file for appended data? in Getting Data In

How do I set the inputs.conf file for continuous monitoring of an input file for appended data?

mahbs — Wed, 18 Oct 2017 13:50:32 GMT

Hi,

I've uploaded a txt file containing CPU performance data to splunk, set the source type to csv and have created an index. I've been able to generate a report from this.

Data is coming from the txt file correctly under correct index and sourcetype, but the problem is that the file is constantly appended with new data and the appended data doesn't come up until I add the same file again from data inputs and do the same steps again.

Can someone please help on how to get the data indexed in Splunk as soon as the input file gets updated? From my research, I've found that I need to customize the inputs.conf file, but I have no idea where to start.

Your help would be very much appreciated

Thanks

Re: How do I set the inputs.conf file for continuous monitoring of an input file for appended data?

elliotproebstel — Wed, 18 Oct 2017 14:54:04 GMT

When you say you've uploaded the file to Splunk, do you mean that you copied it from elsewhere and used the "Upload" option under "Add Data"?

If you upload a file, it will not be monitored for new data; it will only be indexed once. If the file is on your indexer, you can use the "Monitor" option from "Add Data" to allow Splunk to monitor the file for new data. If the file is on another computer, you'll need some way for that data to reach the indexer - either by installing a universal forwarder on that computer and adding the file as a monitored data source or via another method (e.g. syslog forwarding).

Re: How do I set the inputs.conf file for continuous monitoring of an input file for appended data?

mahbs — Wed, 18 Oct 2017 15:18:15 GMT

Right, I already have a universal forwarder set up. I guess my question is, what is the correct syntax in the inputs.conf file to ensure that the desired file is monitored and then sent to splunk?

Re: How do I set the inputs.conf file for continuous monitoring of an input file for appended data?

gcusello — Wed, 18 Oct 2017 15:35:05 GMT

Hi mahbs,
see http://docs.splunk.com/Documentation/Splunk/latest/Data/Usingforwardingagents
on your forwarder create a stanza in inputs.conf that contains:
if Unix

[monitor:///your_path/your_filename.csv]
disabled=0
index=your_index
sourcetype=csv

if Windows

[monitor://c:\your_path\your_filename.csv]
disabled=0
index=your_index
sourcetype=csv

Restart forwarder and you'll start to receive logs (if Forwarder is correctly configured to send logs to your indexers).

For test you can create this inputs.conf in $SPLUNK_HOME/etc/systel/local, if it's a production system, it's better to create a Technical Addon (TA) containing this inputs.conf and deploy it using a Deployment Server.

Bye.
Giuseppe

Re: How do I set the inputs.conf file for continuous monitoring of an input file for appended data?

mahbs — Wed, 18 Oct 2017 15:40:31 GMT

Right I see. I noticed in the monitor section, the filename is of type csv, my one however is of type log, would that be a problem?

Re: How do I set the inputs.conf file for continuous monitoring of an input file for appended data?

gcusello — Wed, 18 Oct 2017 15:55:04 GMT

In monitor put path and name of your file.
Remember that props.conf os usually located on indexers.
Only ingesting csv files must be located also on forwarders.
Bye.
Giuseppe

Re: How do I set the inputs.conf file for continuous monitoring of an input file for appended data?

mahbs — Thu, 19 Oct 2017 09:03:49 GMT

Hi, it can't seem the find the file, my path is something like this: foo/foo1/x.log <- is that a legal path?

Re: How do I set the inputs.conf file for continuous monitoring of an input file for appended data?

gcusello — Thu, 19 Oct 2017 09:12:40 GMT

Hi mahbs,
to be sure of your path go in CLI and use the same path with ls -al command:

ls -al /foo/foo1/x.log

if you can find it use this path in monitor, otherwise search the correct one:

[monitor:///foo/foo1/x.log]

note that after monitor: you have always two slashes and a third one from the path, totally three slashes.
Only to understand: monitoring a windows path you have

[monitor://c:\temp\log.log]

Bye.
Giuseppe

Re: How do I set the inputs.conf file for continuous monitoring of an input file for appended data?

mahbs — Thu, 19 Oct 2017 09:39:27 GMT

Hi Giuseppe,

Apologies for the non -stop questions. I've found out the correct path, its: /foo/foo1/foo1.1/x.log

so essentially my path should be like this:

[monitor:///foo/foo1/foo1.1/x.log]
disabled=0
index=your_index
sourcetype=csv

For some reason, splunk is still not being updated. Am I missing anything?

Re: How do I set the inputs.conf file for continuous monitoring of an input file for appended data?

gcusello — Thu, 19 Oct 2017 11:21:01 GMT

Hi mahbs,
you said that new data are added to the same file, is it correct?
The easiest way is to have different filenames (e.g. x_20171019.txt).
Otherwise there could be a problem because Splunk check the first 256 chars to find differences and to add new events.

To change this parameter is a delicate operation (see https://docs.splunk.com/Documentation/Splunk/7.0.0/Admin/Inputsconf 😞

initCrcLength =
* This setting adjusts how much of a file the input reads before trying to identify whether it is a file that has already been seen. You might want to adjust this if you have many files with common headers (comment headers, long CSV headers, etc) and recurring filenames.
* CAUTION: Improper use of this setting will cause data to be re-indexed. You might want to consult with Splunk Support before adjusting this value - the default is fine for most installations.
* Defaults to 256 (bytes).
* Must be in the range 256-1048576.

Bye.
Giuseppe

Re: How do I set the inputs.conf file for continuous monitoring of an input file for appended data?

mahbs — Thu, 19 Oct 2017 11:47:35 GMT

Hey Giuseppe, I think I've made some progress. This is the message im getting in splunk::
 Received event for unconfigured/disabled/deleted index="xxx" What could this mean?

Re: How do I set the inputs.conf file for continuous monitoring of an input file for appended data?

gcusello — Thu, 19 Oct 2017 13:20:03 GMT

Hi mahbs,
this means that the index that you set in inputs.conf doesn't exist.
Try to use an existing index in inputs.conf: index=<your_index>

Bye.
Giuseppe