topic Unable to see data in fields on Splunk 6 in Getting Data In

Unable to see data in fields on Splunk 6

afhussain — Mon, 28 Sep 2020 14:53:22 GMT

I am fairly new to splunk so please pardon any beginner's mistakes:

I am trying to setup Splunk to receive csv files via Universal Forwarder on windows 2k3 server and use header of the file to attach fields. I have done following:

On the universal forwarder:
[monitor://c:\some directory]
disabled = false
sourcetype = SBRAccounting
index=sbr
crcSalt =

On the Splunk server where indexer and search head are same, i created a new custom app and the /opt/splunk/etc/apps/SBRAccounting/local, I created props.conf file with following content:
[SBRAccounting]
HEADER_FIELD_LINE_NUMBER = 1
FIELD_DELIMITER = ,
FIELD_QUOTE = "

I see the data coming into Splunk with sourcetype=SBRAccounting and index=sbr but fields are not there.

I created a dummy file on the Splunk server and created inputs.conf file with following to prove props.conf file is configured correctly, and it seems to show field along with the data.
[monitor:///var/tmp/dummy.act]
sourcetype=SBRAccounting

Sample of the data is following:

"field1", "field2", "field3", ... 'field100"
"data1", "data2", "data3",,,,,,,,"data(n)"
"data1", "data2", "data3",,,,,,,,"data(n)"

I feel that data coming in via forwarders is not being passed by my custom apps's props.conf file. I even created an inputs.conf file with following and that didn't work either.
[splunktcp://:7001]
sourcetype=SBRAccounting

I need your help please.

Thanks

Re: Unable to see data in fields on Splunk 6

a212830 — Sun, 13 Oct 2013 01:43:42 GMT

Having same issue - no response yet...

Re: Unable to see data in fields on Splunk 6

rsennett_splunk — Sun, 13 Oct 2013 18:12:49 GMT

Reading carefully through the doc it looks like the props.conf on both the forwarder and the indexer need to contain the directives that explain what to do with the header.

http://docs.splunk.com/Documentation/Splunk/6.0/Data/Extractfieldsfromfileheadersatindextime

So while your experiment has you half way there, and you might have accidentally hit pay dirt, it isn't quite aligned yet.

The doc repeatedly instructs a "restart" after config changes but you can also do a

[[yourserver]][[yourport]]/debug/refresh
and get the same result without restarting... ususally

Forward data extracted from header files You can also forward fields extracted from a file with headers to another Splunk instance.

To forward fields extracted from structured data files, follow this procedure:

On the Splunk instance that monitors the files, edit props.conf and inputs.conf as described in "Edit configuration files to create and reference sourcetypes" earlier in this topic.

Next, configure the system to forward data to another Splunk instance.
Note: Read "Set up forwarding and receiving" in the Distributed Deployment Manual for instructions on how to configure Splunk instances to forward and receive data.

On the Splunk instance that is to receive the data, configure Splunk to be a receiver.

On the receiving Splunk instance, create props.conf in $SPLUNK_HOME/etc/system/local, if it does not already exist.

Copy the appropriate stanzas from props.conf on the monitoring Splunk instance to the props.conf you just created on the receiving Splunk instance.

Restart Splunk on the receiving instance.

Restart Splunk on the monitoring instance.

On the receiving instance, use the Search app to confirm that Splunk has extracted the fields from the structured data files and indexed them properly.

Another thing to note however, is that there is a Caveat noted in the next section of the doc. You have empty fields in your data. If you use the automatic extraction facility, Splunk will not extract fields that basically do not exist. In other words... if the field is represented only by , , it will not extract the field. If that field is represented by ," ", it will. See the doc for more details.