Getting Data In

Unable to see data in fields on Splunk 6

afhussain
Explorer

I am fairly new to splunk so please pardon any beginner's mistakes:

I am trying to setup Splunk to receive csv files via Universal Forwarder on windows 2k3 server and use header of the file to attach fields. I have done following:

On the universal forwarder:
[monitor://c:\some directory]
disabled = false
sourcetype = SBRAccounting
index=sbr
crcSalt =

On the Splunk server where indexer and search head are same, i created a new custom app and the /opt/splunk/etc/apps/SBRAccounting/local, I created props.conf file with following content:
[SBRAccounting]
HEADER_FIELD_LINE_NUMBER = 1
FIELD_DELIMITER = ,
FIELD_QUOTE = "

I see the data coming into Splunk with sourcetype=SBRAccounting and index=sbr but fields are not there.

I created a dummy file on the Splunk server and created inputs.conf file with following to prove props.conf file is configured correctly, and it seems to show field along with the data.
[monitor:///var/tmp/dummy.act]
sourcetype=SBRAccounting

Sample of the data is following:

"field1", "field2", "field3", ... 'field100"
"data1", "data2", "data3",,,,,,,,"data(n)"
"data1", "data2", "data3",,,,,,,,"data(n)"

I feel that data coming in via forwarders is not being passed by my custom apps's props.conf file. I even created an inputs.conf file with following and that didn't work either.
[splunktcp://:7001]
sourcetype=SBRAccounting

I need your help please.

Thanks

Tags (3)

rsennett_splunk
Splunk Employee
Splunk Employee

Reading carefully through the doc it looks like the props.conf on both the forwarder and the indexer need to contain the directives that explain what to do with the header.

http://docs.splunk.com/Documentation/Splunk/6.0/Data/Extractfieldsfromfileheadersatindextime

So while your experiment has you half way there, and you might have accidentally hit pay dirt, it isn't quite aligned yet.

The doc repeatedly instructs a "restart" after config changes but you can also do a

[[yourserver]][[yourport]]/debug/refresh
and get the same result without restarting... ususally

Forward data extracted from header files You can also forward fields extracted from a file with headers to another Splunk instance.

To forward fields extracted from structured data files, follow this procedure:

  1. On the Splunk instance that monitors the files, edit props.conf and inputs.conf as described in "Edit configuration files to create and reference sourcetypes" earlier in this topic.

  2. Next, configure the system to forward data to another Splunk instance.
Note: Read "Set up forwarding and receiving" in the Distributed Deployment Manual for instructions on how to configure Splunk instances to forward and receive data.


  1. On the Splunk instance that is to receive the data, configure Splunk to be a receiver.

  2. On the receiving Splunk instance, create props.conf in $SPLUNK_HOME/etc/system/local, if it does not already exist.

  3. Copy the appropriate stanzas from props.conf on the monitoring Splunk instance to the props.conf you just created on the receiving Splunk instance.

  4. Restart Splunk on the receiving instance.

  5. Restart Splunk on the monitoring instance.

  6. On the receiving instance, use the Search app to confirm that Splunk has extracted the fields from the structured data files and indexed them properly.

Another thing to note however, is that there is a Caveat noted in the next section of the doc. You have empty fields in your data. If you use the automatic extraction facility, Splunk will not extract fields that basically do not exist. In other words... if the field is represented only by , , it will not extract the field. If that field is represented by ," ", it will. See the doc for more details.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

a212830
Champion

Having same issue - no response yet...

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...