Splunk Search
Highlighted

Missing interesting fields in the Search & Reporting screen

Explorer

When I use the Splunk's Search & Reporting screen, it does not list any of the Interesting fields that are in the csv files it indexed.

I added a Hadoop Connect input and is configured as:
Resource name: 192.168.56.102:9000/WeatherStationInfo/
White list regex: *.txt
Set the source type: Manual
Source type: weatherInfo
Host field value: splunk
Index: weather

/opt/splunk/etc/system/local/props.conf contains:
[weatherInfo]
INDEXEDEXTRACTIONS = csv
KV
MODE = none
NOBINARYCHECK = 1
SHOULDLINEMERGE = false
TIME
FORMAT = %Y-%m-%d %H:%M:%S
pulldown_type = 1

One of the files Splunk indexed was:
Time,TemperatureF,DewpointF,PressureIn,WindDirection,WindDirectionDegrees,WindSpeedMPH,WindSpeedGustMPH,Humidity,HourlyPrecipIn,Conditions,Clouds,dailyrainin,SolarRadiationWatts/m^2,SoftwareType,DateUTC
2014-01-01 00:00:00,45.2,24.4,30.16,ENE,71,1.0,4.0,44,0.00,,,0.00,0.0,WUHU216DAVISVP2,2014-01-01 08:00:00,
2014-01-01 00:05:00,45.3,24.0,30.16,ENE,65,2.0,4.0,43,0.00,,,0.00,0.0,WUHU216DAVISVP2,2014-01-01 08:05:00,
2014-01-01 00:10:00,45.6,24.2,30.16,ENE,65,2.0,4.0,43,0.00,,,0.00,0.0,WUHU216DAVISVP2,2014-01-01 08:10:00,
...

The fields listed on the first line, such as TemperatureF, are not listed as one of the Interesting fields and I cannot use them to search. Where am I doing wrong?

Thanks, Bill.

0 Karma
Highlighted

Re: Missing interesting fields in the Search & Reporting screen

SplunkTrust
SplunkTrust

Try adding CHECK_FOR_HEADER = true to your props.conf stanza.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: Missing interesting fields in the Search & Reporting screen

Explorer

I did as you suggested but it is still failing to list any of the fields in the csv files. The /opt/splunk/etc/system/local/props.conf now contains:
[weatherInfo]
INDEXEDEXTRACTIONS = csv
KV
MODE = none
NOBINARYCHECK = 1
SHOULDLINEMERGE = false
TIME
FORMAT = %Y-%m-%d %H:%M:%S pulldowntype = 1
CHECK
FOR_HEADERS = true

To force Splunk to re-index the files, I cleaned all Splunk events by executing the "./splunk clean eventdata" command. I then renamed all the files in the 192.168.56.102:9000/WeatherStationInfo/ directory so that Splunk will reprocess the files.

0 Karma
Highlighted

Re: Missing interesting fields in the Search & Reporting screen

Explorer

I just made Splunk index only one of the files that I made local to Splunk using the same settings I used for the other source and Splunk successfully provided the list of fields in the csv file. Could this be a problem when using Hadoop?

My next test will be to index the files found in a local directory to Splunk.

0 Karma
Highlighted

Re: Missing interesting fields in the Search & Reporting screen

Path Finder

Tried adding header. It is still not working. Anybody have any answer for this?

0 Karma
Highlighted

Re: Missing interesting fields in the Search & Reporting screen

Splunk Employee
Splunk Employee

As you pointed out, the CSV header extractions is successful when indexing this file directly with weatherInfo sourcetype.
Did you check proper inputs.conf stanza for Hadoop Connect input?

0 Karma
Highlighted

Re: Missing interesting fields in the Search & Reporting screen

Explorer

I just checked the Hadoop inputs.conf file and its contents all looks accurate including the "sourcetype". The file only contains the following:
[hdfs://192.168.56.102:9000/WeatherStationInfo/]
index = weather
sourcetype = weatherInfo
whitelist = *.txt

0 Karma
Highlighted

Re: Missing interesting fields in the Search & Reporting screen

Explorer

I am still working on this problem. I find that Splunk does process the headers if Splunk imports the files locally. But if the file is from HDFS, the headers are not processed. Is this what I am to expect?

0 Karma