Splunk Search

Can't extract fields names from tab delimited source

nosignal
Explorer

Hi. I'm new to Splunk. I've got basic import and searching working on the windows install, but I want to get the field names too, and 9 attempts later, I still can't get it to work.

I'm starting with a tab-delimited text file as source. E.g.:

19/09/2011 20:39:30 Red         Square      
19/09/2011 22:04:02 Green   Bob Square      
19/09/2011 22:05:26 Blue            Triangle        

Note: some field values are empty for some events.
I did not include the field names in the first line - I read about CHECK_FOR_HEADER causing more problems than it is worth.

I add the data via the Splunk web interface: "Add data" -> "A file or directory of files" -> "Consume any file on this Splunk server" -> "Preview data before indexing"...

Trying "Start a new source type" -> "adjust timestamp and event break settings" -> "Every line is one event ex: access logs"
Did not import the field names, which is understandable.

I've never seen my field names come in properly, but I am looking for them in the search results page below "Field discovery is: On" - when I "View all Fields", none of them have field names I care about. I also try changing to "Table View" and all of the data is still under one column called "_raw".

So I followed the instructions here and edited the files at $SPLUNK_HOME/etc/system/local to include this in props.conf:

[My Source Type 1]
MAX_TIMESTAMP_LOOKAHEAD = 20
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
pulldown_type = 1
REPORT-myname = mydelim

And include this in transforms.conf:

[mydelim]
DELIMS = "\t"
FIELDS = "TimeStamp","Colour","First Name","Shape"

I also tried

[mydelim]
DELIMS = "\t"
FIELDS = f1,f2,f3,f4

I had to restart the splunk server to get the changes to be recognised.
Then I followed the same procedure above for adding data, but this time chose "Apply an existing source type" and chose "My Source Type 1".

Unfortunately, neither attempt worked, nor did a bunch of subtle variations. I am deleting all data between tests.

What am I doing wrong?
Can someone please confirm I am looking for field names in the correct way?

Many thanks.

0 Karma
1 Solution

nosignal
Explorer

I finally figured this out - I had edited the original post to use the REPORT thing. This actually worked, I just did not realise it - my two tests for looking for field names were bad tests - they did not show any field names even though there were now extracted field names!

This is pretty bizzare, and in my opinion, there should be something in the search GUI that can show all field names - maybe there is a way and I don't see it?
But oh well, but maybe I'll get used to it.

Part of what threw me is that I thought "View All 7 Fields" referred to all fields in existence, but no - it only refers to fields THAT ARE VISIBLE in the search results and you have to MAKE FIELDS VISIBLE MANUALLY. So you get into a chicken-and-egg situation where the fields you are interested in are not listed because you have not made them visible and you can't make them visible because they are not listed! I'd love to know of a good solution to this - I've found a crude hack below.

The best tests I can find now are:

  1. It used to show "Field discovery is: On" but now that the field name extraction is working, it shows "Field discovery is: Off"
  2. If I search for a field name that I know I specified in the transform.conf, it lists results, and then I can add that to my selected/visible fields. But for numerous fields, this is a hassle. E.g. Colour=Blue
  3. From here I discovered you can do a search that lists all field names, and then you can make all of them selected visible more easily. E.g. sourcetype="My Source Type 1" | stats dc(*) as *

I hope that helps someone.

View solution in original post

nosignal
Explorer

I finally figured this out - I had edited the original post to use the REPORT thing. This actually worked, I just did not realise it - my two tests for looking for field names were bad tests - they did not show any field names even though there were now extracted field names!

This is pretty bizzare, and in my opinion, there should be something in the search GUI that can show all field names - maybe there is a way and I don't see it?
But oh well, but maybe I'll get used to it.

Part of what threw me is that I thought "View All 7 Fields" referred to all fields in existence, but no - it only refers to fields THAT ARE VISIBLE in the search results and you have to MAKE FIELDS VISIBLE MANUALLY. So you get into a chicken-and-egg situation where the fields you are interested in are not listed because you have not made them visible and you can't make them visible because they are not listed! I'd love to know of a good solution to this - I've found a crude hack below.

The best tests I can find now are:

  1. It used to show "Field discovery is: On" but now that the field name extraction is working, it shows "Field discovery is: Off"
  2. If I search for a field name that I know I specified in the transform.conf, it lists results, and then I can add that to my selected/visible fields. But for numerous fields, this is a hassle. E.g. Colour=Blue
  3. From here I discovered you can do a search that lists all field names, and then you can make all of them selected visible more easily. E.g. sourcetype="My Source Type 1" | stats dc(*) as *

I hope that helps someone.

View solution in original post

.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!