topic Re: Fields not extracted automatically, in Splunk Search

Fields not extracted automatically,

shangshin — Mon, 28 Sep 2020 14:23:49 GMT

Hi, I am using splunk 5.0.3 but found fields can't be extracted automatically on the splunk UI. To test, I loaded the sample csv file and use the customized sourcetype test_csv_log defined in props.conf. However, the fields like c1, c2, etc defined in transforms.conf are not auto-discovered by splunk. I am wondering if I miss anything? P.S. I did select verbose mode when doing the search......

Thanks!

sample.csv

07/19/2013 08:18:16:369 EDT, john,car, note,king,queen
07/19/2013 12:53:16:369 EDT, ws,ed,rf,tg,yh,uj

in props.conf

[test_csv_log]
TZ = 'America/New_York'
NO_BINARY_CHECK = 1
pulldown_type = 1
REPORT-r15 = test_csv_fields

in transforms.conf

[test_csv_fields]
DELIMS = ","
FIELDS = c1,c2,c3,c4,c5,c6,c7,c8,c8,c9,c10

Re: Fields not extracted automatically,

bmacias84 — Fri, 19 Jul 2013 21:44:36 GMT

you sample csv has variable colum length?

Re: Fields not extracted automatically,

bcavagnolo — Mon, 28 Sep 2020 14:28:28 GMT

I am having this same issue. In transforms.conf I have:
[myfield-mv]
REGEX = (?Pblahblahregex)
MV_ADD = true
SOURCE_KEY = myinputfield
...and in props.conf I have:
REPORT-myfield = myfield-mv
...but myfield does not appear among the "interesting fields" in searches from the we interface. However, if I search like this:
* | rex field=myinputfield "(?Pblahblahregex)"
...i do see myfield in the "interesting fields". Help!

Re: Fields not extracted automatically,

Gilberto_Castil — Wed, 31 Jul 2013 17:20:33 GMT

It is interesting that this works well by following the standard procedure (as you have done). For reference and comparison, here is the configuration for this test.

#inputs.conf
[monitor:///answers/7-31-2013/1/data]
disabled = false
index = test
sourcetype = answers-1375288490

#props.conf
[answers-1375288490]
REPORT-r15 = csv_fields_1375288490

#transforms.conf
[csv_fields_1375288490]
DELIMS = ","
FIELDS = c1,c2,c3,c4,c5,c6,c7,c8,c8,c9,c10

Here is what we see in SplunkWeb.

At this point, I will venture say there is something not clicking right in your test setup. Can you also post your inputs.conf?

Assuming that you are _not able to see the data displayed, the same can be accomplished in the UI with the following:

sourcetype="answers-1375288490" | rex "EDT,\s+(?<message>.+)" | rex field=message max_match=0 "(?<c>\w+)(?:,|$)"

And, these are the results. Note the field "C" is available.

Or, you may also try this:

sourcetype="answers-1375288490" | rex "EDT,\s+(?<c>.+)" | makemv delim="," c

Surely you will agree that your objective is possible in a number of ways. Let's get back to your test and compare.

--gc

Re: Fields not extracted automatically,

shangshin — Wed, 31 Jul 2013 17:42:24 GMT

This is very useful. Thank you very much!

Re: Fields not extracted automatically,

bcavagnolo — Thu, 01 Aug 2013 18:57:55 GMT

Hey Gilberto. This problem still persists for me (see my comment under the question with props.conf and transforms.conf snippets). I am able to see the field when I query for it explicitly in splunk web with rex, but not otherwise. Note that the log data was all imported with command-line oneshot calls like this:

splunk add oneshot logfile -index main -sourcetype mysrctype -host myhost

...so there is not inputs.conf segment. Can you spot a problem with my configuration that might explain this?