Field Extraction of CSV on forwarder

[lohit]

Hi,

I have setup an universal forwarder to monitor a csv file and send the output to indexer(single instance acting as splunk web also). below are the config files on Universal forwarder.

input.conf [monitor://C:\logs_sql\logs.csv]

props.conf [source::C:\logssql\logs.csv] SHOULDLINEMERGE= false REPORT-logcsv=logcsv

tranforms.conf

[log_csv] DELIMS="," FIELDS="visitorid","uniqueid","country","Browser","Referrer","entrydate","IP"

Sample log file

visitorid,uniqueid,country,Browser,Referrer,entrydate,IP 10,26837:63373,United Kingdom,Mozilla/4.0,2009-10-02 14:59:37.437000000,16410458

After performing these changes nothing chnaged in output in splunk web.All the changes have been done on Universal forwarder only. Please let me know where i am going wrong?

[Ayn]

The problem lies in what you've said yourself - all the changes have been made on the forwarder, not your indexer. Almost all field extraction takes place at search-time, not index-time, so fields are extracted when you run your query on the indexer. As such, all search-time settings should go on the indexer, not the forwarder.

[lohit]

Thanks a ton Ayn,Kristian for your help. am able to run searches over it. I would like to know one thing i have uninstall Splunk UV and trying to delete source,sourcetype,host from Splunk web using |delete(have assigned can_delete role) .Initially it was working fine. But i follow uninstall/install UV process 5-6 times. and after that i am not able to delete it. If i am trying to view the events from source/sourcetype it is showing No events but on splunkweb it is still listed among source/sourcetype. What should be the reason around this.

[kristian_kolb]

Also, DELIMS and FIELDS should be on separate lines in transforms.conf.

[log_csv]
DELIMS =","
FIELDS = blah, bleh, bluh, bloh ...

/K

[Ayn]

Yes, that is correct.

[lohit]

if i got u correctly u saying the indexed logs will still be shown in csv format. and i have to remember the fields i specified in tranform.conf and use that field in query itself.

[Ayn]

Like I said, THAT will NOT change. Raw output will still be the same regardless of how many fields you extract from it.

[lohit]

Correct but it is not happening. file is still showing in csv default format

[Ayn]

Yes, if everything went OK you should be able to do something like sourcetype="dblogs1" | table country and get values for the country field.

[lohit]

i have specified the sourcetype a custom one.Also in splunk web shouldnot it be like country=US etc.

[Ayn]

The raw output itself will not change. What will happen when this works properly is that the fields you've specified will be available.

Your settings look OK, provided that they correspond to the CSV file's format, that sourcetype is correct etc.

[lohit]

@Kristian.

these are file on indexer
props.conf
[dblogs1]
SHOULD_LINEMERGE= false
REPORT-logcsv=log_csv

tranforms.conf
[log_csv]
DELIMS="," FIELDS="visitorid","uniqueid","country","Browser","Referrer","entrydate","IP"

I guess mapping is right but not getting desired output

[kristian_kolb]

Re-indexing not needed here. This is all search-time stuff, but with a REPORT directive in props.conf without a matching stanza in transforms.conf, the DELIMS/FIELDS will not be applied at all.

Thus, ensure that they match, like so (only relevant parts shown);

props.conf
[dblogs1]
REPORT-blah = log_csv

transforms.conf
[log_csv]

/K

[lohit]

@kristian

yeah its a typo. but taken care in conf files.

@linu

I have deleted the data from splunk web and reintsall splunk forwarder.What steps should i follow to get it done?

[linu1988]

Were the data re-indexed?

[kristian_kolb]

SHOULDLINEMERGE != SHOULD_LINEMERGE

logcsv != log_csv

typos?

[lohit]

Hi Ayn..i have done the follwing changes

On forwarder

inputs.conf
[monitor://C:logs_sql/logs.csv]
sourcetype=dblogs1

ON indexer

props.conf
[dblogs1]
SHOULDLINEMERGE= false
REPORT-logcsv=logcsv

tranforms.conf
[log_csv]
DELIMS="," FIELDS="visitorid","uniqueid","country","Browser","Referrer","entrydate","IP"

Still files in splunk web is appearing in default way.