topic Re: How to use sourcetype with inputcsv? in Getting Data In

How to use sourcetype with inputcsv?

ketaka — Tue, 25 Jun 2019 02:11:10 GMT

I want to apply "sourcetype" when reading csv file by "inputcsv" command.

Is this possible by setting "props.conf"?

I want to set the header arbitrarily as below and before.

Before

AAA,AAA,AAA
System1,User1,NW1
System2,User2,NW2
System3,User3,NW3

After

Sys,Usr,NW
System1,User1,NW1
System2,User2,NW2
System3,User3,NW3

This is because duplicate field names are fixed in a system that outputs CSV.

Re: How to use sourcetype with inputcsv?

denzelchung — Tue, 25 Jun 2019 03:01:52 GMT

You might want to try the accepted answer in this post: https://answers.splunk.com/answers/432261/is-there-a-way-to-rename-csv-column-headers-prior.html

Re: How to use sourcetype with inputcsv?

ketaka — Tue, 25 Jun 2019 05:07:44 GMT

Thank you for your quick reply

I saw the posted post.
However, it does not work as expected.

It seems that my study is not enough to understand.
The relationship between the inputcsv command and props.conf and transforms.conf can not be understood.
I will continue to investigate.

The set contents and commands are described below.

Thank you

input csv file path
/Splunk/var/run/splunk/csv/inputtest.csv
use spl
|inputcsv inputtest.csv

conf file settings

transforms.conf /Splunk/etc/system/local/transforms.conf

~~~
[inputtest]
DELIMS = ","
FIELDS = "Sys","Usr","NW"
~~~

props.conf /Splunk/etc/system/local/props.conf

~~~
[inputtest]
KV_MODE = none
filename = inputtest.csv
REPORT-testHeader = inputtest
~~~

Re: How to use sourcetype with inputcsv?

DavidHourani — Tue, 25 Jun 2019 09:05:27 GMT

Hi @ketaka,

If you wish to apply a sourcetype to a CSV file you need to index it. Using inputcsvonly reads the csv file just as an inputlookup would. If you don't want to index your data then you can rename the fields using the rename command.

If your only solution is to apply a sourcetype you will have to index your csv file. Below some docs with options and params for indexing csv and other structured files.

This is a great read for indexing csv files of all kinds :
https://docs.splunk.com/Documentation/Splunk/7.3.0/Data/Extractfieldsfromfileswithstructureddata
All available configurations for structured data in props can be found here :
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf#Structured_Data_Header_Extraction_and_configuration

Let me know if that helps.

Cheers,
David

Re: How to use sourcetype with inputcsv?

ketaka — Wed, 26 Jun 2019 06:01:54 GMT

Hi David,

Thank you for your comment.

Unfortunately I don't want to make an index.
I want to load a CSV file each time and perform new tabulation.
and I want to distinguish duplicate header names as aliases.

I think this is a departure from the benefits of Splunk.
However, there is an instruction to move from Excel VBA to Splunkn, and I am looking for a way.

I thought that I could cope by loading csv file with SPL's inputcsv command each time.

I'm continuously examining the inputcsv command, inputs.conf, and props.conf.

Please be aware that terms and recognition may not be correct.

Thanks

ketaka

Re: How to use sourcetype with inputcsv?

DavidHourani — Wed, 26 Jun 2019 09:36:59 GMT

So you want to fixup the data using VBA and then read it with Splunk using inputcsv ?

The problem I see with inputcsv is that you won't be able to apply logic using props.conf as this only applies to sourcetypes for data that has been indexed. So if you want to apply something to your data before searching that data should be indexed.

Re: How to use sourcetype with inputcsv?

ketaka — Wed, 26 Jun 2019 14:55:03 GMT

CSV files are output in a fixed form from other departments' systems.
I'm trying to replace the file processing Excel with Splunk.

I see, props.conf works only for indexed data..

It turns out that we need to think about different means.

Thank you for your reply many times.

(The following may be unnecessary content, sorry.

The summary of what you want to do is to load three types of standard CSV files, and output one CSV file after processing.
At that time, it is necessary to delete previous data and perform new processing.
I think this process is not suitable for Splunk, which indexes and accumulates data.
If you look only at my work on the project, it may not be worthwhile to use Splunk.