topic question on data inputs for multiple csv files in Getting Data In

question on data inputs for multiple csv files

jiaqya — Fri, 13 Apr 2018 15:33:28 GMT

i see that i can chose the single csv file type for a csv file and verify the columns are right and then insert into the index via data input option

but when i have multiple csv files in a folder, i try to enter the name with wildcard ie * , and this time it just does not consider it to be a csv file and just dumps everything into the index on a single line.
i would like to have the csv format when i insert multiple csv files , just like when i do it for single csv file..

how to do this..

Re: question on data inputs for multiple csv files

somesoni2 — Fri, 13 Apr 2018 16:50:42 GMT

Did you try explicitly assigning the sourcetype as csv?

Re: question on data inputs for multiple csv files

jiaqya — Fri, 13 Apr 2018 16:56:24 GMT

no, while on the gui, with multiple files , it didnt allow me to specify the file type, i used a different name for source type other than csv

Re: question on data inputs for multiple csv files

adonio — Fri, 13 Apr 2018 18:00:05 GMT

@jiaqya,
when you add a monitor stanza (monitoring a directory) from the GUI, make user to apply th csv sourcetype to that particular input when you are prompt to do so. exactly like @somesoni2 suggested

Re: question on data inputs for multiple csv files

Sukisen1981 — Fri, 13 Apr 2018 18:18:51 GMT

say you have a folder named New Folder > now place 2 csv files inside that > now settings - data inputs > Files & directories > add new
Give your folder containing the csv files (New Folder) , does it work now?

Re: question on data inputs for multiple csv files

jiaqya — Fri, 13 Apr 2018 18:34:03 GMT

i tried this and provided csv as the source type, doing this now its not even indexing it...
i have 5 files inside the folder, E:\NewFolder* "csv" is the sourcetype , index=test_index
i tried modifying the files, but nothing on the index..

Re: question on data inputs for multiple csv files

Sukisen1981 — Fri, 13 Apr 2018 18:43:20 GMT

hmm this is certainly weird. I just placed 2 files in a folder and used file and directories to just upload the folder, it got indexed.
2 things here and i Know that you must have checked thoroughly but just asking again-
by any chance the files are not excels right? sometimes one forgets to save them as csv
do not click on select and set sourcetype, select 'automatic' which comes by default....

Re: question on data inputs for multiple csv files

jiaqya — Fri, 13 Apr 2018 18:52:24 GMT

no, these are csv files .
now when i do automatic, it is dumping the data but all in a single line.

alternately , when i choose each file, i find that data inserts properly, with correct columns/rows, but when i do multiple csv files like mentioned above, i get things in single line..

Re: question on data inputs for multiple csv files

jiaqya — Fri, 13 Apr 2018 19:15:14 GMT

i tried again adding a new data input, the csv files have around 290 rows each.

strangely enough 72 rows got inserted correctly.. i will have a check tomorrow...

Re: question on data inputs for multiple csv files

jiaqya — Sat, 14 Apr 2018 10:28:38 GMT

i copied these files to dev environment and then did the same process of adding the files with wild card, it works on the dev server and also it comes up with correct format and columns and correct no of rows.

but when i do the same on prod server, it does not index , or its partial or it indexes in single line.

there is some difference between these 2 , have no idea.. both are forwarders...

any ideas ?

Re: question on data inputs for multiple csv files

jiaqya — Tue, 29 Sep 2020 19:02:15 GMT

Full path to your data Set host Source type Index Number of files App Status Actions
/home/testuser/LAN_*.csv Constant Value csv john_test 3 search Enabled | Disable Delete

as you can see under the data inputs , this is from dev, it shows 3 files under "number of files". this was immediate...

on the other servers it shows blank.. its not detecting the existence of files properly...

Re: question on data inputs for multiple csv files

Sukisen1981 — Sat, 14 Apr 2018 14:59:49 GMT

Hi - this is indeed very strange, it looks like some problem with the splunk prod server. I am sorry but this is beyond my knowledge to solve. I retried repeating what I did (using files and directories) , this time i placed 5 different CSVs and they all got indexed with individual rows. What is even more strange is in one of your attempts ,you said that you received 72/290 rows correctly! It should either work or not work! Maybe you need to contact splunk directly about this behaviour