question on data inputs for multiple csv files

[jiaqya]

i see that i can chose the single csv file type for a csv file and verify the columns are right and then insert into the index via data input option

but when i have multiple csv files in a folder, i try to enter the name with wildcard ie * , and this time it just does not consider it to be a csv file and just dumps everything into the index on a single line.
i would like to have the csv format when i insert multiple csv files , just like when i do it for single csv file..

how to do this..

[somesoni2]

Did you try explicitly assigning the sourcetype as csv?

[jiaqya]

no, while on the gui, with multiple files , it didnt allow me to specify the file type, i used a different name for source type other than csv

[Sukisen1981]

say you have a folder named New Folder > now place 2 csv files inside that > now settings - data inputs > Files & directories > add new
Give your folder containing the csv files (New Folder) , does it work now?

[jiaqya]

i tried this and provided csv as the source type, doing this now its not even indexing it...
i have 5 files inside the folder, E:\NewFolder* "csv" is the sourcetype , index=test_index
i tried modifying the files, but nothing on the index..

[Sukisen1981]

hmm this is certainly weird. I just placed 2 files in a folder and used file and directories to just upload the folder, it got indexed.
2 things here and i Know that you must have checked thoroughly but just asking again-
by any chance the files are not excels right? sometimes one forgets to save them as csv
do not click on select and set sourcetype, select 'automatic' which comes by default....

[jiaqya]

no, these are csv files .
now when i do automatic, it is dumping the data but all in a single line.

alternately , when i choose each file, i find that data inserts properly, with correct columns/rows, but when i do multiple csv files like mentioned above, i get things in single line..

[jiaqya]

i tried again adding a new data input, the csv files have around 290 rows each.

strangely enough 72 rows got inserted correctly.. i will have a check tomorrow...

[jiaqya]

i copied these files to dev environment and then did the same process of adding the files with wild card, it works on the dev server and also it comes up with correct format and columns and correct no of rows.

but when i do the same on prod server, it does not index , or its partial or it indexes in single line.

there is some difference between these 2 , have no idea.. both are forwarders...

any ideas ?

[jiaqya]

Full path to your data Set host Source type Index Number of files App Status Actions
/home/testuser/LAN_*.csv Constant Value csv john_test 3 search Enabled | Disable Delete

as you can see under the data inputs , this is from dev, it shows 3 files under "number of files". this was immediate...

on the other servers it shows blank.. its not detecting the existence of files properly...

[Sukisen1981]

Hi - this is indeed very strange, it looks like some problem with the splunk prod server. I am sorry but this is beyond my knowledge to solve. I retried repeating what I did (using files and directories) , this time i placed 5 different CSVs and they all got indexed with individual rows. What is even more strange is in one of your attempts ,you said that you received 72/290 rows correctly! It should either work or not work! Maybe you need to contact splunk directly about this behaviour

[adonio]

@jiaqya,
when you add a monitor stanza (monitoring a directory) from the GUI, make user to apply th csv sourcetype to that particular input when you are prompt to do so. exactly like @somesoni2 suggested