topic Re: Importing csv files from directory in Getting Data In

Importing csv files from directory

Sasquatchatmars — Tue, 01 Dec 2020 13:05:32 GMT

Hi all,

I have been trying to monitor a directory with csv files. Let me explain. I have multiple PS scripts running and they are exporting the results to csv files in a directory. I have configured a data input on the corresponding directory and whitelisted the csv files. Which gives me the following in the input.conf file.

[monitor://C:\Program Files\Splunk\etc\apps\search\bin\Powershell\Results] disabled = false index = powershell_scripts whitelist = \.csv$

Everytime I run a PS script to test if the input works, the script creates the csv file or updates it but it isn't ingested in Splunk. Does someone knows why this could be?

Thank you,

Sasquatchatmars

Re: Importing csv files from directory

gcusello — Tue, 01 Dec 2020 13:10:09 GMT

Hi @Sasquatchatmars,

If the result is always the same, the file isn't indexed twice.

If you could run the PS script from Splunk as scripted inputs, you don't have any problem because the script output is sent directly to Splunk.

For more infos, see at https://docs.splunk.com/Documentation/SplunkCloud/latest/AdvancedDev/ScriptedInputsIntro

Ciao.

Giuseppe

Re: Importing csv files from directory

Sasquatchatmars — Tue, 01 Dec 2020 13:19:43 GMT

Hi @gcusello,

It doesn't indexes it at all.

I tried the modular input. Somehow at some points the scripts sees some kind of errors because it is based on a list of servers. These servers are not always working so it generates an error. At that moment the idexing stops and doesn't continue.

By the way I tried indexing it file by file which works. But what i really want is to monitor all the csv files in the directory without everytime needing to specify the file path in the data inputs.

Thanks,

Bob van Scheijndel

Re: Importing csv files from directory

gcusello — Tue, 01 Dec 2020 13:45:12 GMT

Hi @Sasquatchatmars,

the content of the files is frequently the same or it's always different?

If it's always the same, Splunk doesn't index twice a file also with a different name.

The filenames are always the same or ther are different?

try to add crcSalt = <SOURCE> to the input stanza and restart forwarder.

Ciao.

Giuseppe

Re: Importing csv files from directory

Sasquatchatmars — Wed, 02 Dec 2020 07:16:26 GMT

Hi @gcusello,

This did work tahnk you but I found I found an easier way. I just added a TimeStamp column to my csv file so the file changes every time.

Thank you anyway!

Sasquatchatmars

Re: Importing csv files from directory

gcusello — Wed, 02 Dec 2020 07:28:56 GMT

Hi @Sasquatchatmars,

as I said, Splunk reads a file and, if there are differences, indexs the file or the new lines, otherwise it doean't index the file.

Adding a column with timestamp you modify every time the file so splunk understand that has to index it.

Good for you.

Please accept the answer for the other people of Community.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: Importing csv files from directory

Sasquatchatmars — Wed, 02 Dec 2020 08:29:34 GMT

Hi @gcusello,

Thanks yes indeed, you said that 😊

Oh sorry I forgot, I'll accept it right away.

Thank you,

Sasquatchatmars