Getting Data In

Importing csv files from directory

Sasquatchatmars
Communicator

Hi all,

I have been trying to monitor a directory with csv files. Let me explain. I have multiple PS scripts running and they are exporting the results to csv files in a directory. I have configured a data input on the corresponding directory and whitelisted the csv files. Which gives me the following in the input.conf file. 

 

[monitor://C:\Program Files\Splunk\etc\apps\search\bin\Powershell\Results]
disabled = false
index = powershell_scripts
whitelist = \.csv$

 

Everytime I run a PS script to test if the input works, the script creates the csv file or updates it but it isn't ingested in Splunk. Does someone knows why this could be? 

Thank you,

Sasquatchatmars

Labels (2)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @Sasquatchatmars,

If the result is always the same, the file isn't indexed twice.

If you could run the PS script from Splunk as scripted inputs, you don't have any problem because the script output is sent directly to Splunk.

For more infos, see at https://docs.splunk.com/Documentation/SplunkCloud/latest/AdvancedDev/ScriptedInputsIntro

Ciao.

Giuseppe

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi @Sasquatchatmars,

If the result is always the same, the file isn't indexed twice.

If you could run the PS script from Splunk as scripted inputs, you don't have any problem because the script output is sent directly to Splunk.

For more infos, see at https://docs.splunk.com/Documentation/SplunkCloud/latest/AdvancedDev/ScriptedInputsIntro

Ciao.

Giuseppe

Sasquatchatmars
Communicator

Hi @gcusello,

It doesn't indexes it at all. 

I tried the modular input. Somehow at some points the scripts sees some kind of errors because it is based on a list of servers. These servers are not always working so it generates an error. At that moment the idexing stops and doesn't continue. 

By the way I tried indexing it file by file which works. But what i really want is to monitor all the csv files in the directory without everytime needing to specify the file path in the data inputs.

Thanks,

Bob van Scheijndel

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Sasquatchatmars,

the content of the files is frequently the same or it's always different?

If it's always the same, Splunk doesn't index twice a file also with a different name.

The filenames are always the same or ther are different?

try to add crcSalt = <SOURCE> to the input stanza and restart forwarder.

Ciao.

Giuseppe

0 Karma

Sasquatchatmars
Communicator

Hi @gcusello,

This did work tahnk you but I found I found an easier way. I just added a TimeStamp column to my csv file so the file changes every time. 

Thank you anyway! 

Sasquatchatmars

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Sasquatchatmars,

as I said, Splunk reads a file and, if there are differences, indexs the file or the new lines, otherwise it doean't index the file.

Adding a column with timestamp you modify every time the file so splunk understand that has to index it.

Good for you.

Please accept the answer for the other people of Community.

Ciao and happy splunking.

Giuseppe

P.S.: Karma Points are appreciated 😉

0 Karma

Sasquatchatmars
Communicator

Hi @gcusello,

Thanks yes indeed, you said that 😊

Oh sorry I forgot, I'll accept it right away.

Thank you,

Sasquatchatmars

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...