All Apps and Add-ons

Upload CSV files for Monitoring using Splunk Universal Forwarder

Explorer

Hi
I have a Splunk Universal Forwarder installed on Windows Systems and I am able to get Installed Softwares (1st phase PoC)
Now I intend to get CSV reports from AV server for all Windows Systems and use them to further analyse my Systems Status.
The AV CSV report will be updated on a daily basis by IT team and I intend to pick up the changes only and update my analysis.

I have tried to do a pilot run of uploading a CSV file using UF on my own windows 10 system as per below steps:
1. Created a custom CSV file.
2. Stopped the UF

  1. Added a monitor command in the inputs.conf file at the path
    C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkTAwindows\local

  2. The inputs.conf entry reads as below:
    [monitor://C:\Users<UID>\Desktop\splunk*.csv]
    disabled = 0
    index = index1
    sourcetype = csv1

  3. Restarted the Splunk UF

  4. I could see the logs in the Index

  5. Prob 1: Now I tried to change the CSV file and added some more rows but the same were not immidiately visible.

  6. Prob 2: I tried to create a new Index index2 and change the inputs.conf file to redirect the logs to new index, but I see no logs in SPlunk Search

  7. Prob3: I have created a completely new file and changed its location but kept the Index to index1, but still I dont see any logs.

I am currently perplexed as to how exactly the Splunk Forwarder will behave.

P.S. I have not edited the props.conf or transform.conf files, as I am not sure that they are needed.

Any HELP highly Appreciated

Regards
VS

0 Karma

Motivator

Here are few things to check:

  • First obvious, restart splunk after every change to .conf files.

  • If you're monitoring a CSV file that doesn't have a timestamp column it is possible that your events are being timestamped incorrectly. If you're able, try searching a much broader time range (All-Time) to see if anything from your CSV is being indexed? --- Add a timestamp column to the CSV and have it populated on every row

  • $SPLUNK_HOME/var/log/splunk/splunkd.log should show that it starts monitoring the folder and then should mention it detected some file etc. --- Look for any errors/warnings at the time of restart.

  • Splunk uses intiCrcLength and crcSalt settings to determine how much of a file (by default, the first 256 bytes of a file) the input reads before trying to identify whether it is a file that has already been seen. This behavior prevents the input from indexing the same file twice, even though you may have renamed it. You might want to adjust this.

Read more here. For example adding crcSalt = readitagain under monitor stanza will re-index the entire file one more time.

Finally, I recommend checking out Lookup File Editor app. That app is free and it allows you to make new lookup files and edit them in a nice interface.

0 Karma

Motivator

UF and Indexer are on same machine?

0 Karma

Explorer

No Indexer is on Unix Server and UF is on Windows system

0 Karma

Motivator

@vikfnu
Are you appending data to the file or changing the existing data?

0 Karma

Explorer

1 I want to test upload of CSV file via UF to an Index
2. I want to test changes in the CSV files to be uploaded via UF to same Index

0 Karma

Motivator

@vikfnu I understood your point. 1. To upload the data, you can use monitor stanza on UF. That will work fine.

  1. For monitoring file: splunk works on CRC check pointers, so if new data arrives in the file means only append then only it will be able to process the data. And there is also polling time as well, which will create a little delay in data to reach splunk indexer.
0 Karma