How to end File monitored data input from HF to Splunk Cloud?

[SplunkExplorer]

Hi Splunkers, I have to forward data inside csv files from an on prem HF to Splunk Cloud and I'm facing some issues, cause data seem to not be forwarded. Let me share with you some additional bits.

Info about data

• Source data are on a cloud instance (Forcepoint) provided by vendor
• A script has been provided by vendor to pull data from cloud
• The script is installed and configured on our Splunk HF
• Data are saved locally on HF
• Data are in .csv files 

Info about HF configuration

• We create a new data inputs under Settings -> Data inputs -> Local inputs -> Files & Directories
• We set as data input the path were .csv are saved after script execution
• We set the proper sourcetype and index
• Of course, we configured the HF to send data to Splunk Cloud. We downloaded the file from cloud, from "Universal Forwarder" app and installed it as app on HF: the outputs.conf is proper configured, other data are sent without problem to Splunk cloud (for example, Network input ones goes to Cloud without issues; same for Windows ones)

Info about sourcetype and index and their deployment

• We create a custom addon that simply provide the sourcetype "forcepoint"
• Sourcetype is configured to extract data from CSV; that means that we set parameter 

Indexed_extractions=csv ​

• We installed addon on both HF and Splunk Cloud
• The index, called simply "web", has been created on both HF and Splunk Cloud

By thw way, seems that data are not sent from HF to Cloud. So, did I forgot some steps? Or I made wrong some of above ones?

[SplunkExplorer · Accepted solution]

I performed all checks suggested and nothing seem to be wrong; after more than 1 day, logs start to come to cloud. My assumption is that some latency problems delayed log receiving and, after initial burst, they start to come.

[richgalloway]

You have the right steps, but perhaps something in the details is amiss.

Verify the inputs.conf stanza points to the correct file/directory.

Verify the file permissions allows reading by the HF.

Check the splunkd.log files on the HF to see if any messages might explain why the file is not uploaded.

Confirm the CSV file has timestamps for each event and that the timestamps are correctly extracted.  Timestamps that are in the future or too far in the past will not be found by Splunk.  Try searching a wide time range to see if the data has bad timestamps

index=web earliest=0 latest=+10y

[SplunkExplorer]

Hi @richgalloway, thanks for your answer.

I can share with you some other bits.

• Previously, we used another sourcetype provided by a Splunk supported addon, which now can no longer be used after a check with support. Even if with some problems, data was sent to cloud while using it, so the HF has the right permission to read pulled csv files.
• I tested the custom addon on a local test environment and here all data are correctly extracted, even timestamp.
• I thought about inputs.conf file, but not sure about which one I have to analyze: the one in SPLUNK_HOME/etc/system/local? The one on SPLUNK_HOME/etc/system/default? Others?

[richgalloway]

I'm curious about why a sourcetype can no longer be used.  Sourcetypes never expire.  Perhaps it's an add-on that can't be used?

The inputs.conf file to check is the one that references the file or directory we're talking about.  Use btool to find it.

splunk btool --debug inputs list | grep "<<CSV file or directory name>>"

Have you checked the logs?

Have you tried the search I suggested?

Have you tried looking in other indexes?

[SplunkExplorer]

You are right, the problem is in the addon linked to previous sourcetype.

Thanks for your suggestions, I have all data I need to perform analysis. I'm going to do them.