topic Re: CSV File Ingestion problem in Getting Data In

CSV File Ingestion problem

rajyah — Mon, 21 Jan 2019 07:44:51 GMT

Good day sirs, would you be so kind to please help me regarding csv file ingestion? Here's the scenario:

When I try to upload the csv manually from remote server, the data within it are ingested. But if I monitor it, it wont. This csv was rsynced from remote server to local, I assumed it was because of permission but I checked it wasnt the case. Are there still anything I need to check? Using btool? How?

Any thoughts will do. Please help.

Thank you.

Re: CSV File Ingestion problem

gcusello — Mon, 21 Jan 2019 08:38:38 GMT

Hi rajyah,
when you say "I monitor" do you mean using a Universal Forwarder?
If yes, at first check if it's correctly configured:

firewall ports are open (check them with telnet <Indexer_Ip_Address> 9997)
Splunk Universal Forwarder is up and running (/opt/splunkforwarder/bin/splunk status)
grants of the csv files are ok (644)
Splunk Forwarder's outputs.conf it's OK (there a room for the Indexer), you can check it searching on Indexer: index=_internal host=your_host
Splunk Forwarder's inputs.conf it's OK (the csv path in the monitor command is correct)

After these checks, you can analyze your situation.

Bye.
Giuseppe

Re: CSV File Ingestion problem

rajyah — Mon, 21 Jan 2019 08:44:24 GMT

Oh, I forgot to mention sir that Im only running single instance without forwarders. I see, so it must be permission. Seems weird though, other CSVs has same grants but was still ingested. Ill double check sir. Thank you and Ill add an update sir after confirming.

Re: CSV File Ingestion problem

rajyah — Mon, 21 Jan 2019 08:45:46 GMT

Is there a way sir to check the splunkd.log? How?

Re: CSV File Ingestion problem

gcusello — Mon, 21 Jan 2019 08:51:38 GMT

Hi rajyah,
in this case, check if the path you configured in your inputs.conf is correct, then be sure that this file isn't the copy of another file because Splunk doesn't index twice a file (you can do it only when you manually index).
To force to reindex, you can put in your inputs.conf the option

crcsalt = <SOURCE>

Anf then change the filename.

Bye.
Giuseppe

Re: CSV File Ingestion problem

gcusello — Mon, 21 Jan 2019 08:53:36 GMT

you can see splunkd.log directly on you Splunk server at

/opt/splunk/var/log/splunk

or using Splunk with this search:

index=_internal sourcetype=splunkd

Bye.
Giuseppe

Re: CSV File Ingestion problem

rajyah — Thu, 24 Jan 2019 10:12:24 GMT

Sir, I just found an oddity behind this problem. If I use 'index once' instead of 'continous monitoring', the data in csv are indexed but when I choose the latter it doesn't. Please enlighten me.

Re: CSV File Ingestion problem

gcusello — Thu, 24 Jan 2019 12:19:29 GMT

Using "once" it's the same thing that manually index.
did you tried crcSalt?

Bye.
Giuseppe

Re: CSV File Ingestion problem

rajyah — Mon, 04 Feb 2019 02:48:38 GMT

Tried crcSalt. Thank you!