topic Directory / Source in Getting Data In

Directory / Source

rene847 — Fri, 06 Feb 2015 14:33:05 GMT

Hi,

First: I have a directory with five sources. Sometimes a source (always the same) is not loaded into Splunk even if the file has been modified, I dont know why yet the file date is really changed)? Its 5 CSV files.
What I want to do, instead of modifying the file, my script will delete the file and create a new one (instead of changing its content) ... I'm looking for solutions.

Finally, there is there way to monitor whether the five sources were picked up by splunk (If only 4 sources, alert me). I am looking for an query that would make this, do you have an idea?

Thank you
Best regards

Re: Directory / Source

jpvlsmv — Fri, 06 Feb 2015 15:07:53 GMT

Splunk looks at the beginning of a file (the first 5 or so lines) to see if it's already been seen, look at the documentation for crcSalt in props.conf.

I had near-duplicate files that were not being indexed until I set in props.conf crcSalt=<SOURCE>. I found out about this by reviewing "index=_internal sourcetype=splunkd" and the filename that was missed

With this setting, Splunk used the file name as part of the file identification process, and my nearly-identical files (with different paths) were happily imported.

As for monitoring, I would suggest that you could search the splunkd.log and look for the filenames. I can't tell you the exact syntax, but you may be able to group the relevant lines into a transaction, and count how many there are. If there are less than 5, splunk has missed a file.

Re: Directory / Source

rene847 — Mon, 28 Sep 2020 18:50:57 GMT

This is not because "crc".
I takes the contents, I insert it in another file, I put the same name file in the directory and Hop, Splunk takes.
It's really weird because, as I say, with another file, Splunk takes the file and when I query, I found data !!!!

Sorry, I have 6 files in the folder (not 5):
DcServerX_eventlogs_application.CSV
DcServerXeventlogs_DS.csv
DcServerXeventlogs_FRS.csv
DcerverXeventlogs_security.csv
DcServerXeventlogs_system.csv
And the file DcServerX_eventlogs_DNS.csv
This is the file that causes me problems sometimes and I do not know why.

My Inputs.conf is (fonctional for 5 files??):
[monitor://D:SplunkDataInputTI-WINEVENTS-RNI.csv]
index = TI-WINEVENTS-RNI
host_segment = 3

What do you think for solution?
.
.
.
Here's a search (Newest to oldest) :
.
.
04/02/15 04:17:39,455

02-04-2015 04:17:39.455 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:/SplunkDataInput/TI-WINEVENTS-RNI/BLABLABLA_eventlogs_DNS.csv'.
host = TOTO source = C:\Program Files/SplunkUniversalForwarder/var/log/splunk/splunkd.log sourcetype = splunkd
.
.
03/02/15 04:15:58,401

02-03-2015 04:15:58.401 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='D:\SplunkDataInput\TI-WINEVENTS-RNI\BLABLABLA_eventlogs_DNS.csv'.
host = TOTO source = C:/Program Files/SplunkUniversalForwarder/var/log/splunk/splunkd.log sourcetype = splunkd
.
.
03/02/15 04:15:58,401

02-03-2015 04:15:58.401 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='D:/SplunkDataInput/TI-WINEVENTS-RNI/BLABLABLAeventlogs_DNS.csv'.
host = TOTO source = C:/Program Files/SplunkUniversalForwarder/var/log/splunk/splunkd.log sourcetype = splunkd
.
.
02/02/15 04:15:18,748

02-02-2015 04:15:18.748 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='D:/SplunkDataInput/TI-WINEVENTS-RNI/BLABLABLA_eventlogs_DNS.csv'.
host = TOTO source = C:/Program Files/SplunkUniversalForwarder/var/log/splunk/splunkd.log sourcetype = splunkd
.
.

Re: Directory / Source

rene847 — Mon, 28 Sep 2020 18:51:00 GMT

Real time, I put a new file (with the same content), here the log:
.
.
06/02/15 19:45:51,454

02-06-2015 19:45:51.454 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:/SplunkDataInput/TI-WINEVENTS-RNI/BLABLABLA_eventlogs_DNS.csv'.
host = TOTO source = C:/Program Files/SplunkUniversalForwarder/var/log/splunk/splunkd.log sourcetype = splunkd
.
.
I put the old file and :
.
06/02/15 19:46:07,569

02-06-2015 19:46:07.569 +0000 INFO WatchedFile - Will begin reading at offset=3717 for file='D:/SplunkDataInput/TI-WINEVENTS-RNI/BLABLABLA_eventlogs_DNS.csv'.
host = TOTO source = C:/Program Files/SplunkUniversalForwarder/var/log/splunk/splunkd.log sourcetype = splunkd

Re: Directory / Source

jpvlsmv — Mon, 09 Feb 2015 14:41:35 GMT

The "new file" with the same content is being reported as too small to index... perhaps splunk is trying to read the file before it has finished copying into place?

Otherwise, I'm out of ideas 🙂 Sorry.

Re: Directory / Source

rene847 — Mon, 09 Feb 2015 15:50:50 GMT

Thank for your support, I appreciate !
Best Regards

Re: Directory / Source

rene847 — Mon, 09 Feb 2015 17:38:59 GMT

There he has someone who has an idea of my problem?

Re: Directory / Source

rene847 — Wed, 17 Jun 2015 18:51:03 GMT

Finally,
We destroyed the directory, create a new, new query and now functional
weird