Getting Data In

Directory / Source

rene847
Path Finder

Hi,

First: I have a directory with five sources. Sometimes a source (always the same) is not loaded into Splunk even if the file has been modified, I dont know why yet the file date is really changed)? Its 5 CSV files.
What I want to do, instead of modifying the file, my script will delete the file and create a new one (instead of changing its content) ... I'm looking for solutions.

Finally, there is there way to monitor whether the five sources were picked up by splunk (If only 4 sources, alert me). I am looking for an query that would make this, do you have an idea?

Thank you
Best regards

0 Karma
1 Solution

rene847
Path Finder

Finally,
We destroyed the directory, create a new, new query and now functional
weird

View solution in original post

0 Karma

rene847
Path Finder

Finally,
We destroyed the directory, create a new, new query and now functional
weird

0 Karma

rene847
Path Finder

There he has someone who has an idea of my problem?

0 Karma

rene847
Path Finder

Real time, I put a new file (with the same content), here the log:
.
.
06/02/15 19:45:51,454

02-06-2015 19:45:51.454 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:/SplunkDataInput/TI-WINEVENTS-RNI/BLABLABLA_eventlogs_DNS.csv'.
host = TOTO source = C:/Program Files/SplunkUniversalForwarder/var/log/splunk/splunkd.log sourcetype = splunkd
.
.
I put the old file and :
.
06/02/15 19:46:07,569

02-06-2015 19:46:07.569 +0000 INFO WatchedFile - Will begin reading at offset=3717 for file='D:/SplunkDataInput/TI-WINEVENTS-RNI/BLABLABLA
_eventlogs_DNS.csv'.
host = TOTO source = C:/Program Files/SplunkUniversalForwarder/var/log/splunk/splunkd.log sourcetype = splunkd

0 Karma

jpvlsmv
Path Finder

The "new file" with the same content is being reported as too small to index... perhaps splunk is trying to read the file before it has finished copying into place?

Otherwise, I'm out of ideas 🙂 Sorry.

0 Karma

rene847
Path Finder

Thank for your support, I appreciate !
Best Regards

0 Karma

rene847
Path Finder

This is not because "crc".
I takes the contents, I insert it in another file, I put the same name file in the directory and Hop, Splunk takes.
It's really weird because, as I say, with another file, Splunk takes the file and when I query, I found data !!!!

Sorry, I have 6 files in the folder (not 5):
DcServerX_eventlogs_application.CSV
DcServerX
eventlogs_DS.csv
DcServerX
eventlogs_FRS.csv
DcerverX
eventlogs_security.csv
DcServerX
eventlogs_system.csv
And the file DcServerX
_eventlogs_DNS.csv
This is the file that causes me problems sometimes and I do not know why.

My Inputs.conf is (fonctional for 5 files??):
[monitor://D:SplunkDataInputTI-WINEVENTS-RNI.csv]
index = TI-WINEVENTS-RNI
host_segment = 3

What do you think for solution?
.
.
.
Here's a search (Newest to oldest) :
.
.
04/02/15 04:17:39,455

02-04-2015 04:17:39.455 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='D:/SplunkDataInput/TI-WINEVENTS-RNI/BLABLABLA_eventlogs_DNS.csv'.
host = TOTO source = C:\Program Files/SplunkUniversalForwarder/var/log/splunk/splunkd.log sourcetype = splunkd
.
.
03/02/15 04:15:58,401

02-03-2015 04:15:58.401 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='D:\SplunkDataInput\TI-WINEVENTS-RNI\BLABLABLA_eventlogs_DNS.csv'.
host = TOTO source = C:/Program Files/SplunkUniversalForwarder/var/log/splunk/splunkd.log sourcetype = splunkd
.
.
03/02/15 04:15:58,401

02-03-2015 04:15:58.401 +0000 INFO WatchedFile - Checksum for seekptr didn't match, will re-read entire file='D:/SplunkDataInput/TI-WINEVENTS-RNI/BLABLABLA
eventlogs_DNS.csv'.
host = TOTO source = C:/Program Files/SplunkUniversalForwarder/var/log/splunk/splunkd.log sourcetype = splunkd
.
.
02/02/15 04:15:18,748

02-02-2015 04:15:18.748 +0000 INFO WatchedFile - Will begin reading at offset=0 for file='D:/SplunkDataInput/TI-WINEVENTS-RNI/BLABLABLA
_eventlogs_DNS.csv'.
host = TOTO source = C:/Program Files/SplunkUniversalForwarder/var/log/splunk/splunkd.log sourcetype = splunkd
.
.

0 Karma

jpvlsmv
Path Finder

Splunk looks at the beginning of a file (the first 5 or so lines) to see if it's already been seen, look at the documentation for crcSalt in props.conf.

I had near-duplicate files that were not being indexed until I set in props.conf crcSalt=<SOURCE>. I found out about this by reviewing "index=_internal sourcetype=splunkd" and the filename that was missed

With this setting, Splunk used the file name as part of the file identification process, and my nearly-identical files (with different paths) were happily imported.

As for monitoring, I would suggest that you could search the splunkd.log and look for the filenames. I can't tell you the exact syntax, but you may be able to group the relevant lines into a transaction, and count how many there are. If there are less than 5, splunk has missed a file.

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...