Getting Data In

Mulitiple Files in the same directory

bandit
Motivator

I've seen the documentation and believe there is a way to dynamically do this with props.conf but I'm not understanding how to do it. I my case I'm working with 15 different source types with different file names, but at the same nested directory level.

Only one works at a time, but if both are enabled, only the last one works. Both stanzas below are similar but one has disktool.txt and one has diskview.txt.

inputs.conf
[monitor://\\host.share.comUploadDatasupportdata_Customers...*.disktool.txt] crcSalt = <source> index = eql_disktool sourcetype = disktool

[monitor://\\host.share.comUploadDatasupportdata_Customers...*.diskview.txt] crcSalt = <source> index = eql_diskview sourcetype = diskview

Thanks,

Rob

dwaddle
SplunkTrust
SplunkTrust

I would recommend an approach similar to this:

(inputs.conf on the forwarder)

[monitor://\\host.share.comUploadDatasupportdata_Customers]
whitelist = disk(view|tool)\.txt$

(props.conf on the forwarder & indexer)

[source::...diskview.txt]
sourcetype=diskview

[source:...disktool.txt]
sourcetype=disktool

[diskview]
TRANSFORMS-index = diskview-index

[disktool]
TRASNFORMS-index = disktool-index

(transforms.conf on the indexer)

[diskview-index]
DEST_KEY=_MetaData:Index
REGEX = .
FORMAT = diskview

[disktool-index]
DEST_KEY=_MetaData:Index
REGEX = .
FORMAT = disktool

This avoids have overlapping (or nearly overlapping) monitor stanzas, and sets the sourcetype of each file by name. Once the sourcetype is set, it uses index-time transforms to move the data into the correct indexes.

lguinn2
Legend

Wow - a million files is definitely a performance problem. Are all the files "live" or are some of them stale? Check out some of the inputs.conf settngs - or better yet, move stale files to another directory after some appropriate time lapse (like a week).

0 Karma

bandit
Motivator

I'm now thinking this may be just a performance issue since a single indexer is trying to ingest more than a million files. It may be just working through one rule at a time. That would make sense why each rule works individually.

0 Karma

bandit
Motivator

Thanks, will let you know

0 Karma

lukejadamec
Super Champion

Try it without the crcsalt, and see if you get my results. I have not used that yet, because it is bad juju.

0 Karma

bandit
Motivator

For me, I get the same behavior on my local laptop with no share. Doesn't seem to like the combination of wilcard ... and a similar path. If I disable the last source, the next to last source starting indexing events 🙂

0 Karma

lukejadamec
Super Champion

There is definitely something wrong with shares. I cannot get this to break on local drives. I'll test it on shares tomorrow.

0 Karma

bandit
Motivator

alt text

Rule works as long as you only have one monitor stanza active otherwise it seems to conflict with others.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...