Mulitiple Files in the same directory

[bandit]

I've seen the documentation and believe there is a way to dynamically do this with props.conf but I'm not understanding how to do it. I my case I'm working with 15 different source types with different file names, but at the same nested directory level.

Only one works at a time, but if both are enabled, only the last one works. Both stanzas below are similar but one has disktool.txt and one has diskview.txt.

inputs.conf
[monitor://\\host.share.comUploadDatasupportdata_Customers...*.disktool.txt] crcSalt = <source> index = eql_disktool sourcetype = disktool

[monitor://\\host.share.comUploadDatasupportdata_Customers...*.diskview.txt] crcSalt = <source> index = eql_diskview sourcetype = diskview

Thanks,

Rob

[dwaddle]

I would recommend an approach similar to this:

(inputs.conf on the forwarder)

[monitor://\\host.share.comUploadDatasupportdata_Customers]
whitelist = disk(view|tool)\.txt$

(props.conf on the forwarder & indexer)

[source::...diskview.txt]
sourcetype=diskview

[source:...disktool.txt]
sourcetype=disktool

[diskview]
TRANSFORMS-index = diskview-index

[disktool]
TRASNFORMS-index = disktool-index

(transforms.conf on the indexer)

[diskview-index]
DEST_KEY=_MetaData:Index
REGEX = .
FORMAT = diskview

[disktool-index]
DEST_KEY=_MetaData:Index
REGEX = .
FORMAT = disktool

This avoids have overlapping (or nearly overlapping) monitor stanzas, and sets the sourcetype of each file by name. Once the sourcetype is set, it uses index-time transforms to move the data into the correct indexes.

[lguinn2]

Wow - a million files is definitely a performance problem. Are all the files "live" or are some of them stale? Check out some of the inputs.conf settngs - or better yet, move stale files to another directory after some appropriate time lapse (like a week).

[bandit]

I'm now thinking this may be just a performance issue since a single indexer is trying to ingest more than a million files. It may be just working through one rule at a time. That would make sense why each rule works individually.

[bandit]

Thanks, will let you know

[lukejadamec]

Try it without the crcsalt, and see if you get my results. I have not used that yet, because it is bad juju.

[bandit]

For me, I get the same behavior on my local laptop with no share. Doesn't seem to like the combination of wilcard ... and a similar path. If I disable the last source, the next to last source starting indexing events 🙂

[lukejadamec]

There is definitely something wrong with shares. I cannot get this to break on local drives. I'll test it on shares tomorrow.

[bandit]

Rule works as long as you only have one monitor stanza active otherwise it seems to conflict with others.