I've seen the documentation and believe there is a way to dynamically do this with props.conf but I'm not understanding how to do it. I my case I'm working with 15 different source types with different file names, but at the same nested directory level.
Only one works at a time, but if both are enabled, only the last one works. Both stanzas below are similar but one has disktool.txt and one has diskview.txt.
inputs.conf [monitor://\\host.share.comUploadDatasupportdata_Customers...*.disktool.txt] crcSalt = <source> index = eql_disktool sourcetype = disktool [monitor://\\host.share.comUploadDatasupportdata_Customers...*.diskview.txt] crcSalt = <source> index = eql_diskview sourcetype = diskview
I would recommend an approach similar to this:
(inputs.conf on the forwarder)
[monitor://\\host.share.comUploadDatasupportdata_Customers] whitelist = disk(view|tool)\.txt$
(props.conf on the forwarder & indexer)
[source::...diskview.txt] sourcetype=diskview [source:...disktool.txt] sourcetype=disktool [diskview] TRANSFORMS-index = diskview-index [disktool] TRASNFORMS-index = disktool-index
(transforms.conf on the indexer)
[diskview-index] DEST_KEY=_MetaData:Index REGEX = . FORMAT = diskview [disktool-index] DEST_KEY=_MetaData:Index REGEX = . FORMAT = disktool
This avoids have overlapping (or nearly overlapping) monitor stanzas, and sets the sourcetype of each file by name. Once the sourcetype is set, it uses index-time transforms to move the data into the correct indexes.
Wow - a million files is definitely a performance problem. Are all the files "live" or are some of them stale? Check out some of the
inputs.conf settngs - or better yet, move stale files to another directory after some appropriate time lapse (like a week).
I'm now thinking this may be just a performance issue since a single indexer is trying to ingest more than a million files. It may be just working through one rule at a time. That would make sense why each rule works individually.
For me, I get the same behavior on my local laptop with no share. Doesn't seem to like the combination of wilcard ... and a similar path. If I disable the last source, the next to last source starting indexing events 🙂