topic Re: Mulitiple Files in the same directory in Getting Data In

Mulitiple Files in the same directory

bandit — Thu, 10 Oct 2013 22:27:06 GMT

I've seen the documentation and believe there is a way to dynamically do this with props.conf but I'm not understanding how to do it. I my case I'm working with 15 different source types with different file names, but at the same nested directory level.

Only one works at a time, but if both are enabled, only the last one works. Both stanzas below are similar but one has disktool.txt and one has diskview.txt.

inputs.conf
[monitor://\\host.share.comUploadDatasupportdata_Customers...*.disktool.txt] crcSalt = <source> index = eql_disktool sourcetype = disktool

[monitor://\\host.share.comUploadDatasupportdata_Customers...*.diskview.txt] crcSalt = <source> index = eql_diskview sourcetype = diskview

Thanks,

Rob

Re: Mulitiple Files in the same directory

bandit — Thu, 10 Oct 2013 23:39:09 GMT

Rule works as long as you only have one monitor stanza active otherwise it seems to conflict with others.

Re: Mulitiple Files in the same directory

lukejadamec — Fri, 11 Oct 2013 00:50:17 GMT

There is definitely something wrong with shares. I cannot get this to break on local drives. I'll test it on shares tomorrow.

Re: Mulitiple Files in the same directory

bandit — Fri, 11 Oct 2013 01:37:35 GMT

For me, I get the same behavior on my local laptop with no share. Doesn't seem to like the combination of wilcard ... and a similar path. If I disable the last source, the next to last source starting indexing events 🙂

Re: Mulitiple Files in the same directory

lukejadamec — Fri, 11 Oct 2013 01:43:40 GMT

Try it without the crcsalt, and see if you get my results. I have not used that yet, because it is bad juju.

Re: Mulitiple Files in the same directory

bandit — Fri, 11 Oct 2013 01:44:58 GMT

Thanks, will let you know

Re: Mulitiple Files in the same directory

bandit — Fri, 11 Oct 2013 21:56:21 GMT

I'm now thinking this may be just a performance issue since a single indexer is trying to ingest more than a million files. It may be just working through one rule at a time. That would make sense why each rule works individually.

Re: Mulitiple Files in the same directory

lguinn2 — Mon, 14 Oct 2013 08:51:21 GMT

Wow - a million files is definitely a performance problem. Are all the files "live" or are some of them stale? Check out some of the inputs.conf settngs - or better yet, move stale files to another directory after some appropriate time lapse (like a week).

Re: Mulitiple Files in the same directory

dwaddle — Mon, 14 Oct 2013 13:31:27 GMT

I would recommend an approach similar to this:

(inputs.conf on the forwarder)

[monitor://\\host.share.comUploadDatasupportdata_Customers]
whitelist = disk(view|tool)\.txt$

(props.conf on the forwarder & indexer)

[source::...diskview.txt]
sourcetype=diskview

[source:...disktool.txt]
sourcetype=disktool

[diskview]
TRANSFORMS-index = diskview-index

[disktool]
TRASNFORMS-index = disktool-index

(transforms.conf on the indexer)

[diskview-index]
DEST_KEY=_MetaData:Index
REGEX = .
FORMAT = diskview

[disktool-index]
DEST_KEY=_MetaData:Index
REGEX = .
FORMAT = disktool

This avoids have overlapping (or nearly overlapping) monitor stanzas, and sets the sourcetype of each file by name. Once the sourcetype is set, it uses index-time transforms to move the data into the correct indexes.