Index organization for large number of files

[deanx]

We have 14 directories of log files which contain ~3,100 files. Each day the logs are rotated and 3,100 new files are created with wholly new filenames (eg: Job_ACCOUNTS_PAYABLE_DM_0298.log becomes Job_ACCOUNTS_PAYABLE_DM_0299.log). There might be >300 differently named files like this in each directory.

I am trying to make some order out of this when it gets indexed. The best idea I have come up with is to send each of the different directories as a different sourcetype. That helps, but not entirely .. but it is still alot of sources and I am wondering if others have found a better way?

The application produces these files this way and can not be altered, so I am hoping a good index strategy would help.

[Runals]

Through your props and transforms you could have each of those write to separate sourcetypes- especially if they follow that same general structure. Job_(sourcetype)_\d+.log. I think the bigger question is how different those files really are and how are they going to be used. You could look at doing eventtypes as a way to do high level categorization as well.

[ShaneNewman]

We only index the most current log file, avoiding the need to do what you are having to do... You should not need to add different sourcetypes to each, especially if they are all the same sourcetype at the end of the day. If you are looking for a way to distinguish them, I would use regex on the source to extract host and other distinguishable fields.