Thanks EmEdwards, that did the trick. ... View more

I think so, for now at least. ... View more

Unfortunately right now log files are the only option, they have discussed being able to forward logs to other systems but as of right now that requires recompiling dll's and a few other things, and it would only end up in SQL server which would then need a license for plus would also introduce a further delay. I think for now dedup or mark the duplicate records as deleted and later on hopefully they will add the syslog option. It would be the ideal scenario and should be relatively easy from a coder perspective. ... View more

Sadly this app is not of of those that knows what syslog is. There is scope for having the vendor add it, down the road but this will take some time. thanks, ... View more

Unfortunately the logs primary function is within the application, they are used by the users of the application and so need to be synchronised across both machines. So which ever machine is active there is a complete list available to the user. So they must be replicated across the 2 systems. But yes completely agree if the logs did not need to replicated across both systems this would not be an issues. ... View more

Don't really like the idea of using up double the license, but looks like that might have to be the way. ... View more

I'm sorry, I'm not sure what you mean exactly. But the application that generates the logs has no concept of what syslog is, it can only write to a file which is then rolled over once per day (usually) can be more often. ... View more

I've been using dedup, but was hoping there was a way to no index it to begin with, as the log files are identical and add no value to the index. The application also uses the log files internally for it's users to query in the native environment. If the files aren't synchronised between the servers then they will get different results depending on which is the current active server. Either can be active at any one time. ... View more

Monitoring the file before it's renamed will index all the data but once it's renamed it will index the new file will it not? So there will be duplicate entries in the index. Or is it smart enough to know the records are the same and wont index them again? I guess will just have to try something and see what happens. thanks, Tony ... View more

Thanks for the response gokadroid. The problem is not selecting the file to index, the problem I'm trying to avoid is that if I index the file as it's being created when it's finished and renamed it will be indexed again and I'll end up with duplicate entries. For instance, say we had the following set of files. I'm currently indexing everything except file 3. File 3 is the file that is currently being written to by the system. file 3: >logfile_20170103100502_LIVE.log file 2: >logfile_20170102100503_20170103100502.log file 1: >logfile_20170101100501_20170102100503_.log etc.... Once file 3 rolls over at the end of the period a new file is created and we end up with something like the following: file 4: >logfile_20170104100501_LIVE.log file 3: >logfile_20170103100502_20170104100501.log file 2: >logfile_20170102100503_20170103100502_.log file 1: >logfile_20170101100501_20170102100503_.log etc.... At this point file 3 is indexed. I would like to index file 4 in this case as I would like to report on information in "real time" if I can. as well as not double index any of the data. Also another problem would be that once the data is indexed the index file it belonged to will no longer exist so not sure what it would do then either. Hope this clears up the problem. Thanks. Tony ... View more

Hi Giuseppe, Thanks for the reply. The index is configured as per the default for Splunk, max size of 500Gb and don't believe there is a limit on retention. The files are on the indexer, I've done the search many days after the original import and has not changed. Probably should have tried the crSalt option. The weird thing is there is data before and after the missing sections, very weird, I initially thought it may have been something to do with the original source files maybe not being formated quite right or something but I looked at ones that worked and ones that didn't and couldn't really find a problem with them. At the moment I'm dropping one file into the monitored folder every minute and it appears to be picking up everything, just have to wait for it catch finished dropping in another 6 years worth of files to finish. Tony ... View more

Thanks for the suggestion jeff, I didn't see that as an option under the syslog: stanza but gave it a go and it didn't do anything. I also tried putting it in the computer sending logs to the heavy forwarder in it's tcpout: stanza and it pretty much blocked everything, I could see some data between the the computer and the heavy forwarder but nothing was going to the siem, I assume because the heavy forwarder didn't know how to process any of the uncooked data. ... View more