We have a sourcetype for /var/log/messages that is logged in the local server time on almost every host. We have one host, however, which logs /var/log/messages (and some other files) in GMT. The system time on this host is set to the local timezone. This host is sending logs via the universal forwarder. I understand that I need to probably need to modify props.conf on the Splunk host which is indexing the logs. My question is, how do I specify that I only want sourcetype foo from host bar (and ONLY host bar) to be in GMT? Can I do something like this? [source::/var/log/messages AND host::bar] TZ=GMT Thanks! ... View more

We have an internally hosted Splunk environment which indexes logs from most of our internal hosts. We also have a number of hosts in the Amazon cloud which we would like to Splunk. We are looking into the best way to achieve this. I hoped there would be a white paper on the subject, or at least forum posts but I didn't find much. Obviously all environments are unique, but how are people handling this case? The first thing that springs to mind is to add a Splunk forwarder in the DMZ and allow the cloud instances to talk to it with authentication via certificates. The DMZ forwarder would then forward logs into the existing Splunk environment. Is the Universal Forwarder secure enough to expose it to the public Internet? I also ran across this PDF: https://s3.amazonaws.com/aws001/guided_trek/Splunk_and_Amazon_Web_Services_Tech_Brief_Final_6.06.11.pdf On page two they show an environment with indexers and search heads both internally and in the cloud which allows for centralized searching. Does anyone here have any experience with this design? Any thoughts or comments would be appreciated. Thanks, -Graham ... View more

We have a number of MS SQL Server clusters with the Splunk Universal Forwarder installed. We would like to index the SQL Server ERRORLOG and SQLAGENT.OUT files, which live on a disk shared by the cluster members. Only the active member of the cluster will see the shared disk where the errorlog and sqlagent.out files live. The shared disk will always have the same drive letter on whichever node is active. In this case, I am guessing the correct thing to do is to have an identical forwarder configuration on each cluster node. Is that correct? If so, in the case of a failover, will the universal forwarder on a previously inactive node notice that it can suddenly read the errorlog and sqlagent.out files and happily start forwarding events to the indexing host? Or would a restart of the forwarder be required? I understand we would end up with some duplicate events in this case, but we could control that by configuring the earliest indexable event to be very recent. Comments? Thanks! ... View more