I'm migrating from using a tcp input to using syslog-ng to write from tcp to disk and then from disk into Splunk. This brings up the angle on how we manage the syslog-ng process being started etc. Is there a good way to have Splunk do this? It seems appropriate to actually deploy the new binaries, and it's config(s) in an app, but I still need a responsible way to have it start that can remain within the remit of the Splunk admins, not the wider world that look after the kit the UF is running on. I guess I could use a scripted input to run a wrapper script, which will start the syslog-ng process as a daemon if it's not running, with the output of the script being a suitable log message about the processes status (and maybe other stats whilst we're there for good luck) but I'm not sure if this is an ugly hack or an nice idea. I'm certainly aware there are more conventional ways to do this, but it's important that this be kept within the limited Splunk scope due to the way the company works. ... View more

Hi, I've heard comments against configuring Splunk to read gzipped files, horror stories of it not always noticing the file was indeed a gz and logging the compressed raw data instead. I'm looking to piggyback on an existing process that drops a pile of gzipped logs onto a server with a universal forwarder already installed, and don't want to have to delve into custom scripts to first decompress the files to a temp location if there are no genuine known concerns around Splunk's consistent reliability when it comes to indexing gzipped files.. ... View more

Howdy, I've a load balancer which is happily sending event logs when certain events happen in a web app flow. It will send a log when an HTTP request is made, when the server responds, when it serves from cache, when it fails a security violation, when it can't reach the server it needed to etc. My goal is to dump rich and "stateless" data about each event, and deal with it later. I want to build the logging logic in the logging systems, not in the load balancer, as for one reason, it's much much easier to change out of band systems like a logging server then in band network architecture. This later would be, in one example, building Apache Combined style logs to make visible to those who know and understand that log format. I could do this dynamically using a transaction search (and by inserting a random tracking ID into each log message etc.) but this is slow and pretty intensive. Given that I know I want one final log entry to reflect the life of every single web request made, is there a way to build these new log entries dynamically at index time? I am pretty sure I could trivially schedule a cron job to search and re log the results it creates of the previous 5 minutes etc, but I'm wondering if there's a slick integrated way to do this instead where upon getting one of the various responses to an http request (response, cache, violation, failure) I can quickly find the matching incoming request and log it nicely in one entry. Thanks ... View more

I'm using this query to graph how many web requests are being logged per second: index="bigip_ltm" (event=HTTP_REQUEST OR event=HTTP_RESPONSE) client_ip=1.2.3.4 | timechart count(event) by event But, in line with many questions here, the count() is graphed over a time interval derived from the search period. I've tried many permutations of span=1s, bucket commands etc, but I can't work out how to plot an average one second value over whatever period of time is represented on the graph. In this question http://splunk-base.splunk.com/answers/46978/average-field-value-per-second the "per_second" data is in the logs, but i want a per_second of the count of the number of logs, so one step further removed. ... View more