Is there a good way manage the process of using syslog-ng to write from tcp to disk and then from disk into Splunk?

[acidkewpie]

I'm migrating from using a tcp input to using syslog-ng to write from tcp to disk and then from disk into Splunk. This brings up the angle on how we manage the syslog-ng process being started etc. Is there a good way to have Splunk do this? It seems appropriate to actually deploy the new binaries, and it's config(s) in an app, but I still need a responsible way to have it start that can remain within the remit of the Splunk admins, not the wider world that look after the kit the UF is running on.

I guess I could use a scripted input to run a wrapper script, which will start the syslog-ng process as a daemon if it's not running, with the output of the script being a suitable log message about the processes status (and maybe other stats whilst we're there for good luck) but I'm not sure if this is an ugly hack or an nice idea. I'm certainly aware there are more conventional ways to do this, but it's important that this be kept within the limited Splunk scope due to the way the company works.

[mwk1000 · Accepted solution]

I did just that, I created a Splunk application with syslogcontrol (start/stop/status) compiled the binaries , installed into an app structure with all needed libraries, and updated the syslog-ng configuration files to use environment variables for the "base". It will run from the app deployment structure self contained.

Now we just deploy our "app_syslogcollector" and we are up and running as the Splunk account. The log file rotation is also self controlled from within the app. It works very well all under our control , self contained deployed to over 20 hosts...

The syslog-ng configs create directories based on the index/sourcetypes we want /apps/syslogs///syslog.log

A separate simple inputs app monitors the paths and inputs the files. ( put the asterisk's back in where they belong )
[ monitor:///apps/syslogs/cisco-rtr/hostname/syslog.log]
host_segment=4
index=network
sourcetype=os:cisco:syslog

Note we listen above 1024 to avoid root requirements and conflicts with existing syslog, we are a RECEIVER not a local host syslog.
The disk acts as a buffer for "incidents/outages" Splunk will pick up where it left off up to your log rotation settings. Accepts TCP/UDP will forward again externally if you need to ...

[mwk1000]

Multi-threaded so it's a beast and will sustain very high volumes. ( The driving reason it was created )

[acidkewpie]

Yes, that's basically exactly what I'm looking to implement. I'd be interested to hear more about how you're actually controlling the daemon though.

Currently i've a simple script in our test environment that does this...

#!/bin/sh

APP=$1
CONF=$SPLUNK_HOME/etc/apps/$APP/local/syslog-ng.conf
SYSLOG_PID=`ps -ef | grep syslog-ng | grep $APP | awk '{print $2}'`
if [ "$SYSLOG_PID" ]
then
  echo syslog-ng for $CONF already running - $SYSLOG_PID
else
  echo syslog-ng not running under $CONF, starting now
  /usr/local/sbin/syslog-ng -f $CONF
fi

So attempts to force a start if it's not running, and logs the status either way. This doesn't seem great, but as a service that should really end up being very boring and consistent, I'm not too concerned.

I'm also looking at running this for each logica group of source types, merging it in with the same apps that reads the files, so potentially a half dozen different syslog daemons one day, but not many.

What are you referring to with syslogcontrol? Thanks.