You could use the batch input type in Splunk, that will delete the file after reading it. However, you will have to make sure that the files are written slowly elsewhere, and atomically moved into the monitored path after they are finished. Else Splunk could start reading and deleting a partial file. See http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/inputsconf for more info on batch:// input configuration. ... View more

When the forwarder makes its TCP connection with one indexer, it will often send multiple packets over the connection before switching to another indexer. The forwarder could easily split an event between packets - since an event can consist of multiple lines. In fact, the forwarder does not really know where the event boundaries lie. So the forwarder continues to send to the same indexer until it hits EOF on the data stream that it is monitoring. Then it switches. There are variations, This could change and it isn't formally documented. But this is the short explanation of "why." So there is a consequence. Again, I advise against this, and so will every other Splunk consultant. Also, I feel your DNS pain... ... View more

It does, but I've looked at the data on a useAck enabled connection and I don't see any actual data coming back, just an empty ACK from the indexer. If there's no actual protocol level conversation, I don't see what useACK is really doing, and if it's only on the ACK packet, what the reason for it not working with raw data would be. ... View more

Yes, that's basically exactly what I'm looking to implement. I'd be interested to hear more about how you're actually controlling the daemon though. Currently i've a simple script in our test environment that does this... #!/bin/sh APP=$1 CONF=$SPLUNK_HOME/etc/apps/$APP/local/syslog-ng.conf SYSLOG_PID=`ps -ef | grep syslog-ng | grep $APP | awk '{print $2}'` if [ "$SYSLOG_PID" ] then echo syslog-ng for $CONF already running - $SYSLOG_PID else echo syslog-ng not running under $CONF, starting now /usr/local/sbin/syslog-ng -f $CONF fi So attempts to force a start if it's not running, and logs the status either way. This doesn't seem great, but as a service that should really end up being very boring and consistent, I'm not too concerned. I'm also looking at running this for each logica group of source types, merging it in with the same apps that reads the files, so potentially a half dozen different syslog daemons one day, but not many. What are you referring to with syslogcontrol? Thanks. ... View more

current docs say that it can be a any fixed string or "SOURCE", so as long as symlinks are followed, then we can just symlink the dir at a higher level and remain reasonably "Clean" at least at the Splunk level i'd have thought. * If set, string is added to the CRC. * If set to the literal string "SOURCE" (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC. When crcSalt is invoked, it is usually set to <SOURCE>. ... View more

That all sounds reasonable as long as it reliable here. These are daily batch files, no manageable delay is really a problem, and it's done overnight when things are relatively sleepy. Where would the files be decompressed to by default? Ultimately this is a temp hack before we get a real time stream of equivalent data, so looks good all round to me. Thanks ... View more

All Splunk user authentication is handled on the search heads. If you grant a third party controlled search head access to your indexers, they will have full and complete access to your data, and it will be totally up to them to control access to indexes. Two ways that you might approach this problem. The first is if it's a friendly third party, you could provide them your set of roles, and then ask that they do their authentication in alignment with that. In some situations, I've heard of the central org taking ownership of the remote search head, and just managing it. The second way is that you could segment your indexers into two groups -- one that has sensitive data and one that doesn't. This adds more complexity to the environment because you have to route either hosts to different sets of indexers (e.g., web servers to one, internal servers to another) or route based on different data sources (e.g., PCI data to a couple of dedicated indexers, all other data to the normal sets). More complexity, but by controlling where the data is sent you get to control what indexers the third party search head is allowed to hit, and thus what data they can see. ... View more

Messing with IP's won't help because often you'd have one public IP on the NAT with multiple public ports that are translated to multiple private IP's on port 8089. Something like this: - pub_ip:pub_port_1 -> priv_ip_1:8089 - pub_ip:pub_port_2 -> priv_ip_2:8089 What you need is to rewrite destination IP and port on your search heads to point to PUBLIC IP and port. Here is how: #!/bin/bash PATH=/bin:/usr/bin:/user/local/bin:/sbin:/usr/sbin:/usr/local/sbin REMOTE_PUB_IP=<your_public_ip_here> REMOTE_PUB_PORT=(8089 8090 8091) REMOTE_PRV_IP=(your_private_ips_here, separated by space) REMOTE_PRV_PORT=(8089 8089 8089) run_cmd () { if [[ -z "$DEBUG" ]]; then $* else echo $* fi } # Enable IP forwarding sysctl net.ipv4.ip_forward=1 # Flush all NAT rules iptables -t nat -F (( max_index = ${#REMOTE_PUB_PORT[*]} - 1 )) for i in $( seq 0 $max_index ); do run_cmd "iptables -t nat -A PREROUTING -p tcp --dest ${REMOTE_PRV_IP[i]} --dport ${REMOTE_PRV_PORT[i]} -j DNAT --to ${REMOTE_PUB_IP}:${REMOTE_PUB_PORT[i]}" run_cmd "iptables -t nat -A OUTPUT -p tcp --dest ${REMOTE_PRV_IP[i]} --dport ${REMOTE_PRV_PORT[i]} -j DNAT --to ${REMOTE_PUB_IP}:${REMOTE_PUB_PORT[i]}" done Anybody knows how to do that using firewall-cmd? ... View more

Bluecoat proxysg can create tcp tunnel (it will not go above layer 4) between the 2 networks ... View more

What Splunk component is hosting your TCP data input and what constitutes "health at a deeper level"? I am pretty sure that if you can establish a TCP connection to a port assigned to a TCP input on an indexer, you could take that as a very good sign that this thing is up and running and will process data sent to it. I am not aware of any health probe message you could send which would respond with a predefined "I'm here, I'm good" message, nor am I aware that that was ever reason for concern. You cannot do searches against a splunk port setup to listen for a TCP (or UDP) input stream, but I maybe misunderstanding what you are saying. Is your "round the back" idea to send some eyecatcher message to the port, then run a search to see whether that message was indexed? If so, I would keep it simple.... 😉 ... View more

Well as above, this is a script sending data to a raw tcp input. ... View more

Actually, is there any perspective abuot doing this as an rtsearch instead of a schedule? It looks to me that to main problem is that as a rtsearch there's no built in mechanism to run it constantly. For reference I've a vague search that looks like this... 'http_id=* | transaction http_id maxspan=2m | eval timestamp=strftime(_time, "%d/%m/%Y %H:%M:%S %z") | strcat client_ip " - - [" timestamp "] " http_method " " http_path " HTTP/" http_version " " http_status _raw| fields _raw | collect index=access_logs' ... View more

Yeah, that's pretty useful. I thought there needed to be another aggregation stage but couldn't work out what it might be. I've now got this index="bigip_ltm" (event=HTTP_REQUEST OR event=HTTP_RESPONSE) client_ip=1.2.3.4 | timechart count by event | timechart per_second(HTTP_REQUEST) per_second(HTTP_RESPONSE) I don't like having to field values end up as static field names, but I presume that's pretty much unavoidable? There's no way to graph all the "event" values implicitly? Either way, that's got me what I asked for! thanks! ... View more

acid, thanks, I looked for this command for months !!! ... View more