One of our Splunk environments receives data from a FIFO pipe. That is, syslog-ng takes incoming syslog data and sends it straight to Splunk via the pipe. It also takes incoming syslog data and writes it to the file system in the format of /logs/hostname/year/month/day/logfile
What I find odd is that if I sum up all the space used in the file system for the day, it consistency adds up to about 300-500 MB less than the amount of data that Splunk indexes for the day over the pipe.
If Splunk is receiving the same data over the pipe that is being written to the file system, how is it that it's indexing 300-500 MB more data than it's receiving?
I hope this question makes sense...
We don't have a huge license, so 300-500MB/day can make a difference. I just want to understand where it's going.
(On a side note, we're migrating to a new server, during which time we plan to lose the pipe and begin using monitor. Does that sound like a reasonable approach?)
Thanks!
