Size discrepency between file system and named pipe

Branden — Tue, 12 Apr 2011 07:41:05 GMT

One of our Splunk environments receives data from a FIFO pipe. That is, syslog-ng takes incoming syslog data and sends it straight to Splunk via the pipe. It also takes incoming syslog data and writes it to the file system in the format of /logs/hostname/year/month/day/logfile

What I find odd is that if I sum up all the space used in the file system for the day, it consistency adds up to about 300-500 MB less than the amount of data that Splunk indexes for the day over the pipe.

If Splunk is receiving the same data over the pipe that is being written to the file system, how is it that it's indexing 300-500 MB more data than it's receiving?

I hope this question makes sense...

We don't have a huge license, so 300-500MB/day can make a difference. I just want to understand where it's going.

(On a side note, we're migrating to a new server, during which time we plan to lose the pipe and begin using monitor. Does that sound like a reasonable approach?)

Thanks!

Re: Size discrepency between file system and named pipe

Stephen_Sorkin — Tue, 12 Apr 2011 19:48:15 GMT

How are you measuring the amount of data that Splunk indexed?

Re: Size discrepency between file system and named pipe

Branden — Thu, 14 Apr 2011 22:09:22 GMT

I am looking at the License Usage saved search that it comes with.

Re: Size discrepency between file system and named pipe

woodcock — Thu, 28 May 2015 20:22:26 GMT

FIFOs are deprecated and unsupported so you should discontinue using them.

topic Re: Size discrepency between file system and named pipe in Getting Data In

Size discrepency between file system and named pipe

Re: Size discrepency between file system and named pipe

Re: Size discrepency between file system and named pipe

Re: Size discrepency between file system and named pipe