how is data stored in Splunk?

[MaryCampbell]

I hope you don't mind but I have a couple of more questions which have come back to me:

Volume, one of our customers has 1 million CDRs per day would there be any issue with this in Splunk?

We zip the older files with bzip, can these be read in the same way as non zipped files

How is data stored in Splunk is in a database?

In a distributed setting how would remote files be transmitted SFTP? Are these compressed first.

Thanking you in advance Mary

[Ayn]

No, volume is not an issue. Splunk has been deployed in environments where it's handling several terabytes a day. It all comes down to sizing your Splunk deployment correctly. More info here: http://docs.splunk.com/Documentation/Splunk/5.0.1/installation/capacityplanningforalargersplunkdeplo...

bzip2-compressed files can be read in the same way as non-compressed ones.

Data is stored in a Splunk-specific kind of database known as an "index". More information available (among other places) here: http://docs.splunk.com/Documentation/Splunk/5.0.1/Indexer/HowSplunkstoresindexes

In a distributed environment you usually install Splunk forwarders (agents) on the systems data is to be read from. Communication between the forwarders and the indexer is done using a proprietary protocol that can be encrypted and compressed. More information available here: http://docs.splunk.com/Documentation/Splunk/5.0.1/Deploy/Aboutforwardingandreceivingdata