how to convert the raw data into index ?

[prasadjvv522]

HI All,

I am new to splunk, In my organization having 14 node cluster and they already configured splunk.

I found ,indexed files in splunk server (default db location)
and 14 nodes configuration files on /opt/splunkforwarder/etc/apps/search/local/inputs.conf.

My doubt is
how the data goes into splunk server from host(i think splunkforwarder),in splunkserver side the data can be visible like 143948074789891008.tsidx . how it is converted to tsidx ?

Please help me and correct me.

Thanks
Prasad

[yannK]

Hum this is the core of the product, it indexes events, and store them into buckets (the rawdata folder), and creates tsidx (timeseries index pointer) to make them searchable. (in case of replication, not all copies are searchable)
The format and process are of course proprietary, but you can find some details of the different pipelines involved.
http://docs.splunk.com/Documentation/Splunk/6.1.4/Deploy/Datapipeline

For the collection they are many way :
see http://docs.splunk.com/Documentation/Splunk/latest/Data/WhatSplunkcanmonitor

• local monitoring : inputs monitoring files and folders
• remote monitoring for windows : WMI inputs or AD monitor
• network inputs : udp or tcp ports are listening on the indexers, by example for syslog on pot 514.
• forwarding : the indexers have a listening port (splunktcp on port 9997 by example), and forwarders agents on remote servers are monitoring and sending data. see http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Aboutforwardingandreceivingdata see http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Distributedoverview

in case of doubt, run a btool on the inputs, or use the SOS app metrics dashboards to identify forwarders.
./splunk cmd btool inputs list --debug
http://docs.splunk.com/Documentation/Splunk/6.1.4/Troubleshooting/Usebtooltotroubleshootconfiguratio...