Why is my new RHEL 6 server is not indexing data?

[Clopresti]

Test Environment consists of: 1 UF 6.2.0 on RHEL 6 sending to Splunk 6.2.1 on RHEL 6 server.

On the UF "splunk list forward-server" shows the forwarder as active and "splunk list monitor" shows the log files to monitor. Splunk log confirms connection to 9997 on indexer.

On the indexer port 9997 is created to receive and netstat confirms connectivity from UF. SELinux is disabled. Searching "index=_internal source=*metrics.log tcpin_connections" shows _tcp_Kprocessed=62.46. But when searching the index there is nothing. Starting in debug mode doesn't show any errors so I'm not sure where else to check or what other permissions might need to be adjusted.

Does anyone have any suggestions or ideas?

[Clopresti]

The UF is running as root and is able to tail the log.

UF input.conf
[monitor:///var/log/splunk/ucs-c2xx-m2/*]
index = cisco_ucs
crcSalt =

UF output.conf
[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection)
forwardedindex.filter.disable = false

[tcpout:cisco_ucs]
server=10.200.60.16:9997

When i run index=* earliest=1 latest=now I get no results

When I run index=_internal the host show as the Indexer but in the message I see the sourceHost as my UF

Index exists on Indxer and there is nothing in main.

[jayannah]

when u type index=_internal, you should the hostname of uni. forwarder in host field. Looks like your forwarder communication with indexer not working. Did you enable 9997 port on indexer? Are there any firewall between indexer and forwarder?

Configure the receiving port on Indexer (inputs.conf for receiving data on port say 9997)
read details at http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Enableareceiver

[jayannah · Accepted solution]

1. When you search index=_internal, do you see the forwarder hostname in the host field?
2. Did you check on default index.. i.e index=main (if your default index is main).? If you do not have a specific index name or specified an incorrect index name (spell mistakes) in inputs.conf, the indexer will index such data on the default index if the index doesn't exists.
3. Did you create an index on Indexer and use the same index name in the forwarder inputs.conf?
4. Check if the input file has read access permission for the log file path for the user running splunkd

If the above steps don't solve your problem, please put your inputs.conf file of the forwarder and indexer here.

[Clopresti]

Thanks jayannah and MuS. It turns out that there was a setting in the /system/local/ of the forwarder which i just kept ignoring which basically conflicted with what i wanted my outputs.conf to do. The forwarder and indexers were doing what they were supposed to...

[MuS]

Check the log file prrmission, the user running splunk must be able to read the files. Also try searching all index over all time, like this

index=* earliest=1 latest=now