Getting Data In

Why is my new RHEL 6 server is not indexing data?

Clopresti
New Member

Test Environment consists of: 1 UF 6.2.0 on RHEL 6 sending to Splunk 6.2.1 on RHEL 6 server.

On the UF "splunk list forward-server" shows the forwarder as active and "splunk list monitor" shows the log files to monitor. Splunk log confirms connection to 9997 on indexer.

On the indexer port 9997 is created to receive and netstat confirms connectivity from UF. SELinux is disabled. Searching "index=_internal source=*metrics.log tcpin_connections" shows _tcp_Kprocessed=62.46. But when searching the index there is nothing. Starting in debug mode doesn't show any errors so I'm not sure where else to check or what other permissions might need to be adjusted.

Does anyone have any suggestions or ideas?

0 Karma
1 Solution

jayannah
Builder
  1. When you search index=_internal, do you see the forwarder hostname in the host field?
  2. Did you check on default index.. i.e index=main (if your default index is main).? If you do not have a specific index name or specified an incorrect index name (spell mistakes) in inputs.conf, the indexer will index such data on the default index if the index doesn't exists.
  3. Did you create an index on Indexer and use the same index name in the forwarder inputs.conf?
  4. Check if the input file has read access permission for the log file path for the user running splunkd

If the above steps don't solve your problem, please put your inputs.conf file of the forwarder and indexer here.

View solution in original post

0 Karma

Clopresti
New Member

The UF is running as root and is able to tail the log.

UF input.conf
[monitor:///var/log/splunk/ucs-c2xx-m2/*]
index = cisco_ucs
crcSalt =

UF output.conf
[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection)
forwardedindex.filter.disable = false

[tcpout:cisco_ucs]
server=10.200.60.16:9997

When i run index=* earliest=1 latest=now I get no results

When I run index=_internal the host show as the Indexer but in the message I see the sourceHost as my UF

Index exists on Indxer and there is nothing in main.

0 Karma

jayannah
Builder

when u type index=_internal, you should the hostname of uni. forwarder in host field. Looks like your forwarder communication with indexer not working. Did you enable 9997 port on indexer? Are there any firewall between indexer and forwarder?

Configure the receiving port on Indexer (inputs.conf for receiving data on port say 9997)
read details at http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Enableareceiver

0 Karma

jayannah
Builder
  1. When you search index=_internal, do you see the forwarder hostname in the host field?
  2. Did you check on default index.. i.e index=main (if your default index is main).? If you do not have a specific index name or specified an incorrect index name (spell mistakes) in inputs.conf, the indexer will index such data on the default index if the index doesn't exists.
  3. Did you create an index on Indexer and use the same index name in the forwarder inputs.conf?
  4. Check if the input file has read access permission for the log file path for the user running splunkd

If the above steps don't solve your problem, please put your inputs.conf file of the forwarder and indexer here.

0 Karma

Clopresti
New Member

Thanks jayannah and MuS. It turns out that there was a setting in the /system/local/ of the forwarder which i just kept ignoring which basically conflicted with what i wanted my outputs.conf to do. The forwarder and indexers were doing what they were supposed to...

0 Karma

MuS
SplunkTrust
SplunkTrust

Check the log file prrmission, the user running splunk must be able to read the files. Also try searching all index over all time, like this

index=* earliest=1 latest=now
0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...