I am using Splunk universal forwarder to receive data in Splunk enterprise but data is not shown in the search result.
Splunk Enterprise and universal forwarder are in the same server.
Created index and assigned to the admin role. Set the port for listening in the receiver.
Below are the configuration details:
/opt/splunkforwarder/etc/system/local/outputs.conf [tcpout] defaultGroup=sp_index [tcpout:sp_index] server=10.100.103.209:9997 [tcpout-server://10.100.103.209:9997] ./splunk add forward-server 10.100.103.209:9997 ./splunk add monitor "/var/www/spdev_mythily_data/test_data/*" -index sp_index -sourcetype _json -host 10.100.103.209
If your forwarder is your Indexer, then there is no need to install the UF (in fact, as you are realizing, it may not work correctly out of the box); just install Splunk because the full version has everything that the UF has and more. If you must do it this way (for example, for learning or testing), then start completely over and when you are issuing commands to the forwarder, use
/opt/splunkforwarder/... and only use
/opt/splunk/... when sending commands to the Indexer/Search-Head. So at a minimum, you should change to this:
/opt/splunkforwarder/bin/splunk add forward-server 10.100.103.209:9997 /opt/splunkrorwader/bin/splunk add monitor "/var/www/spdev_mythily_data/test_data/*" -index sp_index -sourcetype
So, what troubleshooting have you done already?
Have you restarted both instances after performing these configurations?
Have you confirmed your Enterprise instance is listening on TCP 9997 (e.g. using netstat)? Have you checked that instance's splunkd.log for errors/warnings?
Have you checked your UF for errors/warnings in the splunkd.log? Both about the input as well as the output?
Have you ran a search over All Time, to ensure you're not missing the data because it has been misplaced on the timeline due to incorrect timestamp extraction?
What is the purpose of having UF and Enterprise on the same machine? Why not configure the input in the enterprise instance itself (to prevent confusion and perhaps conflicting ports).
Don't run Splunk Enterprise and Universal Forwarder on the same server. It's not necessary. Splunk Enterprise can do everything a Universal Forwarder can do.