After configuring new servers on the universal forwarder, why are sourcetypes and hosts missing from search?

[ashitvyas]

Hello All.

I am having existing setup for Splunk for the Aix servers and we just added some new servers to upgrade our application.

On our existing AIX Servers, the Splunk universal forwarder are installed and connected to Splunk app through config files and polling the logs files regularly.

The issue I am encountering right now is when I am trying to setup new servers with Splunk Universal Forwarder, the new hosts and source types are not showing up in my Splunk Search.

I have created a new index and configured that index as well on Splunk app.

I can see the new indexer is getting all the events when I check it from Splunk >Manager>Indexes

I have checked all the configuration file to ensure there is not typo and Splunk can read the log files on Splunk Universal Forwarder

Below what I see on my splunkd.log file as output when I start splunk on Aix server.

11-23-2016 12:08:45.685 -0700 INFO  LicenseMgr - Initing LicenseMgr
11-23-2016 12:08:45.698 -0700 INFO  ServerConfig - My GUID is C46EAB6B-7D93-4072-BDEA-D5D4DDED627A
11-23-2016 12:08:45.698 -0700 INFO  ServerConfig - My server name is "iv54318p".
11-23-2016 12:08:45.698 -0700 INFO  ServerConfig - My hostname is "iv54318p".
11-23-2016 12:08:45.702 -0700 INFO  ServerConfig - Setting HTTP server compression state=on
11-23-2016 12:08:45.702 -0700 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
11-23-2016 12:08:45.702 -0700 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
11-23-2016 12:08:45.702 -0700 INFO  LMConfig - serverName=iv54318p guid=C46EAB6B-7D93-4072-BDEA-D5D4DDED627A
11-23-2016 12:08:45.703 -0700 INFO  LMConfig - connection_timeout=30
11-23-2016 12:08:45.703 -0700 INFO  LMConfig - send_timeout=30
11-23-2016 12:08:45.703 -0700 INFO  LMConfig - receive_timeout=30
11-23-2016 12:08:45.703 -0700 INFO  LMConfig - squash_threshold=1000
11-23-2016 12:08:45.703 -0700 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=false
11-23-2016 12:08:45.703 -0700 INFO  LMStackMgr - closing stack mgr
11-23-2016 12:08:45.703 -0700 INFO  LMSlaveInfo - all slaves cleared
11-23-2016 12:08:45.703 -0700 INFO  LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
11-23-2016 12:08:45.703 -0700 INFO  LMStackMgr - added pool auto_generated_pool_free to stack free
11-23-2016 12:08:45.703 -0700 INFO  LMStackMgr - init completed [C46EAB6B-7D93-4072-BDEA-D5D4DDED627A,Forwarder,runContext_splunkd=false]
11-23-2016 12:08:45.703 -0700 INFO  LicenseMgr - StackMgr init complete...
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - this is not splunkd, will perform partial init
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=Auth state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=FwdData state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=RcvData state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=DistSearch state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=RcvSearch state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=ScheduledSearch state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=Alerting state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=DeployClient state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=DeployServer state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=SplunkWeb state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=SyslogOutputProcessor state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=SigningProcessor state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LMTracker - setting feature=LocalSearch state=ENABLED (fs=1)
11-23-2016 12:08:45.703 -0700 INFO  LicenseMgr - Tracker init complete...
11-23-2016 12:08:46.046 -0700 INFO  ServerConfig - My GUID is C46EAB6B-7D93-4072-BDEA-D5D4DDED627A
11-23-2016 12:08:46.046 -0700 INFO  ServerConfig - My server name is "iv54318p".
11-23-2016 12:08:46.046 -0700 INFO  ServerConfig - My hostname is "iv54318p".
11-23-2016 12:08:46.051 -0700 INFO  ServerConfig - Setting HTTP server compression state=on
11-23-2016 12:08:46.051 -0700 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
11-23-2016 12:08:46.051 -0700 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
11-23-2016 12:08:46.058 -0700 INFO  ulimit - Limit: virtual address space size: unlimited
11-23-2016 12:08:46.058 -0700 INFO  ulimit - Limit: data segment size: 134217728 bytes [hard maximum: unlimited]
11-23-2016 12:08:46.058 -0700 WARN  ulimit - Splunk may not work due to small data segment limit!
11-23-2016 12:08:46.058 -0700 INFO  ulimit - Limit: resident memory size: 33554432 bytes [hard maximum: unlimited]
11-23-2016 12:08:46.058 -0700 WARN  ulimit - Splunk may not work due to small resident memory size limit!
11-23-2016 12:08:46.058 -0700 INFO  ulimit - Limit: stack size: 33554432 bytes [hard maximum: 4294967296 bytes]
11-23-2016 12:08:46.058 -0700 INFO  ulimit - Limit: core file size: 1073741312 bytes [hard maximum: unlimited]
11-23-2016 12:08:46.058 -0700 INFO  ulimit - Limit: data file size: 1073741312 bytes
11-23-2016 12:08:46.058 -0700 WARN  ulimit - Splunk may not work due to low file size limit
11-23-2016 12:08:46.058 -0700 INFO  ulimit - Limit: open files: unlimited
11-23-2016 12:08:46.058 -0700 INFO  ulimit - Limit: cpu time: unlimited
11-23-2016 12:08:46.061 -0700 INFO  loader - Splunkd starting (build 143156).
11-23-2016 12:08:46.061 -0700 INFO  loader - System info: AIX, iv54318p, 1, 7, 00C948174C00.
11-23-2016 12:08:46.061 -0700 INFO  loader - Detected 24 (virtual) CPUs and 49152MB RAM
11-23-2016 12:08:46.061 -0700 INFO  loader - Arguments are: "splunkd" "-p" "8089" "start"
11-23-2016 12:08:46.061 -0700 INFO  loader - Getting configuration data from: /u01/splunk/splunkforwarder/etc/myinstall/splunkd.xml
11-23-2016 12:08:46.062 -0700 INFO  loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /u01/splunk/splunkforwarder/etc/modules
11-23-2016 12:08:46.062 -0700 INFO  loader - loading modules from /u01/splunk/splunkforwarder/etc/modules
11-23-2016 12:08:46.064 -0700 INFO  loader - Writing out composite configuration file: /u01/splunk/splunkforwarder/var/run/splunk/composite.xml
11-23-2016 12:08:46.069 -0700 INFO  BundlesSetup - Setup stats for /u01/splunk/splunkforwarder/etc: cpuTimeUsed=0.011632 sharedServicesGeneration=1 sharedServicesPopulation=1
11-23-2016 12:08:46.121 -0700 INFO  CMConfig - A splunktcp forwarder port is not configured in inputs.conf
11-23-2016 12:08:46.122 -0700 INFO  ClusteringMgr - initing clustering with: ht=60 rf=3 sf=2 ct=60 st=60 rt=60 rct=60 rst=60 rrt=60 rmst=600 rmrt=600 pe=1 im=0 is=0 mob=5 mor=5 pb=5 rep_port:
11-23-2016 12:08:46.149 -0700 INFO  ClusteringMgr - clustering disabled
11-23-2016 12:08:46.172 -0700 WARN  DeploymentClient - Property targetUri not found. DeploymentClient is disabled.
11-23-2016 12:08:46.173 -0700 INFO  LicenseMgr - Initing LicenseMgr
11-23-2016 12:08:46.173 -0700 INFO  LMConfig - serverName=iv54318p guid=C46EAB6B-7D93-4072-BDEA-D5D4DDED627A
11-23-2016 12:08:46.173 -0700 INFO  LMConfig - connection_timeout=30
11-23-2016 12:08:46.173 -0700 INFO  LMConfig - send_timeout=30
11-23-2016 12:08:46.173 -0700 INFO  LMConfig - receive_timeout=30
11-23-2016 12:08:46.173 -0700 INFO  LMConfig - squash_threshold=1000
11-23-2016 12:08:46.173 -0700 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=true
11-23-2016 12:08:46.173 -0700 INFO  LMStackMgr - closing stack mgr
11-23-2016 12:08:46.173 -0700 INFO  LMSlaveInfo - all slaves cleared
11-23-2016 12:08:46.173 -0700 INFO  LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
11-23-2016 12:08:46.173 -0700 INFO  LMStackMgr - added pool auto_generated_pool_free to stack free
11-23-2016 12:08:46.173 -0700 INFO  LMStackMgr - init completed [C46EAB6B-7D93-4072-BDEA-D5D4DDED627A,Forwarder,runContext_splunkd=true]
11-23-2016 12:08:46.173 -0700 INFO  LicenseMgr - StackMgr init complete...
11-23-2016 12:08:46.173 -0700 INFO  LMTracker - init'ing slaveId=C46EAB6B-7D93-4072-BDEA-D5D4DDED627A label=iv54318p [30,30,30,self,0,]
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - attempting to ping master=self from slave=C46EAB6B-7D93-4072-BDEA-D5D4DDED627A
11-23-2016 12:08:46.176 -0700 INFO  LMSlaveInfo - new slave='C46EAB6B-7D93-4072-BDEA-D5D4DDED627A' created
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=Alerting state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=AllowDuplicateKeys state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=Auth state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=DeployClient state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=DeployServer state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=DistSearch state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=FwdData state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=LocalSearch state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=RcvData state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=RcvSearch state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=ScheduledSearch state=DISABLED_DUE_TO_LICENSE (fs=2)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=SigningProcessor state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=SplunkWeb state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting feature=SyslogOutputProcessor state=ENABLED (fs=1)
11-23-2016 12:08:46.176 -0700 INFO  LMTracker - setting masterGuid='C46EAB6B-7D93-4072-BDEA-D5D4DDED627A'
11-23-2016 12:08:46.184 -0700 INFO  LMTracker - attempting to ping master=self from slave=C46EAB6B-7D93-4072-BDEA-D5D4DDED627A success
11-23-2016 12:08:46.184 -0700 INFO  LicenseMgr - Tracker init complete...
11-23-2016 12:08:46.185 -0700 WARN  DeploymentProcessor - License feature=DeployServer not enabled, cannot bring up Deployment Server
11-23-2016 12:08:46.187 -0700 INFO  IndexProcessor - running splunkd specific init
11-23-2016 12:08:46.187 -0700 WARN  DistributedPeerManager - feature=DistSearch not enabled for your license level
11-23-2016 12:08:46.187 -0700 INFO  loader - Initializing from configuration
11-23-2016 12:08:46.189 -0700 WARN  ThruputProcessor - unable to find out defaultDatabase from indexes.conf, defaulting to 'main'
11-23-2016 12:08:46.189 -0700 INFO  PipelineComponent - Pipeline fifo disabled in default-mode.conf file
11-23-2016 12:08:46.190 -0700 INFO  CMConfig - A splunktcp forwarder port is not configured in inputs.conf
11-23-2016 12:08:46.190 -0700 INFO  TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
11-23-2016 12:08:46.190 -0700 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
11-23-2016 12:08:46.190 -0700 INFO  UDPInputProcessor - Registering metrics callback for: udpin_connections
11-23-2016 12:08:46.218 -0700 WARN  ThruputProcessor - unable to find out defaultDatabase from indexes.conf, defaulting to 'main'
11-23-2016 12:08:46.219 -0700 INFO  TcpOutputProc - Initializing with fwdtype=lwf
11-23-2016 12:08:46.232 -0700 INFO  TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
11-23-2016 12:08:46.232 -0700 INFO  TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
11-23-2016 12:08:46.232 -0700 INFO  TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
11-23-2016 12:08:46.232 -0700 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to 10.62.64.53:9992
11-23-2016 12:08:46.232 -0700 INFO  TcpOutputProc - tcpout group default-autolb-group using Auto load balanced forwarding
11-23-2016 12:08:46.232 -0700 INFO  TcpOutputProc - Group default-autolb-group initialized with maxQueueSize=512000 in bytes.
11-23-2016 12:08:46.232 -0700 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to cgyut20841:9992
11-23-2016 12:08:46.232 -0700 INFO  TcpOutputProc - tcpout group group1 using Auto load balanced forwarding
11-23-2016 12:08:46.232 -0700 INFO  TcpOutputProc - Group group1 initialized with maxQueueSize=512000 in bytes.
11-23-2016 12:08:46.232 -0700 INFO  PipelineComponent - Pipeline merging disabled in default-mode.conf file
11-23-2016 12:08:46.232 -0700 INFO  PipelineComponent - Pipeline typing disabled in default-mode.conf file
11-23-2016 12:08:46.232 -0700 INFO  PipelineComponent - Launching the pipelines.
11-23-2016 12:08:46.232 -0700 INFO  loader - Server supporting SSL v2/v3
11-23-2016 12:08:46.232 -0700 INFO  loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
11-23-2016 12:08:46.253 -0700 INFO  TailingProcessor - TailWatcher initializing...
11-23-2016 12:08:46.253 -0700 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
11-23-2016 12:08:46.257 -0700 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
11-23-2016 12:08:46.258 -0700 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
11-23-2016 12:08:46.258 -0700 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
11-23-2016 12:08:46.258 -0700 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
11-23-2016 12:08:46.259 -0700 INFO  TailingProcessor - Parsing configuration stanza: monitor:///u01/logs/soaserver1logs/soa_server1.out.
11-23-2016 12:08:46.259 -0700 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
11-23-2016 12:08:46.261 -0700 INFO  WatchedFile - Will begin reading at offset=4129492 for file='/u01/logs/soaserver1logs/soa_server1.out'.
11-23-2016 12:08:46.532 -0700 INFO  TcpOutputProc - Connected to idx=10.62.64.53:9992
11-23-2016 12:09:16.193 -0700 INFO  CMConfig - A splunktcp forwarder port is not configured in inputs.conf
11-23-2016 12:09:16.508 -0700 INFO  TailingProcessor - Could not send data to output queue (parsingQueue), retrying...
11-23-2016 12:09:17.285 -0700 INFO  TcpOutputProc - Connected to idx=10.62.64.53:9992
11-23-2016 12:09:17.916 -0700 INFO  TailingProcessor -   ...continuing.
11-23-2016 12:10:53.156 -0700 INFO  BatchReader - Removed from queue file='/u01/splunk/splunkforwarder/var/log/splunk/metrics.log.1'.
11-23-2016 12:12:30.001 -0700 INFO  BatchReader - Removed from queue file='/u01/splunk/splunkforwarder/var/log/splunk/metrics.log.2'.
11-23-2016 12:13:46.353 -0700 INFO  ThruputProcessor - Current data throughput (259 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
11-23-2016 12:14:07.886 -0700 INFO  BatchReader - Removed from queue file='/u01/splunk/splunkforwarder/var/log/splunk/metrics.log.3'.
11-23-2016 12:15:44.731 -0700 INFO  BatchReader - Removed from queue file='/u01/splunk/splunkforwarder/var/log/splunk/metrics.log.4'.
11-23-2016 12:17:21.578 -0700 INFO  BatchReader - Removed from queue file='/u01/splunk/splunkforwarder/var/log/splunk/metrics.log.5'.

Please help.

Thanks.
Ash

[kalianov · Accepted solution]

Did you check "Indexes searched by default" in Access controls->Roles after you created new index?

[ashitvyas]

Hi ddrillic, I am not running any search query, it is just that newly configured Host and Source type doesn't show up in our Search page by default.

As kalianov mentioned, I should have added the newly created index as "Indexes searched by default.."

Thanks.

[ashitvyas]

HI Kalianov,

Unfortunately that was the only step I have missed through out my setup process.

Now I am able to see the new host and source type updated on Search page after adding the newly created index into the "Indexes searched by default"

Thanks
Ash

[ddrillic]

Right, what is the search query you are running?