Not Seeing Monitored Files from Forwarder in My Indexer

[scaldwell1]

I'm testing Splunk with the following configuration: Splunk 4.3 indexer and Splunk Universal Forwarder 4.3 on a separate machine. Both machines are running Ubuntu 10.04, and I installed both by running the script (not using .deb) so I wasn't prompted to do any setup of inputs.conf at install time.

I have been trying to add Apache log files to inputs.conf and see them in the Manager>Data Inputs>Files and Directories section of the indexer to no avail. Here is the etc/system/local/inputs.conf on the forwarder:

[default]

host = virtapache-2

[monitor:///var/log/apache2/*.log$]

index = default sourcetype =
access_common disabled = false

./splunk list forward-server:

Active forwards:
192.168.x.x:9997 Configured but inactive forwards:
None

And ./splunk list monitor:

Monitored Directores...

....

/var/log/apache2/*.log$

            /var/log/apache2/access.log
            /var/log/apache2/error.log
            /var/log/apache2/mod_jk.log
            /var/log/apache2/other_vhosts_access.log

So for all intents and purposes it looks like the forwarder is configured correctly and should be sending those logs to the indexer. The indexer was set up to receive as per instructions. Netstat on the indexer:

tcp 0 0 192.168.x.x:9997

192.168.x.x:40189 ESTABLISHED 18711/splunkd

tcp 0 0 0.0.0.0:5506

0.0.0.0:* LISTEN 18711/splunkd

tcp 0 0 0.0.0.0:9997

0.0.0.0:* LISTEN 18711/splunkd

tcp 0 0 0.0.0.0:8089

0.0.0.0:* LISTEN 18711/splunkd

Netstat on the forwarder:

tcp 0 0 0.0.0.0:8089

0.0.0.0:* LISTEN 13611/splunkd

tcp 0 0 192.168.x.x:39265
192.168.x.x:9997 ESTABLISHED 13611/splunkd

tcp 0 0 192.168.x.x:40189
192.168.x.x:9997 ESTABLISHED 13611/splunkd

Lastly, the only thing I see that could be fishy in splunkd.log is the following on the forwarder:

01-18-2012 10:28:07.892 -0500 WARN
TcpOutputFd - Connect to
192.168.x.x:9997 failed. Connection refused

01-18-2012 10:28:07.892 -0500 ERROR
TcpOutputFd - Connection to
host=192.168.x.x:9997 failed

01-18-2012 10:28:07.892 -0500 INFO
TcpOutputProc - Detected connection to
192.168.x.x:9997 closed

01-18-2012 10:56:30.611 -0500 INFO
BatchReader - Could not send data to
output queue (parsingQueue),
retrying...

01-18-2012 10:56:30.941 -0500 INFO
TcpOutputProc - Connected to
idx=192.168.x.x:9997

I am quite confused as to what I am missing. It seems like a fairly simple matter to set up an input, set up outputs.conf, restart the service, et voila! However, I don't see these Apache logs anywhere in my Data Inputs using the web GUI on the indexer. Is there anything obvious from the configs above that could be causing this problem?

[lguinn2]

You will not see these inputs under Manager » Data inputs » Files & directories. From the perspective of the indexer, these files are arriving via the forwarder. Manager » Data inputs » Files & directories is only used for identifying inputs that the indexer will collect directly.

The proper way to determine if this data is arriving is to go to the search and run this search

host = virtapache-2 sourcetype=access_common

This will show you the data that has been forwarded. (BTW, I assume that the linebreaks are messed up in your post.)
It does look like your forwarder and indexer may have had some troubles communicating. However, iI am not sure, based on this snippet, if this was a transient problem or something more serious. Remember that the forwarder can deal properly with dropped packets and dropped connection, resending and reconnecting as necessary and caching the local data as needed.

To see what the data flow is, between your forwarder and indexer, run this search (cut and paste!)

index="_internal" source="*metrics.log" group=tcpin_connections | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType=="uf","universal forwarder", fwdType=="lwf", "lightweight forwarder",fwdType=="full", "heavy forwarder", connectionType=="cooked" or connectionType=="cookedSSL","Splunk forwarder", connectionType=="raw" or connectionType=="rawSSL","legacy forwarder")| eval build=if(isnull(build),"n/a",build) | eval version=if(isnull(version),"pre 4.2",version) | eval guid=if(isnull(guid),sourceHost,guid) | eval os=if(isnull(os),"n/a",os)| eval arch=if(isnull(arch),"n/a",arch) | fields connectionType sourceIp sourceHost sourcePort destPort kb tcp_eps tcp_Kprocessed tcp_KBps splunk_server build version os arch guid | stats avg(tcp_KBps) sum(tcp_eps) sum(tcp_Kprocessed) sum(kb) by connectionType sourceIp sourceHost sourcePort destPort splunk_server version

BTW, I didn't write this, I "borrowed" it from the Splunk Deployment Monitor app and modified it.