No data being received from Linux client.
Running Splunk Enterprise 7.0.3 on Windows Server 2012 R2. Receiving data across 7 indexes from 36 Windows Universal Forwarder clients and two syslog servers. Installed Splunk Universal Forwarder 7.0.3 on Redhat 7 server. Client phones home and makes good SSL connection:
5/10/18 8:53:54.613 AM 10..x.x.x - - [10/May/2018:08:53:54.613 -0400] "POST /services/broker/phonehome/connection_10..x.x.x_8089_ SPLUNK_CLIENT _ SPLUNK_CLIENT _784C7242-6306-403F-B877-11B04B59FFC7 HTTP/1.1" 200 832 - - - 0ms
05-10-2018 08:53:49.446 -0400 INFO Metrics - group=tcpin_connections, 10.x.x.x:48460:9997, connectionType=cookedSSL, sourcePort=48460, sourceHost=10.x.x.x, sourceIp=10.x.x.x, destPort=9997, kb=72.01, _tcp_Bps=2378.52, _tcp_KBps=2.32, _tcp_avg_thruput=2.77, _tcp_Kprocessed=3107.70, _tcp_eps=3.55, _process_time_ms=0, evt_misc_kBps=0.06, evt_raw_kBps=1.03, evt_fields_kBps=1.13, evt_fn_kBps=0.84, evt_fv_kBps=0.29, evt_fn_str_kBps=0.06, evt_fn_meta_dyn_kBps=0.00, evt_fn_meta_predef_kBps=0.03, evt_fn_meta_str_kBps=0.68, evt_fv_num_kBps=0.03, evt_fv_str_kBps=0.23, evt_fv_predef_kBps=0.00, evt_fv_offlen_kBps=0.00, evt_fv_fp_kBps=0.00, build=fa31da744b51, version=7.0.3, os=Linux, arch=x86_64, hostname=SPLUNK_CLIENT, guid=784C7242-6306-403F-B877-11B04B59FFC7, fwdType=full, ssl=true, lastIndexer=y.y.y.y:9997, ack=false destPort =9997 host =SPLUNK_SERVER support-server hostname =SPLUNK_CLIENT log_level =INFO source =E:\Splunk\var\log\splunk\metrics.log sourceHost =10.x.x.x sourcePort = 48460 tag =support-server
Deployment App shows in Client's Apps directory:
#var-logs
[monitor:///var/log/*.log]
disabled=false
index=linux
sourcetype=var-logs
Index and sourcetype are correct.
These are the only lines the the Clients splunkd that indicate any sort of problem:
05-10-2018 09:04:03.532 -0400 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 12323 - data_source="/home/USER_NAME/splunk/var/log/splunk/.splunkd.log.swp", data_host=" SPLUNK_CLIENT ", data_sourcetype="swp-too_small"
05-10-2018 09:04:03.533 -0400 WARN TailReader - Insufficient permissions to read file='/home/USER_NAME/splunk/var/log/splunk/.splunkd.log.swx' (hint: No such file or directory , UID: 474401269, GID: 474400513).
05-10-2018 09:04:06.536 -0400 WARN FileClassifierManager - The file '/home/USER_NAME/splunk/var/log/splunk/.splunkd.log.swp' is invalid. Reason: binary
Any ideas?
Thanks
Succeeded in getting data by using the the following command on the client:
./splunk add monitor /var/log
however it still will not work by using a deployed app from the deployment server.
Never Mind!
Here was the problem:
Default instead of default
Doh!