Deployment Architecture

Why am I not receiving any Data from the Linux Server?

JarrettM
Path Finder

No data being received from Linux client.

Running Splunk Enterprise 7.0.3 on Windows Server 2012 R2. Receiving data across 7 indexes from 36 Windows Universal Forwarder clients and two syslog servers. Installed Splunk Universal Forwarder 7.0.3 on Redhat 7 server. Client phones home and makes good SSL connection:

5/10/18 8:53:54.613 AM 10..x.x.x - - [10/May/2018:08:53:54.613 -0400] "POST /services/broker/phonehome/connection_10..x.x.x_8089_ SPLUNK_CLIENT _ SPLUNK_CLIENT _784C7242-6306-403F-B877-11B04B59FFC7 HTTP/1.1" 200 832 - - - 0ms

05-10-2018 08:53:49.446 -0400 INFO  Metrics - group=tcpin_connections, 10.x.x.x:48460:9997, connectionType=cookedSSL, sourcePort=48460, sourceHost=10.x.x.x, sourceIp=10.x.x.x, destPort=9997, kb=72.01, _tcp_Bps=2378.52, _tcp_KBps=2.32, _tcp_avg_thruput=2.77, _tcp_Kprocessed=3107.70, _tcp_eps=3.55, _process_time_ms=0, evt_misc_kBps=0.06, evt_raw_kBps=1.03, evt_fields_kBps=1.13, evt_fn_kBps=0.84, evt_fv_kBps=0.29, evt_fn_str_kBps=0.06, evt_fn_meta_dyn_kBps=0.00, evt_fn_meta_predef_kBps=0.03, evt_fn_meta_str_kBps=0.68, evt_fv_num_kBps=0.03, evt_fv_str_kBps=0.23, evt_fv_predef_kBps=0.00, evt_fv_offlen_kBps=0.00, evt_fv_fp_kBps=0.00, build=fa31da744b51, version=7.0.3, os=Linux, arch=x86_64, hostname=SPLUNK_CLIENT, guid=784C7242-6306-403F-B877-11B04B59FFC7, fwdType=full, ssl=true, lastIndexer=y.y.y.y:9997, ack=false destPort =9997 host =SPLUNK_SERVER     support-server hostname =SPLUNK_CLIENT log_level =INFO source =E:\Splunk\var\log\splunk\metrics.log sourceHost =10.x.x.x sourcePort =   48460 tag =support-server

Deployment App shows in Client's Apps directory:
#var-logs
[monitor:///var/log/*.log]
disabled=false
index=linux
sourcetype=var-logs

Index and sourcetype are correct. 

These are the only lines the the Clients splunkd that indicate any sort of problem:

05-10-2018 09:04:03.532 -0400 WARN  LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 12323 - data_source="/home/USER_NAME/splunk/var/log/splunk/.splunkd.log.swp", data_host=" SPLUNK_CLIENT ", data_sourcetype="swp-too_small"

05-10-2018 09:04:03.533 -0400 WARN  TailReader - Insufficient permissions to read file='/home/USER_NAME/splunk/var/log/splunk/.splunkd.log.swx' (hint: No such file or directory ,                            UID: 474401269, GID: 474400513).

05-10-2018 09:04:06.536 -0400 WARN  FileClassifierManager - The file '/home/USER_NAME/splunk/var/log/splunk/.splunkd.log.swp' is invalid. Reason: binary

Any ideas?

Thanks

0 Karma

JarrettM
Path Finder

Succeeded in getting data by using the the following command on the client:

./splunk add monitor /var/log

however it still will not work by using a deployed app from the deployment server.

0 Karma

JarrettM
Path Finder

Never Mind!

Here was the problem:

Default instead of default

Doh!

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...