Hello @gibu_george_medlife,
First, The monitor/batch stanzas for these logs can reside default or local directories . $SPLUNK_HOME/etc/system/default|local OR $SPLUNK_HOME/etc/app/..local .
When you say same machine, how did you come to that conclusion? by looking at the host name? example:
index=main source="/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log" gives the host name same as the hunk host? If yes, one possible reason i could think of is someone copied the stanzas as is. Example....if i have one server called Server1 with inputs and outputs and if i distribute the configs as is to Server2, unless someone goes in to inputs and changes the host name to $decideonstartup pr manually enter Server2, Logs will be displayed with host=Server1. If this is the case, try to find out IPs of all sourcenames to determine where/which server is sending these logs. Use this search on your hunk server to get the IPs
index="_internal" source="*metrics.log" group=tcpin_connections |
eval sourceHost=if(isnull(hostname), sourceHost,hostname) |
eval connectionType=case(fwdType=="uf","Universal Forwarder", fwdType=="lwf", "Light Weight Forwarder",fwdType=="full", "Splunk Indexer", connectionType=="cooked" or connectionType=="cookedSSL","Splunk Forwarder", connectionType=="raw" or connectionType=="rawSSL","Legacy Forwarder") |
eval build=if(isnull(build),"n/a",build) |
eval version=if(isnull(version),"pre 4.2",version) |
eval guid=if(isnull(guid),sourceHost,guid) |
eval os=if(isnull(os),"n/a",os)|
eval arch=if(isnull(arch),"n/a",arch) |
eval my_splunk_server = splunk_server |
fields connectionType sourceIp sourceHost sourcePort destPort kb tcp_eps tcp_Kprocessed tcp_KBps my_splunk_server build version os arch |
eval lastReceived = if(kb>0, _time, null) |
stats first(sourceIp) as sourceIp first(connectionType) as connectionType first(sourcePort) as sourcePort first(build) as build first(version) as version first(os) as os first(arch) as arch max(_time) as lastConnected max(lastReceived) as lastReceived sum(kb) as kb avg(tcp_eps) as avg_eps by sourceHost |
stats first(sourceIp) as sourceIp first(connectionType) as connectionType first(sourcePort) as sourcePort first(build) as build first(version) as version first(os) as os first(arch) as arch max(lastConnected) as lastConnected max(lastReceived) as lastReceived first(kb) as KB first(avg_eps) as eps by sourceHost |
eval status = if(isnull(KB) or lastConnected<(info_max_time-900),"missing",if(lastConnected>(lastReceived+300) or KB==0,"quiet","active")) | sort sourceHost
Hope this helps!
Thanks,
Raghav
... View more