Does anybody know what could be the cause why the tcpin_connections (group) is missing entirely from _internal index?
This search for checking the Forwarders (see below) worked just fine in the past. Currently our server and Forwarders run 6.5.0. Now it says that 'No results are found' (as there is no tcpin_connections group). tcpout_connections group is visible though.
Also netstat -an shows established connections for port 9997 on Linux (Splunk) server
index=_internal source=*metrics.log group=tcpin_connections | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | rename connectionType as connectType | eval connectType=case(fwdType=="uf","Universal Forwarder", fwdType=="lwf", "lightwt fwder",fwdType=="full", "heavy fwder", connectType=="cooked" or connectType=="cookedSSL","Splunk fwder", connectType=="raw" or connectType=="rawSSL","legacy fwder") | eval version=if(isnull(version),"pre 4.2",version) | rename version as Ver | fields connectType sourceIp sourceHost destPort kb tcp_eps tcp_Kprocessed tcp_KBps splunk_server Ver os arch | eval Indexer= splunk_server | eval Hour=relative_time(_time,"@h") | stats avg(tcp_KBps) sum(tcp_eps) sum(tcp_Kprocessed) sum(kb) by sourceHost sourceIp os arch connectType destPort Indexer Ver | sort Ver
Thanks in advance for Support!
That is strange, I tested your search in my 6.5 environment and I get results. If you just look at the ingested metrics logs, do you see that group? Rather, if you run:
index=_internal source=*metrics.log group=tcpin_connections
Does that yield results? Or:
index=_internal source=*metrics.log | stats count by group
Do you see the various groups? If the answer is no, if you search previous 30 days is there any change in the results? Your search and the ones I've listed above work in 6.5 on my instance so hopefully its just a straight forward issue.
10-03-2016 02:32:38.451 +0100 INFO StatusMgr - destPort=9997, eventType=connect_close, group=tcpin_connections, sourceHost=xx.xx.xx.xx, sourceIp=xx.xx.xx.xx, sourcePort=58796, statusee=TcpInputProcessor
still no tcpin_connections
I also hope it's a straight forward issue, except I have not been able to find it yet ...
During the upgrade to 6.5 there were some challenges as we got an error:
Exception: , Value: [Errno 13] Permission denied: '/opt/splunk/etc/system/local/indexes.conf'
We decided to delete that file, after which the upgrade 'process' went just fine.
Now that the tcpin_connections 'group' seems to be missing, the upgrade probably not went as it should (for 100 %)
The forwarders itself work fine as we have the incoming 'data'.
Is there an easy way to fix this or how can this be resolved?