Deployment Architecture

No tcpin_connections (group) for _internal index

edwinmae
Path Finder

Hi,

Does anybody know what could be the cause why the tcpin_connections (group) is missing entirely from _internal index?

This search for checking the Forwarders (see below) worked just fine in the past. Currently our server and Forwarders run 6.5.0. Now it says that 'No results are found' (as there is no tcpin_connections group). tcpout_connections group is visible though.

Also netstat -an shows established connections for port 9997 on Linux (Splunk) server

index=_internal source=*metrics.log group=tcpin_connections   | eval sourceHost=if(isnull(hostname), sourceHost,hostname)   | rename connectionType as connectType  | eval connectType=case(fwdType=="uf","Universal Forwarder", fwdType=="lwf", "lightwt fwder",fwdType=="full", "heavy fwder", connectType=="cooked" or connectType=="cookedSSL","Splunk fwder", connectType=="raw" or connectType=="rawSSL","legacy fwder")  | eval version=if(isnull(version),"pre 4.2",version)  | rename version as Ver   | fields connectType sourceIp sourceHost destPort kb tcp_eps tcp_Kprocessed tcp_KBps splunk_server Ver os arch  | eval Indexer= splunk_server  | eval Hour=relative_time(_time,"@h")  | stats avg(tcp_KBps) sum(tcp_eps) sum(tcp_Kprocessed) sum(kb) by sourceHost sourceIp os arch connectType destPort Indexer Ver  | sort Ver

--

Thanks in advance for Support!

jcrabb_splunk
Splunk Employee
Splunk Employee

That is strange, I tested your search in my 6.5 environment and I get results. If you just look at the ingested metrics logs, do you see that group? Rather, if you run:

index=_internal source=*metrics.log group=tcpin_connections

Does that yield results? Or:

index=_internal source=*metrics.log | stats count by group

Do you see the various groups? If the answer is no, if you search previous 30 days is there any change in the results? Your search and the ones I've listed above work in 6.5 on my instance so hopefully its just a straight forward issue.

Jacob
Sr. Technical Support Engineer

edwinmae
Path Finder
  • index=_internal source=metrics.log group=tcpin_connections (for let's say last 24 hours) does not provide any results
  • When searching for e.g Last 30 days I do get the (normal) results -- last event was before the upgrade --

10-03-2016 02:32:38.451 +0100 INFO StatusMgr - destPort=9997, eventType=connect_close, group=tcpin_connections, sourceHost=xx.xx.xx.xx, sourceIp=xx.xx.xx.xx, sourcePort=58796, statusee=TcpInputProcessor

  • index=_internal source=*metrics.log | stats count by group (for let's say last 24 hours)

alt text

still no tcpin_connections

I also hope it's a straight forward issue, except I have not been able to find it yet ...

0 Karma

lguinn2
Legend

Hmm, did you accidentally change some settings that control either the log channels or the indexing of internal logs (.../var/log/splunk)?

0 Karma

edwinmae
Path Finder

During the upgrade to 6.5 there were some challenges as we got an error:

Exception: , Value: [Errno 13] Permission denied: '/opt/splunk/etc/system/local/indexes.conf'

We decided to delete that file, after which the upgrade 'process' went just fine.
Now that the tcpin_connections 'group' seems to be missing, the upgrade probably not went as it should (for 100 %)
The forwarders itself work fine as we have the incoming 'data'.

Is there an easy way to fix this or how can this be resolved?

0 Karma

edwinmae
Path Finder

We removed props.conf and transform.conf (from local) after which the functionality was restored

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...