topic Re: Search String for Connection hangs in Splunk Search

Search String for Connection hangs

jsproesser — Fri, 01 Nov 2019 19:56:32 GMT

I have been toying around with the task of identifying servers on our network with abnormal connection times . We have a set threshold for normal connection times in our environment. However, I want to get ahead and create an alert based off this report that I will transition to a dashboard.

The issue I am having is that I am relatively new to Splunk and still finding my groove in detailed searches. I thought about using eval combined with timestart and timeendpos.

|eval Connection=timestart-timeend....

My question to you all is , how far off am I ? Am I putting too much effort into skinning this cat? Does anyone have any recommendations?

Thanks in advance !

Re: Search String for Connection hangs

arjunpkishore5 — Fri, 01 Nov 2019 21:15:26 GMT

Could you post some sample data and the expected output please. It will be easier for us to analyze 🙂

Re: Search String for Connection hangs

niketn — Sat, 02 Nov 2019 01:10:11 GMT

@jsproesser how many indexers have you got?

Also what is your current SPL? Have you used Job Inspector to see which are the expensive commands in your search?

In order for the community to assist you better please provide more details including SPL and data sample. You should mock/anonymize any sensitive information before posting the same.

Re: Search String for Connection hangs

jsproesser — Mon, 04 Nov 2019 14:25:28 GMT

niketnilay ,

Below is my rather pedestrian SPL . I have not checked the job inspector. I'm just trying to see if I can further improve my search .

index=os sourcetype=linux_secure host=* success

|eval Difference=timeendpos-timestartpos
| stats count by host,Difference

| sort -Difference

Re: Search String for Connection hangs

jsproesser — Wed, 30 Sep 2020 02:52:23 GMT

My issue is that my result are only returning events where the timeendpos is greater than 16

Event

11/2/19
11:57:47.000 PM
Nov 2 23:57:47 (SERVER NAME) debug httpd[12456]: pam_bigip_authz: pam_sm_acct_mgmt returning status SUCCESS
eventtype = nix-all-logs eventtype = nix_security os unix host = (HOST) source = /var/log/secure sourcetype = linux_secure timeendpos = 16 timestartpos = 0

Re: Search String for Connection hangs

jsproesser — Mon, 04 Nov 2019 14:31:32 GMT

arjun,

I have posted an event example below.

My expected output is a list of servers which are experiencing slow connections that exceed our threshold within my organization.