Search String for Connection hangs

[jsproesser]

I have been toying around with the task of identifying servers on our network with abnormal connection times . We have a set threshold for normal connection times in our environment. However, I want to get ahead and create an alert based off this report that I will transition to a dashboard.

The issue I am having is that I am relatively new to Splunk and still finding my groove in detailed searches. I thought about using eval combined with timestart and timeendpos.

|eval Connection=timestart-timeend....

My question to you all is , how far off am I ? Am I putting too much effort into skinning this cat? Does anyone have any recommendations?

Thanks in advance !

[niketn]

@jsproesser how many indexers have you got?

Also what is your current SPL? Have you used Job Inspector to see which are the expensive commands in your search?

In order for the community to assist you better please provide more details including SPL and data sample. You should mock/anonymize any sensitive information before posting the same.

[jsproesser]

niketnilay ,

Below is my rather pedestrian SPL . I have not checked the job inspector. I'm just trying to see if I can further improve my search .

index=os sourcetype=linux_secure host=* success

|eval Difference=timeendpos-timestartpos
| stats count by host,Difference

| sort -Difference

[jsproesser]

My issue is that my result are only returning events where the timeendpos is greater than 16

Event

11/2/19
11:57:47.000 PM
Nov 2 23:57:47 (SERVER NAME) debug httpd[12456]: pam_bigip_authz: pam_sm_acct_mgmt returning status SUCCESS
eventtype = nix-all-logs eventtype = nix_security os unix host = (HOST) source = /var/log/secure sourcetype = linux_secure timeendpos = 16 timestartpos = 0

[arjunpkishore5]

Could you post some sample data and the expected output please. It will be easier for us to analyze 🙂

[jsproesser]

arjun,

I have posted an event example below.

My expected output is a list of servers which are experiencing slow connections that exceed our threshold within my organization.