topic Re: Ellobrate this event in Monitoring Splunk

How do I Ellobrate this event?

Mohanveera1 — Wed, 30 Mar 2022 17:27:55 GMT

Hi splunk experts,

Can anyone elaborate this below event and tell me why this event is getting triggered? the user name in this event has left the organization and we removed his access and transferred the knowledge objects to other person also but we are getting his name in the below event. and please help me how to avoid this type type of alerts also.

127.0.0.1 - **User name*** [30/Mar/2022:09:29:54.891 +0000] "POST /servicesNS/nobody/search/saved/searches/Single%20User%20Failed%20Attempt/notify?trigger.condition_state=1 HTTP/1.1" 200 1933 "-" "Splunk/8.1.0 (Linux 4.15.0-1023-azure; arch=x86_64)" - 2ms

this event is getting displayed when we search by using the query: index=_internal sourcetype= splunkd_access user=*.

thanks in advance.

Re: Ellobrate this event

ITWhisperer — Wed, 30 Mar 2022 10:02:40 GMT

It looks like something is running on local host. How often does it happen? Could it be a cronjob or something like that?

Re: Ellobrate this event

Mohanveera1 — Wed, 30 Mar 2022 10:12:29 GMT

Hi @ITWhisperer

thank you for your response.

This event is generating everyday and its not only with one user, its with multiple users who left the organization. The cronjob are like knowledge objects, but all the knowledge objects created by the user is assigned to the new user. so can you please suggest me how can we check any new cronjobs that are available. and how can we check in local host as well????

thanks in advance..

Re: Ellobrate this event

ITWhisperer — Wed, 30 Mar 2022 10:17:25 GMT

Every day? At the same time every day or at random times?

What do you mean by "cronjob are like knowledge objects"?

Do you have shell access to your splunk hosts?

What other processes are running on your splunk hosts?

What other users are logged on to your splunk hosts?

Re: Ellobrate this event

Mohanveera1 — Wed, 30 Mar 2022 11:05:06 GMT

hi @ITWhisperer

please find the answers according to your queries... and please revert me incase if any of the info is required.

every day? At the same time every day or at random times? -- these events are triggering for every minute

What do you mean by "cronjob are like knowledge objects"? -- cronjob in splunk is meant to be knowledge objects such as dashboards, alerts, reports which are scheduled to run on specific time. (As of my knowledge) please share what do you mean by cronjob in your opinion.

Do you have shell access to your splunk hosts? -- yes i do have access to shell in splunk host

What other processes are running on your splunk hosts? -- there were so many processes are running in splunk host, as i checked by using command ps aux. Is there any specific processes that i have to look at?

What other users are logged on to your splunk hosts? -- Is this about the user in the splunk or the users that can access the splunk host.

Thanks in advance.....

Re: Ellobrate this event

ITWhisperer — Wed, 30 Mar 2022 11:21:10 GMT

How is the saved search "Single User Failed Attempt" set up? Who owns it? Which user does it run as?

Re: Ellobrate this event

Mohanveera1 — Wed, 30 Mar 2022 11:41:05 GMT

hi @ITWhisperer

the "Single User Failed Attempt" is setup as an alert type. it was previously owned by the member that left the organisation (mentioned in the above thread ***user name***), after he left i have reassigned the alert to myself. this alert does not run by any user as it was kept as real time alert.

Thanks in advance....

Re: Ellobrate this event

VatsalJagani — Wed, 30 Mar 2022 17:09:08 GMT

Grep for this in the savesearches.conf from the Splunk instance in the backend if you can.

cd /opt/splunk/etc/ grep -rinl "Single User Failed Attempt"

See if you can see a file that contains it. (Specifically savedsearches.conf)
That should give an answer in most cases.

Re: Ellobrate this event

ITWhisperer — Wed, 30 Mar 2022 17:23:10 GMT

How did you reassign the alert?

Re: Ellobrate this event

Mohanveera1 — Thu, 31 Mar 2022 04:02:16 GMT

@ITWhisperer

i clicked on the All Configurations in the settings option to get all Knowledge objects and filtered them for the previous owner. Then i reassigned all the alerts and reports from him to the new user.

Re: Ellobrate this event

Mohanveera1 — Thu, 31 Mar 2022 04:46:05 GMT

hi @VatsalJagani

thank you for your response.

in the splunk instance i have go till opt directory, but in opt directory i cant find splunk. it only contains the backup scripts. is there any other way that i can look into.....

Re: Ellobrate this event

VatsalJagani — Thu, 31 Mar 2022 05:26:55 GMT

By /opt/splunk I meant your Splunk installation directory. This is generally the case but it seems in your case Splunk is installed in a different location.

* I hope you are in the backend of search head.

* You can find the Splunk installation directory by running the command, find / -name "splunk" -type d

Re: Ellobrate this event

Mohanveera1 — Thu, 31 Mar 2022 05:45:50 GMT

hi @VatsalJagani

i have checked the path and i do find the savedsearches.conf in mutiple directories.

1. /data/splunk/etc/system/default/savedsearches.conf

2. /data/splunk/etc/apps/search/default/savedsearches.conf

3. /data/splunk/etc/apps/splunk_monitoring_console/default/savedsearches.conf

4. /data/splunk/etc/apps/Splunk_TA_paloalto/default/savedsearches.conf (there are so many savedsearches in different directories with different app name.)

Please suggest me in which do i have to look the savedsearches.conf

Thanks in advance...

Re: Ellobrate this event

VatsalJagani — Thu, 31 Mar 2022 05:49:40 GMT

Try this now:

cd /data/splunk/etc/ grep -rinl "Single User Failed Attempt"

And see where this alert is still present.

Re: Ellobrate this event

Mohanveera1 — Thu, 31 Mar 2022 06:01:49 GMT

Dear @VatsalJagani

thank you for your previous reply i have go through it but there are so many files that came up with permission denied. And i haven't found any single user attempt failed in it.

please revert me in there is any other way..

thanks in advance....

Re: Ellobrate this event

VatsalJagani — Thu, 31 Mar 2022 06:05:10 GMT

Seems like a file permission issue you are facing.

Run the commands with the root user. Or prepend the commands with sudo

Re: Ellobrate this event

Mohanveera1 — Thu, 31 Mar 2022 06:27:15 GMT

Thank you very much @VatsalJagani

i have found the savedsearches.conf but i can,t access the file due to privilege issues. i will contact my server team and transfer the duplicate file to my pc. and can you please suggest me what i have to search in the conf file.

Thanks in advance.

Re: Ellobrate this event

VatsalJagani — Thu, 31 Mar 2022 06:30:52 GMT

Look for the stanza name (below line) in the file. That should give you the answer you are looking for why that username is still showing in the logs.

[Single User Failed Attempt]

Re: Ellobrate this event

Mohanveera1 — Thu, 31 Mar 2022 10:09:28 GMT

hi @VatsalJagani

i have checked the savedsearches.conf for single user attempt and it is as below and everything is normal in it and i didn't find any abnormality in it. please find the screenshot in it and please tell me if you find any abnormality.

thanks in advance.....

Re: Ellobrate this event

VatsalJagani — Thu, 31 Mar 2022 11:33:37 GMT

Which location did you find this file?

Other than that I don't see any issues, unless Splunk is behaving something differently.