Monitoring Splunk

How do I Ellobrate this event?

Mohanveera1
Explorer

Hi splunk experts,

Can anyone elaborate this below event and tell me why this event is getting triggered? the user name in this event has left the organization and we removed his access and transferred the knowledge objects to other person also but we are getting his name in the below event. and please help me how to avoid this type type of alerts also.

127.0.0.1 - **User name*** [30/Mar/2022:09:29:54.891 +0000] "POST /servicesNS/nobody/search/saved/searches/Single%20User%20Failed%20Attempt/notify?trigger.condition_state=1 HTTP/1.1" 200 1933 "-" "Splunk/8.1.0 (Linux 4.15.0-1023-azure; arch=x86_64)" - 2ms

 

this event is getting displayed when we search by using the query: index=_internal sourcetype= splunkd_access user=*.

 

thanks in advance.

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Grep for this in the savesearches.conf from the Splunk instance in the backend if you can.

cd /opt/splunk/etc/
grep -rinl "Single User Failed Attempt"

See if you can see a file that contains it. (Specifically savedsearches.conf)
That should give an answer in most cases. 

0 Karma

Mohanveera1
Explorer

hi @VatsalJagani 

thank you for your response.

in the splunk instance i have go till opt directory, but in opt directory i cant find splunk. it only contains the backup scripts. is there any other way that i can look into.....

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

By /opt/splunk I meant your Splunk installation directory. This is generally the case but it seems in your case Splunk is installed in a different location.

* I hope you are in the backend of search head.

* You can find the Splunk installation directory by running the command, find / -name "splunk" -type d

0 Karma

Mohanveera1
Explorer

hi @VatsalJagani 

i have checked the path and i do find the savedsearches.conf in mutiple directories.

1. /data/splunk/etc/system/default/savedsearches.conf

2. /data/splunk/etc/apps/search/default/savedsearches.conf

3. /data/splunk/etc/apps/splunk_monitoring_console/default/savedsearches.conf

4. /data/splunk/etc/apps/Splunk_TA_paloalto/default/savedsearches.conf (there are so many savedsearches in different directories with different app name.)

Please suggest me in which do i have to look the savedsearches.conf

 

Thanks in advance...

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Try this now:

cd /data/splunk/etc/
grep -rinl "Single User Failed Attempt"

And see where this alert is still present.

0 Karma

Mohanveera1
Explorer

Dear @VatsalJagani 

 

thank you for your previous reply i have go through it but there are so many files that came up with permission denied. And i haven't found any single user attempt failed in it.

please revert me in there is any other way..

thanks in advance....

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Seems like a file permission issue you are facing.

Run the commands with the root user.  Or prepend the commands with sudo

0 Karma

Mohanveera1
Explorer

Thank you very much @VatsalJagani 

i have found the savedsearches.conf but i can,t access the file due to privilege issues. i will contact my server team and transfer the duplicate file to my pc. and can you please suggest me what i have to search in the conf file.

 

Thanks in advance.

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust

Look for the stanza name (below line) in the file. That should give you the answer you are looking for why that username is still showing in the logs.

[Single User Failed Attempt]
0 Karma

Mohanveera1
Explorer

hi @VatsalJagani 

i have checked the savedsearches.conf for single user attempt and it is as below and everything is normal in it and i didn't find any abnormality in it. please find the screenshot in it and please tell me if you find any abnormality.

Mohanveera1_0-1648721206703.png

thanks in advance.....

0 Karma

VatsalJagani
SplunkTrust
SplunkTrust
Which location did you find this file?

Other than that I don't see any issues, unless Splunk is behaving something differently.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

How is the saved search "Single User Failed Attempt" set up? Who owns it? Which user does it run as?

0 Karma

Mohanveera1
Explorer

hi @ITWhisperer 

the "Single User Failed Attempt" is setup as an alert type. it was previously owned by the member that left the organisation (mentioned in the above thread ***user name***), after he left i have reassigned the alert to myself. this alert does not run by any user as it was kept as real time alert.

 

Thanks in advance....

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

How did you reassign the alert?

0 Karma

Mohanveera1
Explorer

@ITWhisperer 


i clicked on the All Configurations in the settings option to get all Knowledge objects  and filtered them for the previous owner. Then i reassigned all the alerts and reports from him to the new user.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

It looks like something is running on local host. How often does it happen? Could it be a cronjob or something like that?

0 Karma

Mohanveera1
Explorer

Hi @ITWhisperer 

 

thank you for your response.

 

This event is generating everyday and its not only with one user, its with multiple users who left the organization. The cronjob are like knowledge objects, but all the knowledge objects created by the user is assigned to the new user. so can you please suggest me how can we check any new cronjobs that are available. and how can we check in local host as well????

 

thanks in advance..

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Every day? At the same time every day or at random times?

What do you mean by "cronjob are like knowledge objects"?

Do you have shell access to your splunk hosts?

What other processes are running on your splunk hosts?

What other users are logged on to your splunk hosts?

0 Karma

Mohanveera1
Explorer

hi @ITWhisperer 

please find the answers according to your queries... and please revert me incase if any of the info is required.

every day? At the same time every day or at random times? -- these events are triggering for every minute

What do you mean by "cronjob are like knowledge objects"? -- cronjob in splunk is meant to be knowledge objects such as dashboards, alerts, reports which are scheduled to run on specific time. (As of my knowledge) please share what do you mean by cronjob in your opinion.

Do you have shell access to your splunk hosts? -- yes i do have access to shell in splunk host

What other processes are running on your splunk hosts? --  there were so many processes are running in splunk host, as i checked by using command ps aux. Is there any specific processes that i have to look at?

What other users are logged on to your splunk hosts? -- Is this about the user in the splunk or the users that can access the splunk host.

 

Thanks in advance.....

 

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...