All Apps and Add-ons

Splunk on splunk SOS activity logging

soc_splunk
Explorer

hello,

our PCI auditor has had a look at the logging capabilities on Splunk and is concerned about the "can_delete" user's capabilities. One thing that will get him to "like" splunk would be if there was some way of logging this action AND the IP it came from.

SOS can log all of the user's search activities, however, it just shows a username, and does not tie that to an IP address. I can't find an IP in the raw logs either. Does anyone know an easy way to find this out and show it to our auditor?

Thanks!

Ken

1 Solution

kmeeks_splunk
Splunk Employee
Splunk Employee

You can see the "delete" activity from the auditlog (so inside the splunk search window type - index=_audit sourcetype=audittrail delete). This will show you all the delete activity. This will also provide you with a search_id. You can locate this search id within the sourcetype="splunk_web_access" within the _internal index. For example: index=_internal sourcetype="splunk_web_access sid=. This will produce the actual web access log that executed the delete command. Providing for you, the IP address and the username.

Hopefully this helps.

View solution in original post

pehlke
Splunk Employee
Splunk Employee

Realistically, though, this is an audit/compliance problem like any other. If you really need this kind of auditing, the answer is to have your splunk instances forward _internal and _audit to another set of splunk instances, to which users on the production instances have no access. You'd then have a different set of trusted users who have access to the audit instances, and those users will audit the activity of the production users.

Standard compliance practice: establish controls, record activity elsewhere, and use out-of-band auditors.

hexx
Splunk Employee
Splunk Employee

Such a correlation is indeed possible:

  • Search for search strings containing the delete command in events from source $SPLUNK_HOME/var/log/splunk/audit.log in the _audit index:


    01-02-2013 05:57:29.121 -0800 INFO AuditLogger - Audit:[timestamp=01-02-2013 05:57:29.121, user=odisciullo, action=search, info=granted , search_id='1357135049.34106', search='search index=_internal | head 1 | delete', autojoin='1', buckets=0, ttl=600, max_count=10000, maxtime=8640000, enable_lookups='1', extra_fields='', apiStartTime='ZERO_TIME', apiEndTime='ZERO_TIME', savedsearch_name=""][n/a]

  • Search for the SID (search ID) string of those searches against events from source $SPLUNK_HOME/var/log/splunk/web_access.log in the _internal index:


    10.160.255.78 - odisciullo [02/Jan/2013:05:57:30.590 -0800] "GET /en-US/splunkd/search/jobs/1357135049.34106/timeline?offset=0&count=1000 HTTP/1.1" 200 - "https://undiag.splunk.com:8000/en-US/app/support_diag/flashtimeline?q=search%20index%3D_internal%20%..." "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.101 Safari/537.11" - 50e43cca977fd3acab1850 13ms

Here's a search that should be able to do this correlation for you and build a table of results showing time, search string, user and client IP for each search that contains the delete command:


index=_internal source=*/web_access.log "/services/search/jobs/"
| rex "/services/search/jobs/(?<search_id>[\d\.]+)"
| join search_id
[search index=_audit delete
| rex "search='(?<search>.*?)', autojoin"
| regex search="\|\s*delete"
| replace '*' with * in search_id
| fields _time, user, search, search_id]
| stats min(_time) AS _time first(clientip) AS "client IP" first(user) AS User values(search) AS search by search_id
| rename search_id AS "search ID"

alacercogitatus
SplunkTrust
SplunkTrust

I will also comment on here that you may want to remove the "can_delete" role from users if you are concerned with the ability to delete events. http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Delete#Description

kmeeks_splunk
Splunk Employee
Splunk Employee

You can see the "delete" activity from the auditlog (so inside the splunk search window type - index=_audit sourcetype=audittrail delete). This will show you all the delete activity. This will also provide you with a search_id. You can locate this search id within the sourcetype="splunk_web_access" within the _internal index. For example: index=_internal sourcetype="splunk_web_access sid=. This will produce the actual web access log that executed the delete command. Providing for you, the IP address and the username.

Hopefully this helps.

DaveSavage
Builder

Thanks kmeeks - very elegant!

0 Karma

soc_splunk
Explorer

Fantastic, Thanks! That works.

0 Karma

DaveSavage
Builder

A familiar challenge, at least to us. We implemented named people and a naming policy as the individual is responsible for their code of conduct. I would negotiate with the auditor around the use of IP addresses - yes, nice to have but of no use if they use a hot desk facility, remote in from home etc etc etc?
Having said all that, there is a useful field Source_Network_Address on WinEventLog:Security 😉

0 Karma

soc_splunk
Explorer

Hi Dave,

Thanks for the input.

Actually we had attempted to blow the auditor away with our "automated investigation with Splunk" - A user performs an action (e.g. accesses a customer's account), Splunk then looks up vacation data to see if that user is scheduled to work, then the access control logs to see if they're in the Building or VPN logs if they're not, then checks if the computer they're logged in to with AD is the same as the one corresponding to the account access, then if the user is allowed to access the account in question and then alert as appropriate if one or more items do not comply.

So the auditor asked "what if an account is hijacked and someone deletes the logs from that account" - Which basically throws our investigation abilities out the window. Any user with that option (currently only me) can just say their account was hijacked. It's a "who guards the guards" game.

Regards,

Ken

DaveSavage
Builder

Yes, see your challenge, that's a fairly rigorous series of checks you are performing!
kmeeks has covered all of the bases. I'd only add from recent experience:
Access and roles - defined and limited not only within Splunk but from the OS / access layer
Policy - both written (as say RMADS) and the Windows type (if that is your OS)
Security Admin to guard for you, daily checks (per policy), security incident process for queries even if false positives
Look at some other apps / plug ins. (UK)GPG13 has similar requirements to PCI by Eqalis. Not a plug, still a fair bit of work to deploy.
Br 😉

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...