Splunk on splunk SOS activity logging

[soc_splunk]

hello,

our PCI auditor has had a look at the logging capabilities on Splunk and is concerned about the "can_delete" user's capabilities. One thing that will get him to "like" splunk would be if there was some way of logging this action AND the IP it came from.

SOS can log all of the user's search activities, however, it just shows a username, and does not tie that to an IP address. I can't find an IP in the raw logs either. Does anyone know an easy way to find this out and show it to our auditor?

Thanks!

Ken

[pehlke]

Realistically, though, this is an audit/compliance problem like any other. If you really need this kind of auditing, the answer is to have your splunk instances forward _internal and _audit to another set of splunk instances, to which users on the production instances have no access. You'd then have a different set of trusted users who have access to the audit instances, and those users will audit the activity of the production users.

Standard compliance practice: establish controls, record activity elsewhere, and use out-of-band auditors.

[hexx]

Such a correlation is indeed possible:

• Search for search strings containing the delete command in events from source $SPLUNK_HOME/var/log/splunk/audit.log in the _audit index:
  
  
01-02-2013 05:57:29.121 -0800 INFO AuditLogger - Audit:[timestamp=01-02-2013 05:57:29.121, user=odisciullo, action=search, info=granted , search_id='1357135049.34106', search='search index=_internal | head 1 | delete', autojoin='1', buckets=0, ttl=600, max_count=10000, maxtime=8640000, enable_lookups='1', extra_fields='', apiStartTime='ZERO_TIME', apiEndTime='ZERO_TIME', savedsearch_name=""][n/a]


• Search for the SID (search ID) string of those searches against events from source $SPLUNK_HOME/var/log/splunk/web_access.log in the _internal index:
  
  
10.160.255.78 - odisciullo [02/Jan/2013:05:57:30.590 -0800] "GET /en-US/splunkd/search/jobs/1357135049.34106/timeline?offset=0&count=1000 HTTP/1.1" 200 - "https://undiag.splunk.com:8000/en-US/app/support_diag/flashtimeline?q=search%20index%3D_internal%20%..." "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.101 Safari/537.11" - 50e43cca977fd3acab1850 13ms


Here's a search that should be able to do this correlation for you and build a table of results showing time, search string, user and client IP for each search that contains the delete command:


index=_internal source=*/web_access.log "/services/search/jobs/"
| rex "/services/search/jobs/(?<search_id>[\d\.]+)"
| join search_id
[search index=_audit delete
| rex "search='(?<search>.*?)', autojoin"
| regex search="\|\s*delete"
| replace '*' with * in search_id
| fields _time, user, search, search_id]
| stats min(_time) AS _time first(clientip) AS "client IP" first(user) AS User values(search) AS search by search_id
| rename search_id AS "search ID"


[alacercogitatus]

I will also comment on here that you may want to remove the "can_delete" role from users if you are concerned with the ability to delete events. http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Delete#Description

[kmeeks_splunk · Accepted solution]

You can see the "delete" activity from the auditlog (so inside the splunk search window type - index=_audit sourcetype=audittrail delete). This will show you all the delete activity. This will also provide you with a search_id. You can locate this search id within the sourcetype="splunk_web_access" within the _internal index. For example: index=_internal sourcetype="splunk_web_access sid=. This will produce the actual web access log that executed the delete command. Providing for you, the IP address and the username.

Hopefully this helps.

[DaveSavage]

Thanks kmeeks - very elegant!

[soc_splunk]

Fantastic, Thanks! That works.

[DaveSavage]

A familiar challenge, at least to us. We implemented named people and a naming policy as the individual is responsible for their code of conduct. I would negotiate with the auditor around the use of IP addresses - yes, nice to have but of no use if they use a hot desk facility, remote in from home etc etc etc?
Having said all that, there is a useful field Source_Network_Address on WinEventLog:Security 😉

[soc_splunk]

Hi Dave,

Thanks for the input.

Actually we had attempted to blow the auditor away with our "automated investigation with Splunk" - A user performs an action (e.g. accesses a customer's account), Splunk then looks up vacation data to see if that user is scheduled to work, then the access control logs to see if they're in the Building or VPN logs if they're not, then checks if the computer they're logged in to with AD is the same as the one corresponding to the account access, then if the user is allowed to access the account in question and then alert as appropriate if one or more items do not comply.

So the auditor asked "what if an account is hijacked and someone deletes the logs from that account" - Which basically throws our investigation abilities out the window. Any user with that option (currently only me) can just say their account was hijacked. It's a "who guards the guards" game.

Regards,

Ken

[DaveSavage]

Yes, see your challenge, that's a fairly rigorous series of checks you are performing!
kmeeks has covered all of the bases. I'd only add from recent experience:
Access and roles - defined and limited not only within Splunk but from the OS / access layer
Policy - both written (as say RMADS) and the Windows type (if that is your OS)
Security Admin to guard for you, daily checks (per policy), security incident process for queries even if false positives
Look at some other apps / plug ins. (UK)GPG13 has similar requirements to PCI by Eqalis. Not a plug, still a fair bit of work to deploy.
Br 😉