Security
Highlighted

How to restrict access for multiple types of logs and grant permissions for appropriate groups?

Communicator

The background:
I have multiple types of logs from multiple groups being piped into Splunk into 1 index
The index=index1
The sourcetype=syslog

The scenario:
One of my groups wants to access their VPN data to generate reports (active/closed, duration, user_auth stuff, etc)
In the future, each group will want the same access (about 25-30 groups)

The Goal:
Here's my approach
1) isolate logs to just show VPN traffic
2) within isolated VPNLOGS I need to be able to isolate the grouptraffic
3) and Lastly, provide group access to their data while keeping each other groups data separate and private
4) and then create fields to visualize (session_ID, username, creation time, bytes in/out, duration, OS, Browser, Client IP, assigned IP)

I'm having a lot of fun trying to figure this out, but I'm hoping for some assistance, direction, advice on a solution and/or if my approach can work, but is not optimized or maybe it's just wrong...

First thought is to optimize/change the way I am receiving logs to help make things easier for me to figure out, but i'm not sure how to tackle that or suggest what needs to be done, so I am working with what I was given..


Current work:
The below search string removes all unnecessary logs using the process field. I then use the transaction command to assemble all the events by session_id field (which is unique only to the VPN logs) and I then pipe into a search to grab the transactions for a specific group. (Not sure if this is the best way)

sourcetype=syslog process=* NOT (logger OR cron OR crond OR syslog-ng OR tmsh OR snmpd OR sshd) | transaction session_id | search group=GROUP1

I was hoping to put this search string into the "restrictsearchterms" field within user permissions/roles and that would provide the solution to data privacy and ability to create reports/dashboard for each group... problem solved... but it a search like this isn't allowed in the "restrictsearchterms"

I've provided some modified logs:
[process=] built in sourcetype field. This field I found target events for each log type

[session_id] I created

<\d+>\w{3}\s\d+\s\d{2}:\d{2}:\d{2}\sf5-sslvpn-\d\s\S+\s\d+:\d:\s(?\w+)

[group=] i created to tell me which group the session_id transaction belongs too

(?=[^R]*(?:Resource: /|R.*Resource: /))^[^/\n]*/(?P\w+)

[username=]

(?=[^U]*(?:Username '|U.*Username '))^[^'\n]*'(?P[^']+)

[user_auth=] I will grab the "Received client info - Type: to make this expression

=========================================== sample logs=================================

_raw Data

<182>Jan 12 16:51:44 f5_vpn-1 logger: [ssl_acc] 127.0.0.1 - - [12/Jan/1980:16:51:43 -0700] "/igeen/igeenPortal.cgi" 200 8787
<182>Jan 12 16:51:44 f5_vpn-1 logger: [ssl_req][12/Jan/1980:16:51:43 -0700] 127.0.0.1 TLSv1 DHE-RSA-AES256-SHA "/igeen/igeenPortal.cgi" 8787
<182>Jan 12 16:51:38 f5_vpn-1 logger: [ssl_req][12/Jan/1980:16:51:38 -0700] 999.999.999.991 TLSv1.2 DHE-RSA-AES256-SHA "/xui/update/configuration/alert/statusmenu/coloradvisory" 3762
<182>Jan 12 16:51:38 f5_vpn-1 logger: [ssl_acc] 198.238.211.12 - user1 [12/Jan/1980:16:51:38 -0700] "/xui/update/configuration/alert/statusmenu/coloradvisory" 200 3762
<182>Jan 12 16:51:38 f5_vpn-1 logger: [ssl_acc] 198.238.121.17 - user2 [12/Jan/1980:16:51:38 -0700] "/xui/update/configuration/alert/statusmenu/coloradvisory" 200 3758
<182>Jan 12 16:51:38 f5_vpn-1 logger: [ssl_req][12/Jan/1980:16:51:38 -0700] 999.999.999.100 TLSv1.2 AES256-SHA "/xui/update/configuration/alert/statusmenu/coloradvisory" 3758
<182>Jan 12 16:51:36 f5_vpn-2 logger: [ssl_req][12/Jan/1980:16:51:36 -0700] 127.0.0.1 TLSv1 DHE-RSA-AES256-SHA "/iControl/iControlPortal.cgi" 8341
<182>Jan 12 16:51:36 f5_vpn-2 logger: [ssl_acc] 127.0.0.1 - - [12/Jan/1980:16:51:36 -0700] "/igeen/igeenPortal.cgi" 200 8341
<182>Jan 12 16:51:36 f5_vpn-2 logger: [ssl_req][12/Jan/1980:16:51:36 -0700] 127.0.0.1 TLSv1 DHE-RSA-AES256-SHA "/iControl/iControlPortal.cgi" 2818
<182>Jan 12 16:51:36 f5_vpn-2 logger: [ssl_acc] 127.0.0.1 - - [12/Jan/1980:16:51:36 -0700] "/igeen/igeenPortal.cgi" 200 2818
<182>Jan 12 16:51:36 f5_vpn-2 logger: [ssl_req][12/Jan/1980:16:51:36 -0700] 127.0.0.1 TLSv1 DHE-RSA-AES256-SHA "/iControl/iControlPortal.cgi" 8787
<182>Jan 12 16:51:36 f5_vpn-2 logger: [ssl_acc] 127.0.0.1 - - [12/Jan/1980:16:51:36 -0700] "/igeen/igeenPortal.cgi" 200 8787
<182>Jan 12 16:51:36 f5_vpn-2 logger: [ssl_req][12/Jan/1980:16:51:36 -0700] 127.0.0.1 TLSv1 DHE-RSA-AES256-SHA "/iControl/iControlPortal.cgi" 8787
<182>Jan 12 16:51:36 f5_vpn-2 logger: [ssl_acc] 127.0.0.1 - - [12/Jan/1980:16:51:36 -0700] "/igeen/igeenPortal.cgi" 200 8787
<141>Jan 12 16:51:35 f5_vpn-1 sbb3[3645]: 01490502:5: 696fdb86: Session deleted due to user inactivity or errors.
<141>Jan 12 16:51:35 f5_vpn-1 sbb[3645]: 01490521:5: 648bd53e: Session statistics - bytes in: 471704, bytes out: 777008
SessionID2 = 648bd53e
<182>Jan 12 16:51:32 f5_vpn-1 logger: [ssl_req][12/Jan/1980:16:51:32 -0700] 999.999.999.991 TLSv1.2 DHE-RSA-AES256-SHA "/xui/update/configuration/alert/statusmenu/coloradvisory" 3762
<182>Jan 12 16:51:32 f5_vpn-1 logger: [ssl_acc] 999.999.999.991 - user21 [12/Jan/1980:16:51:32 -0700] "/xui/update/configuration/alert/statusmenu/coloradvisory" 200 3762
<141>Jan 12 16:51:31 f5_vpn-1 sbb2[3645]: 01490502:5: 4ebfa5a3: Session deleted due to user inactivity or errors.



1 user sessionID - 
<141>Jan 12 13:52:28 f5-vpn-1 tmm[3645]: 01490521:5: a620f620: Session statistics - bytes in: 17611113, bytes out: 58663535 process = tmm session_id = a620f620
<141>Jan 12 13:51:47 f5-vpn-1 tmm[3645]: 01490502:5: a620f620: Session deleted due to user inactivity or errors. process = tmm session_id = a620f620
<141>Jan 12 13:36:49 f5-vpn-1 tmm1[3645]: 01490505:5: a620f620: PPP tunnel 0x57008bd86200 closed. process = tmm1 session_id = a620f620
<141>Jan 12 08:01:40 f5-vpn-1 tmm1[3645]: 01490505:5: a620f620: PPP tunnel 0x57008bd86200 started. process = tmm1 session_id = a620f620
<141>Jan 12 08:01:40 f5-vpn-1 tmm1[3645]: 01490549:5: a620f620: Assigned PPP Dynamic IPv4: 181.231.31.123 Tunnel Type: VPN_TUNNELTYPE_TLS NA Resource: /GROUP1/NA_GROUP1_CMDCENTER GROUP1 = GROUP1 process = tmm1 session_id = a620f620
<141>Jan 12 08:01:23 f5-vpn-2 apd[3594]: 01490102:5: a620f620: Access policy result: Full process = apd session_id = a620f620
<141>Jan 12 08:01:23 f5-vpn-2 apd[3594]: 01490005:5: a620f620: Following rule 'Out' from item 'resources_pc' to ending 'Allow'process = apd session_id = a620f620
<141>Jan 12 08:01:23 f5-vpn-2 apd[3594]: 01490115:5: a620f620: Following rule 'fallback' from item 'Route Domain and SNAT Selection' to terminalout 'Out' process = apd session_id = a620f620
<141>Jan 12 08:01:23 f5-vpn-2 apd[3594]: 01490128:5: a620f620: Webtop '/GROUP1/WT_GROUP1_SSLVPN' assigned process = apd session_id = a620f620 <141>Jan 12 08:01:23 f5-vpn-1 apd[3594]: 01490008:5: a620f620: Connectivity resource '/GROUP1/NA_GROUP1_CMDCENTER' assigned
process = apd session_id = a620f620
<141>Jan 12 08:01:23 f5-vpn-2 apd[3594]: 01490115:5: a620f620: Following rule 'fallback' from item 'GROUP1 role group mapping' to terminalout 'Out'process = apd session_id = a620f620
<141>Jan 12 08:01:21 f5-vpn-2 apd[3594]: 01490010:5: a620f620: Username 'user1'process = apd session_id = a620f620 username = user1
<141>Jan 12 08:01:04 f5-vpn-1 tmm[3645]: 01490500:5: a620f620: New session from client IP 999.999.100.200 (ST=STATE/CC=US/C=NA) at VIP 999.999.166.200 Listener /GROUP1/VS_GROUP1_WC_SSLVPN (Reputation=Unknown)process = tmm session_id = a620f620
<141>Jan 12 08:01:04 f5-vpn-1 tmm[3645]: 01490544:5: a620f620: Received client info - Type: IE Version: 8 Platform: Win7 CPU: WOW64 UI Mode: Full Javascript Support: 1 ActiveX Support: 1 Plugin Support: 0 process = tmm session_id = a620f620
<141>Jan 12 08:01:04 f5-vpn-1 tmm[3645]: 01490506:5: a620f620: Received User-Agent header: Mozilla%2f4.0%20(compatible%3b%20MSIE%208.0%3b%20Windows%20NT%206.1%3b%20WOW64%3b%20Trident%2f4.0%3b%20SLCC2%3b%20.NET%20CLR%202.0.50727%3b%20.NET%20CLR%203.5.30729%3b%20.NET%20CLR%203.0.30729%3b%20Media%20Center%20PC%206.0%3b%20.NET4.0C%3b%20.NET4.0E%3b%20InfoPath.3).
0 Karma
Highlighted

Re: How to restrict access for multiple types of logs and grant permissions for appropriate groups?

SplunkTrust
SplunkTrust

If I remember rightly, this particular question has no simple, perfect answer. There are some options available though, so let me toss in my two cents.

The problem portions of what you are trying to accomplish are twofold - the restrict search terms option is very limited so there are things you simply can't do with it, and your goal number 3 implies you would have to do rather "non-limited" things with your search terms to get the right filtering.

I doubt a solution that just "suggests" restrictions but doesn't really enforce them wouldn't be good enough for your purposes. Still, if that might be sufficient you could prepend a macro to each group's dashboard searches that would do that filtering for you. If you want it to be stunning easy for people who don't know what they are doing to bypass, make sure to name it something like restrict_group_information and be really clear and easy to understand. If you want it to at least be somewhat more obscure (though no less secure) you could name it more generically and bundle it inside the root search and hope no one notices it. For instance, name it include_group_data and sneak your filtering in between a couple of regexes - regex scares most folks away. But again, this isn't very secure.

What I'd suggest is one of the following.

First, you could possibly create summary indexes for each group. There is no single answer to how to do this, a lot will depend on some details of what you really want the individual groups to see.

Second, and I think in this case possibly better, would be to use the "old school" way of creating a summary index (which frankly doesn't have to summarize anything). That way uses the collect command to write events into a summary index you've created. You'll notice, in fact, that Example 2 in that previous link is Collect statistics on VPN connects and disconnects, which while not really what you are trying to do, just seemed very generically pertinent. 🙂 You can then restrict access to the "real" index and only let each group see their summary index.

Now, what would you PUT into the summary index? I mean, you could insert raw events into it or "transaction-ized" events. For raw events, you may not be ABLE to do this easily because you need to only include session information for the right group, but it's difficult to determine which sessions those are with referring to previous events and that opens a can of worms about how to include only new rows without the older, defining events.

But, if a bit of lag in data is OK, you could create transactions and feed those results into it.

Anyway, unless someone has a better answer, you could giev these options a thought and ask again about more direction when you think you may have a way forward.

View solution in original post

Highlighted

Re: How to restrict access for multiple types of logs and grant permissions for appropriate groups?

Communicator

I appreciate you taking the time to offer your advice and give the examples... I was thinking the length of my post would turn people away ....

I've got it to work using the your help. I am using a summary index and a schedule search string with collect command

I've setup roles/permissions by putting the index=summary_idex into the restrict field.

Things I am working with this solution are ... the backfill of the data into the summary_index, an assortment of global permissions I need to clean up and how/wear to attach my field extractions so I don't have to recreate them for each new instance of this setup. Moving forward there are a couple of optimizations and changes i can do to this solution.

0 Karma
Highlighted

Re: How to restrict access for multiple types of logs and grant permissions for appropriate groups?

SplunkTrust
SplunkTrust

If you use all these as the same sourcetype, your sourcetype-based extractions will all work as long as you set the permissions right on those.

0 Karma
Highlighted

Re: How to restrict access for multiple types of logs and grant permissions for appropriate groups?

Communicator

Yes, I am doing this. Thank You!

0 Karma
Highlighted

Re: How to restrict access for multiple types of logs and grant permissions for appropriate groups?

Communicator

Thanks again Rich7177 ... I am currently testing the transaction -> collect -> summary index and things seem to be working good, except for the duration or _time is not working as expected in the summary index. Would you happen to have any knowledge on this?

Here is the search string

 index=index1 (process=zmi* OR process=apz)
 | transaction session_id startswith="New session" endswith="Session deleted" 
 | search department=SALES
 | collect index=logs-sales addtime=true | addinfo

In the summary index the duration field info isn't pulled over but the transaction structure is still there and events are grouped but duration or _time is a bit screwy. If I run the transaction command again in the summary-index, I get a duration=0 field (before the push to the summary, the duration field is correct) ... Any ideas? thanks again

0 Karma
Highlighted

Re: How to restrict access for multiple types of logs and grant permissions for appropriate groups?

Ultra Champion

The title of the question - 'How to restrict access for multiple types of logs and grant permissions for appropriate groups?' and the fact that the current design has one index almost contradict each other, because the access control of Splunk is based on the indexes. I would try to build a set of indexes and assign them the right access level. Not sure whether it's possible ; -)

A related question at setting specific indexes.

0 Karma