Splunk Search

I have a tstats search that works for me (admin) but not other users (who inherit role from 'user). Why is that?

wrangler2x
Motivator

I have a user who is asking how to show earliest logs indexed by the indexer for a particular host. I tried this simple search using tstats, but when he runs it he gets no results back. Here is the search:

| tstats min(_time) as earliest_log max(_time) as latest_log WHERE index=winevent_dc_index host=somehost.uci.edu by host
| convert timeformat="%Y-%m-%d %H:%M:%S" ctime(earliest_log) ctime(latest_log) 

Is there some special capability that needs to be added to his role for this to work? He gets back no results at all.

1 Solution

wrangler2x
Motivator

It seems that without rearchitecting my role-based access (and my Deployment Apps to accommodate a slew of new indexes) I can't use tstats at all for my dashboard for figuring-out earlies and latest logs. So, I've turned to using metadata, and this works (substitute $HOSTNAME$ in the actual search -- this is from my dashboard, prettified.

| metadata type=hosts index=*
| search host=$HOSTNAME$
| stats count by host totalCount firstTime recentTime
| fieldformat Count=tostring(Count, "commas")
| convert timeformat="%m/%d/%Y %T" ctime(recentTime) ctime(firstTime)
| rename totalCount as Count recentTime as "Last Update" firstTime as "Earliest Update"
| fields - count

View solution in original post

wrangler2x
Motivator

It seems that without rearchitecting my role-based access (and my Deployment Apps to accommodate a slew of new indexes) I can't use tstats at all for my dashboard for figuring-out earlies and latest logs. So, I've turned to using metadata, and this works (substitute $HOSTNAME$ in the actual search -- this is from my dashboard, prettified.

| metadata type=hosts index=*
| search host=$HOSTNAME$
| stats count by host totalCount firstTime recentTime
| fieldformat Count=tostring(Count, "commas")
| convert timeformat="%m/%d/%Y %T" ctime(recentTime) ctime(firstTime)
| rename totalCount as Count recentTime as "Last Update" firstTime as "Earliest Update"
| fields - count

View solution in original post

realsplunk
Builder

you need get_metadata capability

0 Karma

micahkemp
Champion

Nice workaround!

0 Karma

micahkemp
Champion

Though technically this is an answer regarding "how do I get this data I need," instead of "why didn't this work, and how would I make it work?" 😉

0 Karma

wrangler2x
Motivator

Yes, I agree with that, though my goal was to provide this functionality for people bringing up forwarders to use.

Actually, the metadata search is much faster than tstats was, so it is a better solution.

I found the information provided by you about naming conventions for indexes very interesting, though. Thanks for the input here.

0 Karma

micahkemp
Champion

This previous answers post provides a way to examine if the restrict search terms are changing your searches:

As a user, you can easily spot if your searches are being filtered using this method by running a search, such as index=*, and click Job > Inspect Job, click Search job properties, and identify potential search-time fields within remoteSearch.

I'd suggest having the user run the tstats search and checking the search job properties for the executed search string to see if it's crafting an invalid search by trying to merge tstats with the restrict search terms.

0 Karma

wrangler2x
Motivator

Yes, that is what is happening. I was not able to see it in the Search job properties, but it is plain as day in the search.log (link at the bottom of the job inspector Search job properties):

01-09-2018 16:46:57.548 INFO SearchProcessor - Final search filter= ( tag::host=oit_dba )
01-09-2018 16:46:57.548 INFO TsidxStats - Initial expanded filtering search: '( index=winevent_dc_index host=somehost.uci.edu ) ( tag::host=oit_dba )'

Back when we put this in the Restrict search terms tstats did not exist. This breaks it, and I don't know how to get around that while maintaining these restrictions.

Any ideas?

0 Karma

micahkemp
Champion

Converted to an answer, since it helped you track down the root cause.

I don’t believe there is a way to enforce search terms that would work with tstats. I think search terms are generally advised against, anyway. The advice is to do all permissions by index, splitting into a new index if necessary to accomplish this.

0 Karma

wrangler2x
Motivator

That's not going to work for me. I take logs from many groups around the campus, with windows events for domain controllers in one index and windows events from other systems in another. There are over 20 group roles to allow them to look at their logs only, and splitting these two indexes into twenty-some-odd indexes is crazy. I already have 21 indexes not counting the Splunk ones.

0 Karma

micahkemp
Champion

I hear what you're saying, though it's still "the answer".

And I will also tell you that you wouldn't be the first admin to use a ton of indexes to accomplish search permissions. I was at a recent customer site and they had roughly a hundred indexes.

What you would want to consider is a naming convention for indexes, something like:

<department>_<data type>

Then you configure your authorize.conf to allow wildcards as necessary, like:

[role_<department>]
srchIndexesAllowed = <department>_*

Again, I understand this doesn't sound like something you want to do, but separating data by index is the way to define search permissions.

wrangler2x
Motivator

Thanks for your suggestions @micahkemp. I definitely will keep these ideas in mind for the future.

0 Karma

micahkemp
Champion

Realizing the full issue isn't solved, if you think the answer posted accurately answered the question posted, please accept it so this question no longer looks open.

0 Karma

elliotproebstel
Champion

Just jumping in here to add some support to what @micah is telling you. Yes, there will likely be some pain at the time of transition between the way you've been doing things and the way he is advising, but it's pretty much the only way I've seen access control be scalable. Using search terms to restrict users is going to run you into this wall over and over again.

Investing some time now to develop a smart naming convention and isolating your logs into indexes that match your access restriction needs is really the best bet.

0 Karma

micahkemp
Champion

Permissions to run tstats are the same permissinos required to search the index(es) tstats fetches from. To see if that's what's causing your issue here try having the users run this search:

index=winevent_dc_index | stats count

In order to grant access to the index (and therefore tstats for that index) a role that the user belongs to will need to have that index added to its search capabilities. Check out the Splunk documentation on that process.

wrangler2x
Motivator

The role for this user allows access to the winevent_dc_index, and executing a search like that returns a count. However, I was just looking at the role and it has tag::host=oit_wsg in the Restrict search terms box under Search Restrictions. This would restrict any search of the logs to hosts that that role is expected to have access to (hosts that had been tagged this way because of an entry into Field Value Pairs in Settings). I verified that the host in the tstats search is tagged that way.

0 Karma