Splunk Search

How can I track user activity trending via searches?

Peter
Path Finder

I am attempting to write a search that can alert if a user deviates from some normal data viewing pattern. The event log in question records every time a user sees a bit of information, identified by the cID. Sometimes they view the same cID multiple times per day, but I only care about the distinct number they view in some time period. Ultimately, I would like to determine the average number of unique cIDs each user views over some time period (maybe daily, maybe weekly) so that I can look for exceptions and trigger an alert automatically.

So if userA views 150 unique cIDs on average each day (over a 30 day span), and one day they view 400 unique cIDs, I would like an alert to be triggered. I have looked at the "anomalies", "delta", and "outlier" commands, but can't seem to get a working search. I am working on a search that takes the avg(dc(cID)) by username, but that seems to be a dead end due to some Splunk restrictions. I'm not set on using avg() as the determining parameter, I just need something that can detect anomalous behavior.

Anyone have a better approach?

Tags (1)

Simeon
Splunk Employee
Splunk Employee

I would summary index the distinct count of cID values and make sure the user field is also indexed. From there, you should be able to run a "stats range" search against the cID returned which will give you the daily difference. Finally, run a search against the output of the "stats range" that is greater than the level you want to trigger upon. So in search language, maybe this:

Save this search to summary index every night (also save the count_cID as a field):

sourcetype=event_log | sistats dc(cID) as count_cID by user

Run this search every 24+ hours to check the change (using a difference of +-100:

index=summary search_name=<above_saved_search> | stats range(count_cID) as cID_change by user | search cID_change > 100

Peter
Path Finder

I'm first tried the search you suggested, but have now tried index=summary search_name="Summary - cID by username" | stats max(psrsvd_ct_sec_cardID) by username. This gives me 378 results in the last 7 days, but the table generated only lists the username and no data for the max(field).

0 Karma

Simeon
Splunk Employee
Splunk Employee

How are you reporting on the field?

0 Karma

Peter
Path Finder

Simeon - The field settings appear to store a static value, rather than allowing me to name the field that the dc(cID) is stored in and I can't seem to actually report on the data. I see that a field "psrsvd_ct_cID" is populated with the relevant data, but I can't actually chart/report on it? Any thoughts or should I open a splunk trouble ticket at this point? I appreciate your help.

0 Karma

Simeon
Splunk Employee
Splunk Employee

Yes, you are correct... you may have to modify your sistats to be something like:

| sistats dc(cID) as count_cid

AND you will need to save count_cid in the field settings for the summary index.

0 Karma

Peter
Path Finder

Simeon - do I need to save the sistats "dc(cID)" data as a new field? The range(cID) command seems to not pull any data (even for the 1 day I have).

0 Karma

Peter
Path Finder

All good help. I've read up on the docs and have populated the search. Giving it a few days to populate and validate. Thank you!

0 Karma

Simeon
Splunk Employee
Splunk Employee

I recommend you read the summary indexing documentation before doing the above. To directly answer your question, you would need to enable the summary indexing via the check box in the saved search and you should specify the completed days range (-1d@d would give you the past whole day).

Lowell
Super Champion

Yes. A search with sistats (just like a search with plain stats) needs to setup to enable summary indexing. (The "si" prefix commands don't magically feed any data to the summary index. They are just indented to be more summary-index friendly commands.)

Peter
Path Finder

And another question: when saving the sistats search, do I need to enable the "Enable Summary Indexing" action checkbox?

0 Karma

Peter
Path Finder

For the sistats scheduled search do I need to specify a time range (e.g. earliest=-1d@d)?

0 Karma

Peter
Path Finder

Thanks so much! I'll try this and get back to you.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

You may want to look at the trendline and streamstats commands

0 Karma

Peter
Path Finder

That works to get the current average for the timeframe, but I need to compare it to the most recent day's count to know if I need to generate an alert. So if I take the average of the last 8 days (earliest=-8d@d latest=-2d@d) I need to compare that average to the DC from earliest=-1d@d so that I can determine the diff from normal.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I think you just need to either "eval" the count into another field name: ... | stats dc(cI) as ct | stats avg(ct)

0 Karma

Peter
Path Finder

Thanks - I'll look into those. The problem I hit with most of these commands is that I am trying to apply them to distinct_count(cID), rather than take the average or trendline of the cID values themselves. cID are unique identifiers, so they have no numeric meaning.

0 Karma
Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! &#x1f44f; Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...