Need help using Tstats getting count of a string in raw logs

[aamer86]

I want to show the count of logs where a string appeared

I have a string and need to know how many times it appears in logs

[landen99]

It won't work with tstats, but rex and mvcount will work.

For the tstats to work, first the string has to follow segmentation rules.
Second, you only get a count of the events containing the string as presented in segmentation form. If the string appears multiple times in an event, you won't see that.

The best you can get is a count of the number of events containing the string if it follows the segmentation rules or it's contained in an indexed field. Any string with major segment breakers in it will fail in tstats TERM.

| tstats count WHERE index=foo sourcetype=bar TERM(mySegment) TERM(mySegment2)

Also, if you ever consider using NOT with tstats, you'll also need to consider the effects of the following Splunk bug: https://answers.splunk.com/answers/787657/not-term-removes-results.html?childToView=787658#answer-78...

The better solution is to extract and count each string with rex:

index=<yourIndexName> "yourString"
| rex max_match=0 "(?<yourStringName>yourString)" 
| eval str_cnt = mvcount(yourStringName)
| stats sum(str_cnt) AS "No. of occurrences"

[woodcock · Accepted solution]

Like this:

| tstats count WHERE index="foo" AND sourcetype="bar" AND TERM(SearchStringHere)

[aamer86]

Thanks @woodcock

This is exactly what I was looking for

[landen99]

It won't work.
First, the string has to follow segmentation rules.
Second, you only get a count of the events containing the string as presented in segmentation form. If the string appears multiple times in an event, you won't see that.

[woodcock]

Agreed, it is a somewhat limited solution but it is the best that can be done.

[aamer86]

Thanks @landen99

does that mean if im looking for a string like this

"eventName": "DeleteBucketPolicy

it wont work due to space and double qoute ?

[woodcock]

Yes, there is no way to use TERM with a string with double-quotes.

[landen99]

TERM will not work with that explicit string, but you can search for the combination of the two substrings: eventname AND deletebucketpolicy (case insensitive). The odds of false positive event matches on those two strings surely must be extremely low to non-existent. A much slower way to do it is just to extract the string with rex multiple times and then count the number of extractions (in the event that an event may contain the string more than once and you want to count every instance).

[arjunpkishore5]

if your data has indexed extractions, then you can use tstats as follows

|tstats count as total where index=<your index> AND <your field>=<your value>

by default, metadata fields such as host, source, sourcetype, and _time are indexed. unless <your field> is defined as an indexed extraction, the tstats won't work.

The other alternative is to build a datamodel on your data to be able to use tstats. if you have a datamodel defined, your tstats would be like this.

| tstats count as total from datamodel=< your datamodel name> where nodename=<your node name> AND <your node name>.<your field>=<your value>

If you have neither

index=<your index> <your field filter>
| stats count as total

[aamer86]

Thanks @arjunpkishore5 but this doesnt get me what I want

So the field is not from the indexed extraction fields i.e. it is not hostname, index, sourcetype or sourse.

I dont want to create a DataModel

I want to use tstats as it is faster

[landen99]

tstats only counts events, not string occurrences.

[arjunpkishore5]

As performant as tstats is, it cannot be used without an indexed field extractions or a datamodel. If you must use tstats, have the fields added to your indexed extractions. This however has performance impact and increased disk usage during indexing. Or, create a datamodel.

[woodcock]

This is not exactly true; it can also leverage the terms in the tsidx file. See my answer.

[niketn]

@aamer86 if the field/fields containing the string you want to count are
(1) indexed extracted or
(2) has accelerated Data Model created then only you can use the tstats command.

Otherwise you will have to perform regular index search.

index=<yourIndexName> "<yourString>"
| stats count as "No. of occurrences"

Please provide more details for the community to assist you better.

[aamer86]

the string is indexed but i need to get numbers over long period so i want to use tstats but the index is not in any Data Model

I remember once I saw tstats used with raw but cant remember the context