topic Re: Need help using Tstats getting count of a string in raw logs in Splunk Search

Need help using Tstats getting count of a string in raw logs

aamer86 — Wed, 27 Nov 2019 12:10:20 GMT

I want to show the count of logs where a string appeared

I have a string and need to know how many times it appears in logs

Re: Need help using Tstats getting count of a string in raw logs

niketn — Wed, 27 Nov 2019 12:24:04 GMT

@aamer86 if the field/fields containing the string you want to count are
(1) indexed extracted or
(2) has accelerated Data Model created then only you can use the tstats command.

Otherwise you will have to perform regular index search.

index=<yourIndexName> "<yourString>"
| stats count as "No. of occurrences"

Please provide more details for the community to assist you better.

Re: Need help using Tstats getting count of a string in raw logs

aamer86 — Wed, 27 Nov 2019 12:54:31 GMT

the string is indexed but i need to get numbers over long period so i want to use tstats but the index is not in any Data Model

I remember once I saw tstats used with raw but cant remember the context

Re: Need help using Tstats getting count of a string in raw logs

arjunpkishore5 — Wed, 27 Nov 2019 13:15:42 GMT

if your data has indexed extractions, then you can use tstats as follows

|tstats count as total where index=<your index> AND <your field>=<your value>

by default, metadata fields such as host, source, sourcetype, and _time are indexed. unless <your field> is defined as an indexed extraction, the tstats won't work.

The other alternative is to build a datamodel on your data to be able to use tstats. if you have a datamodel defined, your tstats would be like this.

| tstats count as total from datamodel=< your datamodel name> where nodename=<your node name> AND <your node name>.<your field>=<your value>

If you have neither

index=<your index> <your field filter>
| stats count as total

Re: Need help using Tstats getting count of a string in raw logs

aamer86 — Wed, 27 Nov 2019 13:22:09 GMT

Thanks @arjunpkishore5 but this doesnt get me what I want

So the field is not from the indexed extraction fields i.e. it is not hostname, index, sourcetype or sourse.

I dont want to create a DataModel

I want to use tstats as it is faster

Re: Need help using Tstats getting count of a string in raw logs

arjunpkishore5 — Wed, 27 Nov 2019 15:05:41 GMT

As performant as tstats is, it cannot be used without an indexed field extractions or a datamodel. If you must use tstats, have the fields added to your indexed extractions. This however has performance impact and increased disk usage during indexing. Or, create a datamodel.

Re: Need help using Tstats getting count of a string in raw logs

woodcock — Wed, 27 Nov 2019 18:00:24 GMT

Like this:

| tstats count WHERE index="foo" AND sourcetype="bar" AND TERM(SearchStringHere)

Re: Need help using Tstats getting count of a string in raw logs

woodcock — Wed, 27 Nov 2019 21:41:54 GMT

This is not exactly true; it can also leverage the terms in the tsidx file. See my answer.

Re: Need help using Tstats getting count of a string in raw logs

aamer86 — Thu, 28 Nov 2019 09:54:45 GMT

Thanks @woodcock

This is exactly what I was looking for

Re: Need help using Tstats getting count of a string in raw logs

landen99 — Mon, 02 Dec 2019 13:14:57 GMT

It won't work.
First, the string has to follow segmentation rules.
Second, you only get a count of the events containing the string as presented in segmentation form. If the string appears multiple times in an event, you won't see that.

Re: Need help using Tstats getting count of a string in raw logs

landen99 — Mon, 02 Dec 2019 13:19:25 GMT

It won't work with tstats, but rex and mvcount will work.

For the tstats to work, first the string has to follow segmentation rules.
Second, you only get a count of the events containing the string as presented in segmentation form. If the string appears multiple times in an event, you won't see that.

The best you can get is a count of the number of events containing the string if it follows the segmentation rules or it's contained in an indexed field. Any string with major segment breakers in it will fail in tstats TERM.

| tstats count WHERE index=foo sourcetype=bar TERM(mySegment) TERM(mySegment2)

Also, if you ever consider using NOT with tstats, you'll also need to consider the effects of the following Splunk bug: https://answers.splunk.com/answers/787657/not-term-removes-results.html?childToView=787658#answer-787658

The better solution is to extract and count each string with rex:

index=<yourIndexName> "yourString"
| rex max_match=0 "(?<yourStringName>yourString)" 
| eval str_cnt = mvcount(yourStringName)
| stats sum(str_cnt) AS "No. of occurrences"

Re: Need help using Tstats getting count of a string in raw logs

woodcock — Mon, 02 Dec 2019 15:25:22 GMT

Agreed, it is a somewhat limited solution but it is the best that can be done.

Re: Need help using Tstats getting count of a string in raw logs

aamer86 — Mon, 30 Dec 2019 15:19:02 GMT

Thanks @landen99

does that mean if im looking for a string like this

"eventName": "DeleteBucketPolicy

it wont work due to space and double qoute ?

Re: Need help using Tstats getting count of a string in raw logs

landen99 — Mon, 30 Dec 2019 17:13:14 GMT

TERM will not work with that explicit string, but you can search for the combination of the two substrings: eventname AND deletebucketpolicy (case insensitive). The odds of false positive event matches on those two strings surely must be extremely low to non-existent. A much slower way to do it is just to extract the string with rex multiple times and then count the number of extractions (in the event that an event may contain the string more than once and you want to count every instance).

Re: Need help using Tstats getting count of a string in raw logs

woodcock — Mon, 30 Dec 2019 17:24:23 GMT

Yes, there is no way to use TERM with a string with double-quotes.

Re: Need help using Tstats getting count of a string in raw logs

landen99 — Mon, 30 Dec 2019 17:27:22 GMT

tstats only counts events, not string occurrences.