extract and count the word occurring in a log

[BasicLearner]

I am new to splunk. The answer can help me learn more.

I have a message in log which looks something like 

k45ksp: k45kspProcessControlBuff task 1 (p_id: 2).

I need to just extract  k45kspProcessControlBuff from above message field  and count how many times it has occurred in a log.

[BasicLearner]

Hi PicleRick,

1. I have one common index for all the source matching my search.

2. The search term occurs more than once in my log and I get many such logs from the index in a day.

3. I need to count how many times the search term occurs throughout my  log. I use | dedup logid so that I am not counting the same log again.

Hope I have correctly answered your doubts. Plz let me know in case...

Thank You. 

[ITWhisperer]

Add "k45kspProcessControlBuff" to your search then stats count

[BasicLearner]

Hi  ITWhisperer,

Thank you for the reply. 

I think I need to reframe my question.

I have a sentence in a log which looks something like 

k45ksp: k45kspProcessControlBuff task 1 (p_id: 2).

In the log, the string "k45kspProcessControlBuff" may occur more than once and there can be more 10 such logs in a day.

I need to count/display how many times this string has occurred in each log in a day.

Using just stats count did not work.

[ITWhisperer]

What have you tried?

[BasicLearner]

Hi 

I  tried this search which gives me total number of occurrences in a day . 

index=* component=*  k45kspProcessControlBuff  earliest=-1d@d

| stats count

But I actually want to list/display/count how many times it occurred in one log  and also against the total logs in each day.

Thank You.

[ITWhisperer]

I am not sure what you mean by log but you could count by source which might amount to the same thing?

| stats count by source

[BasicLearner]

Thank you for your help and support.. 

Log here means log report. Every source has a log report. I want to count how many times this word has occurred in the log report.

Should I use rex command? I am not familiar with rex command.

[ITWhisperer]

It depends on what your events actually look like. Did the command I suggested give you what you wanted? If not, how was it different to what you were expecting?

[BasicLearner]

When I am using below search query

index=* component=*  k45kspProcessControlBuff  earliest=-1d@d

| stats count by source

It is displaying the result something like this 

source                          count

log1                                 1

log2                                 1

log3                                 1

In each log/source/event the string "k45kspProcessControlBuff" is occurring more than once.

I actually want to display how many times the string has occurred in each log/source/event along with the above.

Thank You.

[ITWhisperer]

| rex max_match=0 "(?<pcb>k45kspProcessControlBuff)"
| stats count by pcb source

[BasicLearner]

This gives similar result like the previous one with one extra column with the string name. 

pcb                                                              source         count

k45kspProcessControlBuff                log1                  1

k45kspProcessControlBuff                log2                  1

 I actually want  how many times the search term occurred throughout each log.

Thank You.

[BasicLearner]

Sorry, I am not sure of sharing events from the logs.

I can just tell that there is a message event as described in the previous mail which is 

k45ksp: k45kspProcessControlBuff task 1 (p_id: 2).

This message appears more than once in one log and I need to count how many times it appears in one log.

If I use | stats count by message . It again finds how many events I have with the search term. 

[ITWhisperer]

From the description you have given of the results you are getting, it sounds like you have one event in your logs.

If you have used max_match=0 on the rex command, this will create a multivalue field with all the matching occurrences, which the stats command will count.

If you are still getting a count of 1, you either only have one match to this string, or rex is only returning the first match (because you haven't specified max_match=0) or there is something curious about your log events.

If you can't share them, you will need to find another way to describe what your data looks like.

[BasicLearner]

Thank you ITWhisperer for your continuous efforts to guide me. May be I will have to try with couple of any other options .

[ITWhisperer]

Can you share a couple of events from your logs?

[PickleRick]

First things first - ouch! Searching throughout all your indexes is kinda heavy. Unless you have a very good reason you shouldn't normally do that.

But to the point.

What do your logs look like? Because the output shows clearly that you have just one index per source matching your search.

What's important here is that you get count of events matching your search. So if you have your search term (k45kspProcessControlBuff) multiple times in the event, it will get counted once as it is one event.

So the question is - what is really in your data. Because if you have - for example - badly onboarded logs so that it's not broken into events, you'll get just one event per source.

And the final question is whether you want to find how many events you have with the search term (which is nearly the simplest thing you can do in splunk; and you've been shown how to do that already, provided the data is properly onboarded) or find how many times the search term occures throughout your logs (assuming it can occur multiple times within one event and then it should be counted multiple times) - that's harder.