Query to calculate the number of requests from the source

[aditsss]

Hi Everyone,

I have one requirement.

Below are my logs:

2020-09-30T05:03:29.304446Z app_name=api environment=e1c.a.b.controller.FileController : RID:22abe6c4-6eaf-4d47-8c4a-79b2594ea612-of1-team_g  A_EL_1100:   EVENT RECEIVED FROM SOURCE

2020-09-30T05:00:17.765656Z app_name=api environment=e1c.a.b.controller.FileController : RID:b0d5b62f-080f-4292-a2d1-4991123eecce-of1-team_f  A_EL:ARC_1100:  EVENT RECEIVED FROM SOURCE

In the above logs I have field RID(RequestID) like this which is not extracted:

RID:22abe6c4-6eaf-4d47-8c4a-79b2594ea612-of1-team_g

RID:b0d5b62f-080f-4292-a2d1-4991123eecce-of1-team_f

what I want is I want to display the RequestId from a particular source. Like for below RID source is of1-teamg and RequestId is 2abe6c4-6eaf-4d47-8c4a-79b2594ea612.Source is appended at the end of RID.

RID:22abe6c4-6eaf-4d47-8c4a-79b2594ea612

of1-team_g(Source)

For 2nd RID b0d5b62f-080f-4292-a2d1-4991123eecce-of1-team_f. The source is of1-team_f and RequestId is b0d5b62f-080f-4292-a2d1-4991123eecce.

There are multiple Sources and requestId's.

I want to display the number of requestId received from a particular source.

I only want to display the number of requestId received from a particular source for this pattern "EVENT RECEIVED FROM SOURCE".

Like for below example:

2020-09-30T05:03:29.304446Z app_name=api environment=e1c.a.b.controller.FileController : RID:22abe6c4-6eaf-4d47-8c4a-79b2594ea612-of1-team_g  A_EL_1100:   EVENT RECEIVED FROM SOURCE

RID-22abe6c4-6eaf-4d47-8c4a-79b2594ea612

Source - of1-team_g

Pattern - EVENT RECEIVED FROM SOURCE

Nothing is extracted as of now.

Can some one guide me with the search query Number of RequestId Received from source for particular pattern.

[ITWhisperer]

Assuming sources all are of the same pattern

-- your search including filter for "EVENT RECEIVED FROM SOURCE"
| rex "RID:(?<RID>[\w-]+)-(?<source>\w+-\w+)"
| stats count(RID) as count by source

[aditsss]

@ITWhisperer 

I don't have filter for pattern  "ARC EVENT RECEIVED FROM SOURCE" How can I extract them in search query .Below are the Logs

2020-09-30T05:03:29.304428046Z app_name=abc environment=e1 ns=abc pod_container=api pod_name=api-deployment-20-lmkq6 message=2020-09-29 22:03:29.303 INFO [service,a79e96c3ef2c10c8,a79e96c3ef2c10c8,true] 1 --- [or-http-epoll-2] c.a.b.arc.controller.FileController : RID:22abe6c4-6eaf-4d47-8c4a-79b2594ea612-of1-team_g ARC_EL:ARC_1100: ARC EVENT RECEIVED FROM SOURCE

Also I want to see the RID's. Currently its coming like this:

My search query:

index=ABC ns=XYZ app_name=GHJ | rex "RID:(?<RID>[\w-]+)-(?<source>\w+-\w+)"
| stats count(RID) as count by source|fields source RID count

This is not the source"/var/log/td-agent/tmp/containers.log" dont know why it is coming.

source RID count

/var/log/td-agent/tmp/containers.log		0
of1-team_c		3
of1-team_d		12
of1-team_f		16
of1-team_g		12
team_a-dmrupload		9
team_b-ivurtupload		7
team_e-dmrupload		11

[ITWhisperer]

Sorry my poor choice of fieldname

index=ABC ns=XYZ app_name=GHJ "ARC EVENT RECEIVED FROM SOURCE" 
| rex "RID:(?<RID>[\w-]+)-(?<sourceagent>\w+-\w+)"
| stats count(RID) as count by sourceagent|fields sourceagent RID count

By filter I meant select only those events with  "ARC EVENT RECEIVED FROM SOURCE"  in 

[aditsss]

@ITWhisperer 

Thanks @ITWhisperer  for your guidance.

Yes its showing the correct count. 

There are some logs which contain the source with different pattern like:

RID:5b23febe-1817-405d-8e7f-c4388feb9fbc-of1-team_b-ivurtupload 100: ARC EVENT RECEIVED FROM SOURCE

So with the regex its only showing team_b-ivurtupload I want it should extract from here of1-team_b-ivurtupload

For below logs the regex you give is working completely fine . I am able to get the source as of1-team_g that is what I want

RID:22abe6c4-6eaf-4d47-8c4a-79b2594ea612-of1-team_g ARC_EL:ARC_1100: ARC EVENT RECEIVED FROM SOURCE

I have mix logs . Can you provide me regex that will work for both?

Also I not able to see RID my data is coming like this . RID is coming as blank.

Can you guide me on this.

index=ABC ns=XYZ app_name=api "ARC EVENT RECEIVED FROM SOURCE"
| rex "RID:(?<RID>[\w-]+)-(?<sourceagent>\w+-\w+)"
| stats count(RID) as count by sourceagent| rename sourceagent as "Source"|fields Source count

sourceagent RID count

of1-team_c		1
of1-team_d		4
of1-team_f		2
of1-team_g		2
team_a-dmrupload		1
team_b-ivurtupload		1
team_e-dmrupload		1

[Nisha18789]

Hi @aditsss , please try this

index=ABC ns=XYZ app_name=api "ARC EVENT RECEIVED FROM SOURCE"
| rex "RID:(?<RID>(\w+-){4}\w+)-(?<sourceagent>\w+-\w+)"
| stats count(RID) as count by sourceagent| rename sourceagent as "Source"|fields Source count

[aditsss]

@ITWhisperer @Nisha18789 

Thank you so much It works.

[ITWhisperer · Accepted solution]

Which parts of the pattern are stable? Does the "source" always start with "of1-"? Does RID always follow the same pattern i.e. <8 letters/numbers>-<4 letters/numbers>-<4 letters/numbers>-<12 letters/numbers>?

index=ABC ns=XYZ app_name=api "ARC EVENT RECEIVED FROM SOURCE"
| rex "RID:(?<RID>\w{8}-\w{4}-\w{4}-\w{12})-(?<sourceagent>\S+)"
| stats count(RID) as count by sourceagent| rename sourceagent as "Source"|fields Source count

RID doesn't show up because you just wanted to count them - you could potentially end up with a lot. If you want all the RID values as well

index=ABC ns=XYZ app_name=api "ARC EVENT RECEIVED FROM SOURCE"
| rex "RID:(?<RID>\w{8}-\w{4}-\w{4}-\w{12})-(?<sourceagent>\S+)"
| stats count(RID) as count, values(RID) as RID by sourceagent| rename sourceagent as "Source"|fields Source RID count