Extract Url from access log and count the no of times an API is hitted

[guptapesh]

This is the sample accessLog which is coming up in the Splunk ui.

1. {"timestamp":"2021-10-17T15:03:56,763Z","level":"INFO","thread":"reactor-http-epolpl-20","message":"method=GET, uri=/api/v1/hello1, status=200, duration=1, "logger":"reactor.netty.http.server.AccessLog"}

2. {"timestamp":"2021-10-17T15:03:56,763Z","level":"INFO","thread":"reactor-http-epolpl-20","message":"method=GET, uri=/api/v1/dummy1, status=200, duration=1, "logger":"reactor.netty.http.server.AccessLog"}

I want to extract the url and make a count for all the API's like how many times an API is hitted from the uri part(uri=/api/v1/dummy1).

I tried the below query, but it's not giving the desired result.

index=dummy OR index=dummy1  source=*dummy-service*  logger=reactor.netty.http.server.AccessLog | rex field=message "(?<url>uri.[\/api\/v1\/hello]+)" | chart count by url

Can someone help in this ? 

[WH]

Here is a regex you can use regardless if the date is included at the end of the uri.

(?<url>uri=[A-z0-9-\/]+)(?:,|(?=\/202\d))

[bowesmana]

Is your example JSON correct, i.e. there is no trailing quote mark for message, i.e. you have

"message":"method=GET, uri=/api/v1/hello1, status=200, duration=1, ...

therefore your JSON does not parse correctly, as the logger field is not extracted, so that may be why you get no results. If that is correct, then you could manually extract logger with the additional rex below, or fix the json

| rex field=message "uri=(?<url>uri.[^,]*)"
| rex field=_raw "logger\":\"(?<logger>[^\"]*)"
| search logger=reactor.netty.http.server.AccessLog 
| stats count by url

[guptapesh]

This is the proper formatted log :

{
"timestamp": "2021-10-17T15:03:56,763Z",
"level": "INFO",
"thread": "reactor-http-epolpl-20",
"message": "method=GET, uri=/api/v1/hello1, status=200, duration=1",
"logger": "reactor.netty.http.server.AccessLog"
}

Problem I am facing is like, I am having some GET requests in which values are coming in path variables like below :

1. /api/v1/dummy/D000/2021-10-14T11:00:00+00:00/2021-10-15T06:00:01+00:00
2. /api/v1/dummy/D023/2021-10-14T21:00:00+00:00/2021-10-19T06:00:25+00:00

Now for this particular url (/api/v1/dummy/{id}/{date1}/{date2}), it logs all the request with the above query,

I want a generic regex to collect all the request.

The above query works perfectly for the POST url, in which there are no incoming parameters like (/api/v1/add/user)

[bowesmana]

If the URL has extra parts to the URL path, as opposed to parameters, then would it be acceptable to define the url as the first 3 parts of the url, e.g.

| rex field=uri "(?<path>/[^/]*/[^/]*/[^/? ,]*)"

or you could do something like this

| rex field=uri mode=sed "s/\/\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[^\/?$]*//g"

to remove date components in the path from the uri - this example leaves your ID in place, but you could remove that too if you wanted.