Splunk Search

How to use rex to create a field and using lookup to inject the data in the same field?

mikeyty07
Communicator

I am using rex field to extract the field name and then inject the data so I can get only the desired fields but not able to do so.

My Access logs: 
server
 - - [date& time] "GET /google/page1/page1a/633243463476/googlep1 HTTP/1.1200 350 85

My search query:
<query>
| rex field_=(?<SRC>\d+\.\d+\.\d+\.\d+).+\]\s\"(?<http_method>\w+)\s(?<serviceName>/[^/]+)(?<uri_path>[^?\s]+)\s(?<uri_query>\S+)\"\s(?<statusCode>\d+)\s(?<body_size>\d+)\s\s(?<response_time>\d+)

My Search query with lookup
<query>
| rex field=_raw "(?<SRC>\d+\.\d+\.\d+\.\d+).+\]\s\"(?<http_method>\w+)\s(?<serviceName>/[^/]+)(?<uri_path>[^?\s]+)\s(?<uri_query>\S+)\"\s(?<statusCode>\d+)\s(?<body_size>\d+)\s\s(?<response_time>\d+)"

|search serviceName="/google"
| lookup abc.csv uri_path OUTPUT serviceName apiName | search searviceName=* operationName=*

I am using above query to lookup from csv file but I get all the api has same count and not able to get the stats or logs for only particular.

Is there a way to match this and produce result with both uri_path and api_name? can anyone please help me on this?
Eg: csv file looks like this and i am trying to match apiName and uri_path so the logs are getting properly.

serviceName uri_path http_method apiName
/google /page1/page1a/*/googlep1 post postusingRRR
/google /page1/page1a/sada/*/googlep1 get getusingep2
/google /pag5/ggg/*/ooopp/ggplr delete deleteusing
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust
To support wildcards in a lookup, you must create a lookup definition ("abc", for instance) that points to abc.csv. The definition must use the advanced setting "WILDCARD(uri_path)" in the Match type field.

Go to Settings->Lookups and click "Add new" next to "Lookup definitions".  Fill in the form, selecting 'abc.csv' as the lookup file.  Check the "Advanced options" box then fill in the "Match type" box as shown.  Click Save.

richgalloway_0-1671309159659.png

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The regex does not match the sample log entry.  The regex is looking for an IP address, but none exist in the log.

Additionally, the slashes in the regex must be escaped.  Plus Splunk SPL requires triple-escapes with embedded quotation marks.

This regex works with the sample log entry, except it doesn't extract the SRC field.

 

\]\s\"(?<http_method>\w+)\s(?<serviceName>\/[^\/]+)(?<uri_path>\S+)\s(?<uri_query>\S+)\\\"\s(?<statusCode>\d+)\s(?<body_size>\d+)\s(?<response_time>\d+)

 

Are the asterisks in the uri_path field of the lookup wildcards or literal characters?  Splunk will treat them as literal '*' characters and will fail to find a match.  To support wildcards in a lookup, you must create a lookup definition ("abc", for instance) that points to abc.csv.  The definition must use the advanced setting "WILDCARD(uri_path)" in the Match type field.  Then use 

 

| lookup abc uri_path OUTPUT serviceName apiName 

 

in the query.

There is no 'operationName' field in the sample data so search operationName=* will cause the query to return no results.

---
If this reply helps you, Karma would be appreciated.

mikeyty07
Communicator

The rex search query is good which I can get the fields. Its the same field i am trying to match so it can produce all the results for one api.

this is an example from my csv file.

serviceNameuri_pathhttp_methodapiName
/google/page1/page1a/*/googlep1postpostusingRRR
/google/page1/page1a/sada/*/googlep1getgetusingep2
/google/pag5/ggg/*/ooopp/ggplrdeletedeleteusing

the /*/ value changes in logs so I'm using the apiName to connect. In real the api looks like this
/google/page1/page1a/{pageNumber}/googlep1


Only issue is with the lookup. How do i use the wildcard on this?

0 Karma

richgalloway
SplunkTrust
SplunkTrust
To support wildcards in a lookup, you must create a lookup definition ("abc", for instance) that points to abc.csv. The definition must use the advanced setting "WILDCARD(uri_path)" in the Match type field.

Go to Settings->Lookups and click "Add new" next to "Lookup definitions".  Fill in the form, selecting 'abc.csv' as the lookup file.  Check the "Advanced options" box then fill in the "Match type" box as shown.  Click Save.

richgalloway_0-1671309159659.png

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...