Splunk Search

Error event log format and search

gaurav_maniar
Builder

For any error Splunk gives a request id and link to search for that particular error details. In my, going to that Splunk search opens with following query and giving 0 results.

index=_internal host="abcd.local" source=*web_service.log log_level=ERROR requestid=58eaaa0d981062bf110

After changing the search query to index=_internal 58eaaa0d981062bf110, it gave proper events.
When I checked the events all the '=' were replaced by '%3D', so spunk was not able to search as per the previous query.
It may be a bug or some configuration problem.

Sample event,

127.0.0.1 - admin [10/Apr/2017:03:21:09.136 +0530] "GET /en-US/splunkd/__raw/services/messages?output_mode=json&sort_key=timeCreated_epochSecs&sort_dir=desc&count=1000&_=1491774307426 HTTP/1.1" 200 296 "http://localhost:8000/en-US/app/search/search?q=search%20index%3D_internal%20host%3D%22abcd.local%22%20source%3D*web_service.log%20log_level%3DERROR%20requestid%3D58eaaa0d981062bf110&sid=1491774308.3&display.page.search.mode=verbose&dispatch.sample_ratio=1&earliest=&latest=" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" - 159fb92019a0e06f68644ed17a46ddf6 1ms
0 Karma

jkat54
SplunkTrust
SplunkTrust

%3D is the "url encoded" equals sign. In the event you're looking at that shows the %3D's, they are expected because those are the web ui logs and that's the url you visited (encoded).

something else is your problem such as the field requestid doesn't exist as an extraction in the search app you are linked to, but it is an extracted field in the report/alert/search that generated the link to the results. try making the app that you created the alert/report/search in "global" so that it shares its extractions (and other knowledge objects) with all the other apps including the search app (aka search and reporting app).

0 Karma

gaurav_maniar
Builder

I know that its URL encoding. But the search query was generated by Splunk on error page. If Splunk supposed write that log in URL encoded format than why it is including the field and '=' sign in the search. This doesn't make sense. Splunk writing the event in some different format and searching for that event with some other way.

0 Karma

jkat54
SplunkTrust
SplunkTrust

if you run this search, does it return any results?

index=_internal requestid=*

0 Karma

gaurav_maniar
Builder

it returns the results, but only events for which requestid is defined as extracted filed. You can check the below events, there is no mention of requestid in it. I can not see that error events in this search. All events are from only splunk_web_service source.

2017-04-17 23:40:20,217 INFO    [58f5050c08105c098d0] view:1077 - PERF - viewType=fastpath viewTime=0.1408s templateTime=0.0168s

2017-04-17 23:40:20,200 INFO    [58f5050c08105c098d0] view:1059 - bypass module system fast path

2017-04-17 23:40:17,300 INFO    [58f5050947105c09d50] cached:77 - memoized decorator used on function <function getEntities at 0x10487fb18> with non hashable arguments
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...