All of our answers encourage exploration of JSON parsing, field extraction, and regular expression syntax (with bonus inconsistent escape sequence handling and multiple engines!) in Splunk, but I suspect @vineela just wants to skip ahead to statistical and/or time series analysis of response times. 😉 ... View more

If this isn't working for you, it would seem to suggest that the log field has not been extracted. In this example, representing your event, I have used spath to extract log from the _raw field before switching to with the _raw field to use kv | makeresults | eval _raw="{\"log\":\"21/Mar/2023:20:06:29 +1100 [defaultJmsListenerContainer-1] [correlationId=] ERROR au.com.commbank.pso.payments.pj.listener.util.LoggingUtil - Severity=ERROR, DateTimestamp=21/Mar/2023 20:06:29, ErrorCode=PJ_LISTENER_ERR_0003, ErrorMessage=PJ Listener connection to MQ has failed, MicroserviceName=PJ_LISTENER, ExceptionStackTrace=com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'NPAT01' with connection mode 'Client' and host name '10.31.39.168(1417)'.\\nCheck the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.\"}" | spath | rename _raw as temp, log as _raw | kv | rename temp as _raw | table ErrorCode ErrorMessage ... View more

Hi @vineela, good for you, see next time! Ciao and happy splunking Giuseppe P.S.: Karma Points are appreciated 😉 ... View more

Thanks Splunk champion..That was very clear...I Saw many videos for learning spath command and your solution for my question was crisp and clear...Easily understandable.Very helpful.Thanks a lot again. ... View more

Hi @vineela, good for you, see next time! Ciao and happy splunking Giuseppe P.S.: Karma Points are appreciated 😉 ... View more

While I usually advise not to use regex to extract from structured data, this particular log has many layers of escapement, so the data is JSON wrapped in key-value pair wrapped in JSON.  Still, for me retaining versatility and full dataset is worth trying.  Here goes. (Splunk should have already given you a field named "log".  The first spath is included in the following in case that is not the case.)   | spath ``` only if Splunk doesn't give you field "log" from the outer JSON ``` | rex field=log "entity={(?<entity>.*)}" | rename entity as _raw ``` if you need the original _raw, first rename it temp ``` | rex mode=sed "s/'([^']+)'=/\1=/g" | rex mode=sed "y/'/\"/" ``` reconstruct inner kv pairs ``` | kv | fields eventCode responseBody ``` remove this line to retain all fields ``` | eval responseBody="{". replace(responseBody, ".*{ (.+) }", "\1") . "}" | eval responseBody=replace(responseBody, "([^{\s]+):\s*([^\s}]+)", "\"\1\":\"\2\"") | rex field=responseBody mode=sed "y/ /,/" ``` reconstruct inter JSON ``` | spath input=responseBody   The sample data would give something like eventCode actionExpiryTime actionIdentification bulkIdentification bulkItemResult mandateIdentification mmsServiceBic portingIdentification priority reportIdentification resolutionRequestedBy trigger MDMT.MANDATE_CREATION_COMPLETED null 2916b2805bf211ed8ded12d15e0c927a null null 29168b705bf211ed8ded12d15e0c927a CTBAAUSNBKW null NORM null null MCRT ... View more

Hi @vineela, probably I am not able to describe the solution: if you need an alert to trigger when you have no events (count=0) you have to run your search without the condition count=0: index=a0_payservutil_generic_app_audit_prd MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT* | rex field=log "eventName=\"*(?<eventName>[^,\"\s]+)" | rex field=log "serviceName=\"*(?<serviceName>[^\"]+)" | search eventName="MANDATE_NOTIFICATION_RETRIEVAL.CALLOUT*" AND serviceName="Consume Notification" | stats count by eventName in this way you have an alert to trigger when you haven't results (results=0). You don't need to search the condition count=0 because No results is the condition when count=0 and you cannot display a message, because with the message you have one solution. If instead you want to dispay the result in a panel and you want to manage the condition "No results" displaying your message, you can use my first solution to display a message. You cannot have an intermediate solution, what do you need? an alert: the above solution, a dashboard panel: my first solution. Ciao. Giuseppe ... View more

Hi Whiser,        Thanks a lot...It is working for me ... View more

Hi @vineela, good for you, see next time! Ciao and happy splunking Giuseppe P.S.: Karma Points are appreciated 😉 ... View more

I am receiving error like this: Error in 'stats' command: The eval expression for dynamic field 'eval(if(newtime>=1),newtime,"")' is invalid. Error='The operator at ',newtime,""' is invalid.'. The search job has failed due to an error. You may be able view the job in the     ... View more

This is a bit confusing - are you trying to set severityType to FATAL if response code is one of the 400 codes or 500 and return code is a particular value? | eval severityType=if(httpResponseCode=400 OR httpResponseCode=401 OR httpResponseCode=403 OR httpResponseCode=404 OR (httpResponseCode=500 AND returnCode="APS.API.6544"), "FATAL", "NOT FATAL") ... View more

What do you currently have? Please share the code in a code block </> ... View more

Hi @vineela, at first, how do you have these logs? if they are in text files you have to ingest them using a Universal Forwarder on the server where the files are stored and use a folder monitoring stanza in inputs.conf. My hint is to see some video and read some documentation searching on Google and/or on YouTube "Splunk getting data in" like the following (these aren't the only ones available!) https://docs.splunk.com/Documentation/Splunk/latest/Data/Getstartedwithgettingdatain https://www.splunk.com/en_us/resources/videos/getting-data-in-to-splunk-enterprise-linux.html  When you'll have the logs inside Splunk you have to extract relevant fields and create you searches and statistics. Ciao. Giuseppe ... View more

Can you please provide me the query using rest. I have access to frontend .I have power user access for front end. ... View more

The  message is self-explanatory. The SMTP server you're trying to send your mail through needs authentication before letting you relay an email to the outsode world but you're not doing that. Configure your email action to provide proper authentication or even better - send through your local SMTP server and ask its admin to relay mails from splunk. ... View more

Hi @vineela, the best approach is that you could aggregate results in the base search and then display in each panel a subset of data. Only to explain:  if you want in a panel count the results of:      index=xyz  sourcetype="dtc:hsj" tcode="1324" in another count the results of:      index=xyz  sourcetype="dtc:hsj" tcode="1324"  OR tcode="234"  and in a third one all the others, you could put in the basesearch something like this: index=xyz sourcetype="dtc:hsj" | eval condition=3 | eval condition=case(tcode="1324","1",tcode="1324" OR tcode="234","2") | stats count BY condition then in each panel you will be able to filter results (condition=1 or condition=2 or condition=3 Ciao. Giuseppe ... View more

Assuming your data is consistent and have the same number of comma seperated fields. | rex "(?:[^,]*,){5}(?<event_type>[^,]*)\,(?<event_message>[^,]*),(?<event_code>[^,]*),(?<event_status>[^,]*)" | table _time event_type event_message event_code event_status ... View more

Someone from your team or company should've admin access. Its much easier to get the information through them.  ... View more

| rex "Caused by: (?<cause>([^:]+:){3}).*queue manager '(?<queuemanager>[^']+).*host name '(?<hostname>[^']+)" ... View more

Yes, That worked perfectly. Thanks for the quick response ... View more

Hi @vineela, if you used the best practice to use eventtype to address the indexes in each search it's easy because you have to change the index name only one time, otherwise, you have to change them one by one. A way ro be faster could be, if you have all your alerts in one (or few) app, could be to copy the savedsearches.conf file (the full file or at least only the interesting alerts) from the production environement to the non production environemnt (restarting Splunk!), but this solution doesn't solve the first problem. Ciao. Giuseppe ... View more

That helps me a lot. Thanks ... View more

--- search --- | stats latest(event) as last_event earliest(event) as first_event by transactionid event is the field you want to check and transactionid is your correlation id. stats will remove everything not mentioned from the pipeline, so if there are other field you are interested in, they need to be included in the stats command to e.g. latest(otherfield) as last_otherfield Note that the latest and earliest functions behave slightly differently in different releases of splunk, so you may need to sort by _time first or possibly use last and first functions instead or possibly both of these. ... View more