Developing for Splunk Platform

problem getting results

sarit_s
Communicator

Hello,

I have a query that return results if im running it for 1 hour but if im trying to run the query for more than 1 our it returns no result.. 

index=clientlogs sourcetype=clientlogs Mode=Real ApplicationIdentifier="*" "orders-for-open" (Action="OpenPositionRequest" AND Level=Info) 

| eval StartTime=strptime(ClientDateTime,"%Y-%m-%dT%H:%M:%S.%3N") 
| rename Request_Id AS RequestId
| stats min(StartTime) as StartTime min(_time) AS _time BY RequestId

| join RequestId
    [ search index=clientlogs sourcetype=clientlogs Mode=Real ApplicationIdentifier="*" Message="Create OrderForOpen" 
      | rename OrderID AS PushEventData_Position_OrderID ]
      
      | join PushEventData_Position_OrderID
      [ search index=clientlogs sourcetype=clientlogs Mode=Real ApplicationIdentifier="*" (Message="Position.Open" AND (PushEventData_Position_OrderType=17 OR PushEventData_Position_OrderType=18)) 
         | eval finishTime=strptime(ClientDateTime,"%Y-%m-%dT%H:%M:%S.%3N") 
         | stats min(finishTime) as finishTime min(_time) AS _time BY PushEventData_Position_OrderID ] 

| eval Latency=finishTime-StartTime 
| where Latency>0 
| timechart avg(Latency) span=1m
Labels (2)
0 Karma

johnhua
Builder

Here's a shot at it. Could have typos

 

index=clientlogs sourcetype=clientlogs Mode=Real ((OpenPositionRequest AND orders-for-open) OR ("Create OrderForOpen") OR ("PushEventData_Position_OrderType" AND (":17" OR ":18")))
| eval _time=strptime(ClientDateTime,"%Y-%m-%dT%H:%M:%S.%3N")
| rename Request_Id AS request_id
| eval event_type=CASE(searchmatch("OpenPositionRequest"), "event_start", searchmatch("PushEventData_Position_OrderType"), "event_end")
| transaction request_id startswith=(event_type="event_start") endswith=(event_type="event_end") maxspan=30m  maxopentxn=500000
| eval start_time=strftime(_time, "%Y-%m-%d %H:%M:%S")
| eval end_time=strftime(_time+duration, "%Y-%m-%d %H:%M:%S")
| eval duration_sec=duration
| eval duration_min=ROUND(duration/60, 2)
| table start_time end_time duration_sec duration_min request_id 

 

0 Karma

johnhua
Builder

Your search could be easily simplified with stats or transactions. Also the filtering can be significantly improved.

Could you provide 3 examples of the logs that would be used to complete a transaction? E.g.

  • OpenPositionRequest
  • Create OrderForOpen
  • Position.Open(PushEventData_Position_OrderType)

 

 

0 Karma

sarit_s
Communicator
Message":"Create OrderForOpen"
"Action":"OpenPositionRequest"
"PushEventData_Position_OrderType":17
0 Karma

johnhua
Builder

That helps somewhat. Could you post the entire _raw events?

0 Karma

sarit_s
Communicator

im not sure i can .. i think i will have issue with security

0 Karma

johnhua
Builder

You can always redact the sensitive part or send it to me by PM.

The Request_Id field is found on all in scope events?

For the last event, you are looking specifically for PushEventData_Position_OrderType=17 OR 18?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

What does your job inspector say? For example, are your subsearches being truncated, which may mean your joins find no matches.

Have you tried reworking the search so you don't need to use joins at all? (Joins are usually best avoided if possible due to the restrictions on subsearches.)

0 Karma

sarit_s
Communicator

if im getting results for 1 hour (no matter what time) so the joins returns results 

the problem is it is not returning results for more than 1 our

if you think there is a way to run it without join i will be more than happy to see

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Yes, you said that, but what does the job inspector say happened when the failing search ran compared to the successful search?

To remove the joins, you could try something like a single primary search 

index=clientlogs sourcetype=clientlogs Mode=Real ApplicationIdentifier="*"

Then evaluate various fields depending on what events you have, for example:

| eval StartTime=if(searchmatch("\\\"orders-for-open\\\" (Action=\\\"OpenPositionRequest\\\" AND Level=Info)"),strptime(ClientDateTime,"%Y-%m-%dT%H:%M:%S.%3N"),null())

 and

| eval finishTime=if(searchmatch("(Message=\\\"Position.Open\\\" AND (PushEventData_Position_OrderType=17 OR PushEventData_Position_OrderType=18))"),strptime(ClientDateTime,"%Y-%m-%dT%H:%M:%S.%3N"),null())

Then gather the fields by RequestId

| stats min(StartTime) as StartTime min(finishTime) as finishTime min(_time) AS _time BY RequestId

 This may not be exactly what you need, since it isn't clear exactly what your search is doing or what your events actually look like, but the gist of it is hopefully clear.

0 Karma

sarit_s
Communicator

this is what im getting while running for more than an hour. (im not getting this when running for 1 hour)

sarit_s_0-1643202656989.png

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Did you notice the message in bold about the results being truncated at 50000? This is why the longer time periods fail - the join(s) have truncated results which probably means the events you were attempting to join with have no matches, so the overall search returns no results.

0 Karma

sarit_s
Communicator

what im trying to do is to join 3 searches by unique field and i couldn't find a way to do it without join

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Can you share some anonymised sample events from your 3 searches? Preferably use a code block </> when posting them to prevent formatting corruption

0 Karma

sarit_s
Communicator

what i found is that this part of your query does not return any results :

| eval StartTime=if(searchmatch("\\\"orders-for-open\\\" (Action=\\\"OpenPositionRequest\\\" AND Level=Info)"),strptime(ClientDateTime,"%Y-%m-%dT%H:%M:%S.%3N"),null())
| eval finishTime=if(searchmatch("(Message=\\\"Trading.Position.Open\\\" AND (PushEventData_Position_OrderType=17 OR PushEventData_Position_OrderType=18))"),strptime(ClientDateTime,"%Y-%m-%dT%H:%M:%S.%3N"),null())
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Can you post some events that it should have matched with?

0 Karma

sarit_s
Communicator

this is the first event and we are calulating this field between the first event and the third

ClientDateTime":"2022-01-25T18:21:13.964Z"
Request_Id":"ccb245e4-5c41-4e70-92a0-60ce0d435bff"

the requestid is the common field i want to use, this is the field that combine all the 3 searches together

does it helps ?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Not really - apart from it looks like it might be JSON, so perhaps using spath to extract the fields might make it easier.

If you don't want to provide sample events, it makes it difficult for us to develop sample queries to help you work out what may be the issue, so you will have to try and figure it out yourself.

0 Karma

sarit_s
Communicator

i took the query you suggested and add some other part but it is not returning any results (also for 1 hour)

this part returns results :

index=clientlogs sourcetype=clientlogs Mode=Real ApplicationIdentifier="*" "orders-for-open" (Action="OpenPositionRequest" AND Level=Info)
| eval StartTime=if(searchmatch("\\\"orders-for-open\\\" (Action=\\\"OpenPositionRequest\\\" AND Level=Info)"),strptime(ClientDateTime,"%Y-%m-%dT%H:%M:%S.%3N"),null())
| eval finishTime=if(searchmatch("(Message=\\\"Trading.Position.Open\\\" AND (PushEventData_Position_OrderType=17 OR PushEventData_Position_OrderType=18))"),strptime(ClientDateTime,"%Y-%m-%dT%H:%M:%S.%3N"),null())
| rename Request_Id AS RequestId
| stats min(StartTime) as StartTime min(finishTime) as finishTime min(_time) AS _time BY RequestId

 

but when im adding this part there are no results :

| eval Latency=finishTime-StartTime 
| where Latency>0 
| timechart avg(Latency) span=1m
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

From the first search (without the where command) do the results look correct i.e. has the stats command gathered the correct information from the related events?

0 Karma

sarit_s
Communicator

from high level check looks like it does

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...