Splunk Search

How to create a search for matching next line?

Thulasinathan_M
Communicator

Hi Splunk Experts,

I want to search for a word and then print the current matching line & the immediate next line. Kindly assist. Thanks in advance!!

 

Note: My events are Single-Line events.

Labels (2)
Tags (2)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

Here is a runanywhere example with your logs.

| makeresults
| fields - _time
| eval _raw="16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException 
java.lang.NullPointerException 
at com.Khorn.TerrainControl.Generator.ObjectSpawner.populate(ObjectSpawner.java:128) 
at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:178) 
at net.minecraft.server.Chunk.a(Chunk.java:820)
 at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:96)
11:27:19,522 INFO  app15a.action.CreateCustomerAction - an exception occurred! ============================== 
java.lang.NullPointerException Caused
	at app15a.action.CreateCustomerAction.execute(CreateCustomerAction.java:54)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)"
| multikv noheader=t 
| fields _raw
| streamstats list(_raw) as raw first(_raw) as first_raw window=2
| regex first_raw="NullPointerException"
| where mvcount(raw) > 1
| fields - first_raw
| mvexpand raw
| rename raw as _raw

Things to note is the where command was added to remove the first event since it has no previous

Also, depending on what you are trying to see, the mvexpand (and rename) could be removed to show how the raw events are paired, so when NullPointerException appears on adjacent lines, it is clear why the line is repeated.

Finally, you might want to look at your ingestion so that all the lines from the same "event" are kept together in a single event e.g. breaking at the timestamp not every new line.

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

Try something like this

| streamstats window=2 current=f last(_raw) as previous_raw
| regex previous_raw="match string"

Thulasinathan_M
Communicator

Hi @ITWhisperer ,

Thanks, but this gives the same matching event, not adding the next immediate line.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

True but the data is together.

Another approach could be this

| streamstats list(_raw) as raw first(_raw) as first_raw window=2
| regex first_raw="Location:BBB"
| fields - first_raw
| mvexpand raw
| rename raw as _raw

Thulasinathan_M
Communicator

Thanks @ITWhisperer , tried but still it's displaying only the Event matched line.

index=main Location:BBB
| streamstats list(_raw) as raw first(_raw) as first_raw window=2
| regex first_raw="Location:BBB"
| fields - first_raw
| mvexpand raw
| rename raw as _raw

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Here is a runanywhere example showing it working

ITWhisperer_0-1691148729593.png

 

Thulasinathan_M
Communicator

Thanks @ITWhisperer & apologies for delayed response.

If I do a search with 'Location:BBB', I won't see the 'Timestamp: Address:BBBBB.....' in search event result. That's why streamstats not working in my case?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

It may depend on when you are doing the search - please share your search which is failing to find the events you are looking for

0 Karma

Thulasinathan_M
Communicator

This is my realtime scenario. I search for a Generic exceptions like NPE, ArrayOutOfBoundExceptions, etc. 


My Search: index=main .*NullPointerException.*

Search Results:
Event 1: From host:XXX source:xxx

16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException 

Event 2: From host:XXX source:xxx

java.lang.NullPointerException 

Event 3: From host:YYY source:yyy

java.lang.NullPointerException 

 

*********************************************
Results I'm expecting:

Event 2: From host:XXX source:xxx

16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException 
java.lang.NullPointerException 

Event 2: From host:XXX source:xxx

java.lang.NullPointerException 
at com.Khorn.TerrainControl.Generator.ObjectSpawner.populate(ObjectSpawner.java:128)

Event 3: From host:YYY source:yyy

java.lang.NullPointerException Caused
at app15a.action.CreateCustomerAction.execute(CreateCustomerAction.java:54)
at


**************************************************
Real Logs From host:XXX source:xxx

16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException 
java.lang.NullPointerException 
at com.Khorn.TerrainControl.Generator.ObjectSpawner.populate(ObjectSpawner.java:128) 
at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:178) 
at net.minecraft.server.Chunk.a(Chunk.java:820)
 at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:96)

 
Real Logs From host:YYY source:yyy

11:27:19,522 INFO  app15a.action.CreateCustomerAction - an exception occurred! ============================== 
java.lang.NullPointerException Caused
	at app15a.action.CreateCustomerAction.execute(CreateCustomerAction.java:54)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)

 
***********************************************

Tried alternative exhaustive search using below query but even this is bringing the same search result

index=main
| rex field=_raw "(?<Ex>(.*NullPointerException.*[\r\n].+[^\r\n]))"
| where len(Ex)!=0
| table Ex

 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. I know that it wasn't your question but these are not supposed to be separate events. These are simply subsequent lines of a single multi-line event and your source is not properly onboarded. You should fix your line breaking (and maybe timestamp recognition because I suspect it is broken as well). This will solve more problems than just "next line".

ITWhisperer
SplunkTrust
SplunkTrust

Here is a runanywhere example with your logs.

| makeresults
| fields - _time
| eval _raw="16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException 
java.lang.NullPointerException 
at com.Khorn.TerrainControl.Generator.ObjectSpawner.populate(ObjectSpawner.java:128) 
at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:178) 
at net.minecraft.server.Chunk.a(Chunk.java:820)
 at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:96)
11:27:19,522 INFO  app15a.action.CreateCustomerAction - an exception occurred! ============================== 
java.lang.NullPointerException Caused
	at app15a.action.CreateCustomerAction.execute(CreateCustomerAction.java:54)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)"
| multikv noheader=t 
| fields _raw
| streamstats list(_raw) as raw first(_raw) as first_raw window=2
| regex first_raw="NullPointerException"
| where mvcount(raw) > 1
| fields - first_raw
| mvexpand raw
| rename raw as _raw

Things to note is the where command was added to remove the first event since it has no previous

Also, depending on what you are trying to see, the mvexpand (and rename) could be removed to show how the raw events are paired, so when NullPointerException appears on adjacent lines, it is clear why the line is repeated.

Finally, you might want to look at your ingestion so that all the lines from the same "event" are kept together in a single event e.g. breaking at the timestamp not every new line.

Thulasinathan_M
Communicator

Very Much Thanks to you @ITWhisperer. It worked perfectly, when I didn't add 'NullPointerException' in my main search. But the Events it searched against 2 mins window was (23,976,134) which took lot of time. If I expand the window to 15 mins. The search got aut-cancelled. 😞 Is there any other work-arounds I could do in this situation.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Summary indexes are a good way of extracting useful information so that subsequent searches are faster, for example, you could run a report every minute to find these occurrences and put them in a summary index. You could then use the summary index to reframe the time search on your main index if you need more information from it.

The other option I already mentioned is to change your ingestion so that the multi-line faults are indexed as a single event rather than being spread across multiple events, which might improve your search time for these faults.

Thulasinathan_M
Communicator

Hi @PickleRick@richgalloway 
Yes, I agree with your statements. I don't want to relate events. Based on the search match events, I want that matched event along with next line. Which we usually see in the Show source.

Ex:
Timestamp: Location:AAA
Timestamp: Address:AAAAA.....

Timestamp: Location:BBB
Timestamp: Address:BBBBB.....

Timestamp: Location:CCC
Timestamp: Address:CCCCC.....


From the above example, If I search for specific Location='BBB', based on the matched event. I want to display below matched event and it's immediate next line which contains additional information I'm looking for, this can be achieved using regex/ rex patterns. But I'm missing something "rex field=_raw "[^\r|\n\r|\n](?<MatchedPatterns>.*[\r|\n\r|\n]?.*), it's not giving me the exact result I'm looking for, even if there are better solutions I would be happy to know about it.
Timestamp: Location:BBB
Timestamp: Address:BBBBB.....

0 Karma

yuanliu
SplunkTrust
SplunkTrust

This doesn't answer the essential question that @PickleRick and @richgalloway asked.  Let me ask in a different way: Are these illustrated lines in the same event or two separate events?

Timestamp: Location:AAA
Timestamp: Address:AAAAA.....

If they are in the same event, you need to explain the relevancy of the question because you automatically get the "second line".

If they are in separate events, as @PickleRick already explained, there is no such a thing as "next line".  You have to tell Splunk what else you want.  Is it the next increment in _time?  Is it the next largest Timestamp (which can be calculated into numeric time)?  Or something totally different? (It is important to realize that whether you want to relate events or not, you are relating the first event with the second event.  You have to tell Splunk what that relationship is.  Is there some other fields/keys that relate the second event to the first?)

In all likeliness, you seem to be asking for a transaction-like relationship, i.e., if I see a Location that meets my criteria, I want to see the next Address event.  In pseudo code, something like

| sort - _time
| transaction startswith=Location endswith=Address ``` add key fields if any ```
| where <whatever criteria on Location>

 

Tags (1)

Thulasinathan_M
Communicator

Hi @yuanliu, Generally I'm looking for what we see in the Events -> Event Actions -> Show Source. 
I can't do it by _time, because we've multiple host & source which could overlap. Unfortunately so such fields/ keys to identify it. 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

This view shows you events in the (reverse? I don't remember) order as they are being returned by the search. But there is no guarantee that they will always be in the same order unless there is an explicit or implicit factor ordering them.

Results are by default (unless you sort them otherwise) returned in a reverse chronological order. But if you have two events with the exact same timestamp, I don't think there is ah officially defined and known mechanics for ordering them.

You have to remember that Splunk can consists of many servers and your two "consecutive" events can be indexed on different servers and returned in different order each time you run the search because on one occasion one of the indexers will be quicker to respond but another time it will be the other one.

That's why source onboarding is an important process. If those two lines are generally two parts of the same event happening on the source machine - it might mean that they should be ingested together as a single multiline event. Otherwise in order to reliably correlate multiple events regarding single something (operation, error, entity, whatever...) you need some form of identifier that appears in every of those events.

If you are absolutely sure that your event stream contains monotonic timestamps and multiple "packs of events" are not interleaved you can often use the time ordering. But as I said - multiple events indexed under the exactly same timestamp don't have a guaranteed search order.

I privately suspect that they might be returned in an index-time based "suborder" but I have no docs to support it.

Thulasinathan_M
Communicator

Thanks @PickleRick. May be we have a something in place to display the preceeding & following lines of events which is always correct. I don't have admittance to see how this has been done and the team has the respective privilege are not much supportive. That's why trying to find a solution by search.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I second what @PickleRick said.  There isn't really a concept of "next" in Splunk.  Events return in (roughly) time order based on the search criteria.  Depending on how the data arrived in the indexers, events from multiple sources or hosts could be interleaved.  The immediately next event could be unrelated to the "word" event.

Tell us more the use case and perhaps we can come with another solution.

---
If this reply helps you, Karma would be appreciated.

PickleRick
SplunkTrust
SplunkTrust

The question is a bit less straightforward that you could expect.

1. In general, Splunk works on one event at a time. You can't make it "relate" one event to another. You have to use splunk magic to either group events into aggregate rows of results or copy over data from one event to another.

2. Another thing is what you mean by "next".

Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...