How to create a search for matching next line?

[Thulasinathan_M]

Hi Splunk Experts,

I want to search for a word and then print the current matching line & the immediate next line. Kindly assist. Thanks in advance!!

Note: My events are Single-Line events.

[ITWhisperer]

Try something like this

| streamstats window=2 current=f last(_raw) as previous_raw
| regex previous_raw="match string"

[Thulasinathan_M]

Hi @ITWhisperer ,

Thanks, but this gives the same matching event, not adding the next immediate line.

[ITWhisperer]

True but the data is together.

Another approach could be this

| streamstats list(_raw) as raw first(_raw) as first_raw window=2
| regex first_raw="Location:BBB"
| fields - first_raw
| mvexpand raw
| rename raw as _raw

[Thulasinathan_M]

Thanks @ITWhisperer , tried but still it's displaying only the Event matched line.

index=main Location:BBB
| streamstats list(_raw) as raw first(_raw) as first_raw window=2
| regex first_raw="Location:BBB"
| fields - first_raw
| mvexpand raw
| rename raw as _raw

[ITWhisperer]

Here is a runanywhere example showing it working

[Thulasinathan_M]

Thanks @ITWhisperer & apologies for delayed response.

If I do a search with 'Location:BBB', I won't see the 'Timestamp: Address:BBBBB.....' in search event result. That's why streamstats not working in my case?

[ITWhisperer]

It may depend on when you are doing the search - please share your search which is failing to find the events you are looking for

[Thulasinathan_M]

This is my realtime scenario. I search for a Generic exceptions like NPE, ArrayOutOfBoundExceptions, etc. 


My Search: index=main .*NullPointerException.*

Search Results:
Event 1: From host:XXX source:xxx

16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException 

Event 2: From host:XXX source:xxx

java.lang.NullPointerException 

Event 3: From host:YYY source:yyy

java.lang.NullPointerException 

*********************************************
Results I'm expecting:

Event 2: From host:XXX source:xxx

16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException 
java.lang.NullPointerException 

Event 2: From host:XXX source:xxx

java.lang.NullPointerException 
at com.Khorn.TerrainControl.Generator.ObjectSpawner.populate(ObjectSpawner.java:128)

Event 3: From host:YYY source:yyy

java.lang.NullPointerException Caused
at app15a.action.CreateCustomerAction.execute(CreateCustomerAction.java:54)
at

**************************************************
Real Logs From host:XXX source:xxx

16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException 
java.lang.NullPointerException 
at com.Khorn.TerrainControl.Generator.ObjectSpawner.populate(ObjectSpawner.java:128) 
at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:178) 
at net.minecraft.server.Chunk.a(Chunk.java:820)
 at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:96)

 
Real Logs From host:YYY source:yyy

11:27:19,522 INFO  app15a.action.CreateCustomerAction - an exception occurred! ============================== 
java.lang.NullPointerException Caused
	at app15a.action.CreateCustomerAction.execute(CreateCustomerAction.java:54)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)

 
***********************************************

Tried alternative exhaustive search using below query but even this is bringing the same search result

index=main
| rex field=_raw "(?<Ex>(.*NullPointerException.*[\r\n].+[^\r\n]))"
| where len(Ex)!=0
| table Ex

[PickleRick]

OK. I know that it wasn't your question but these are not supposed to be separate events. These are simply subsequent lines of a single multi-line event and your source is not properly onboarded. You should fix your line breaking (and maybe timestamp recognition because I suspect it is broken as well). This will solve more problems than just "next line".

[ITWhisperer · Accepted solution]

Here is a runanywhere example with your logs.

| makeresults
| fields - _time
| eval _raw="16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException 
java.lang.NullPointerException 
at com.Khorn.TerrainControl.Generator.ObjectSpawner.populate(ObjectSpawner.java:128) 
at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:178) 
at net.minecraft.server.Chunk.a(Chunk.java:820)
 at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:96)
11:27:19,522 INFO  app15a.action.CreateCustomerAction - an exception occurred! ============================== 
java.lang.NullPointerException Caused
	at app15a.action.CreateCustomerAction.execute(CreateCustomerAction.java:54)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)"
| multikv noheader=t 
| fields _raw
| streamstats list(_raw) as raw first(_raw) as first_raw window=2
| regex first_raw="NullPointerException"
| where mvcount(raw) > 1
| fields - first_raw
| mvexpand raw
| rename raw as _raw

Things to note is the where command was added to remove the first event since it has no previous

Also, depending on what you are trying to see, the mvexpand (and rename) could be removed to show how the raw events are paired, so when NullPointerException appears on adjacent lines, it is clear why the line is repeated.

Finally, you might want to look at your ingestion so that all the lines from the same "event" are kept together in a single event e.g. breaking at the timestamp not every new line.

[Thulasinathan_M]

Very Much Thanks to you @ITWhisperer. It worked perfectly, when I didn't add 'NullPointerException' in my main search. But the Events it searched against 2 mins window was (23,976,134) which took lot of time. If I expand the window to 15 mins. The search got aut-cancelled. 😞 Is there any other work-arounds I could do in this situation.

[ITWhisperer]

Summary indexes are a good way of extracting useful information so that subsequent searches are faster, for example, you could run a report every minute to find these occurrences and put them in a summary index. You could then use the summary index to reframe the time search on your main index if you need more information from it.

The other option I already mentioned is to change your ingestion so that the multi-line faults are indexed as a single event rather than being spread across multiple events, which might improve your search time for these faults.

[Thulasinathan_M]

Hi @PickleRick, @richgalloway 
Yes, I agree with your statements. I don't want to relate events. Based on the search match events, I want that matched event along with next line. Which we usually see in the Show source.

Ex:
Timestamp: Location:AAA
Timestamp: Address:AAAAA.....

Timestamp: Location:BBB
Timestamp: Address:BBBBB.....

Timestamp: Location:CCC
Timestamp: Address:CCCCC.....


From the above example, If I search for specific Location='BBB', based on the matched event. I want to display below matched event and it's immediate next line which contains additional information I'm looking for, this can be achieved using regex/ rex patterns. But I'm missing something "rex field=_raw "[^\r|\n\r|\n](?<MatchedPatterns>.*[\r|\n\r|\n]?.*), it's not giving me the exact result I'm looking for, even if there are better solutions I would be happy to know about it.
Timestamp: Location:BBB
Timestamp: Address:BBBBB.....

[yuanliu]

This doesn't answer the essential question that @PickleRick and @richgalloway asked.  Let me ask in a different way: Are these illustrated lines in the same event or two separate events?

Timestamp: Location:AAA
Timestamp: Address:AAAAA.....

If they are in the same event, you need to explain the relevancy of the question because you automatically get the "second line".

If they are in separate events, as @PickleRick already explained, there is no such a thing as "next line".  You have to tell Splunk what else you want.  Is it the next increment in _time?  Is it the next largest Timestamp (which can be calculated into numeric time)?  Or something totally different? (It is important to realize that whether you want to relate events or not, you are relating the first event with the second event.  You have to tell Splunk what that relationship is.  Is there some other fields/keys that relate the second event to the first?)

In all likeliness, you seem to be asking for a transaction-like relationship, i.e., if I see a Location that meets my criteria, I want to see the next Address event.  In pseudo code, something like

| sort - _time
| transaction startswith=Location endswith=Address ``` add key fields if any ```
| where <whatever criteria on Location>

[Thulasinathan_M]

Hi @yuanliu, Generally I'm looking for what we see in the Events -> Event Actions -> Show Source. 
I can't do it by _time, because we've multiple host & source which could overlap. Unfortunately so such fields/ keys to identify it. 

[PickleRick]

This view shows you events in the (reverse? I don't remember) order as they are being returned by the search. But there is no guarantee that they will always be in the same order unless there is an explicit or implicit factor ordering them.

Results are by default (unless you sort them otherwise) returned in a reverse chronological order. But if you have two events with the exact same timestamp, I don't think there is ah officially defined and known mechanics for ordering them.

You have to remember that Splunk can consists of many servers and your two "consecutive" events can be indexed on different servers and returned in different order each time you run the search because on one occasion one of the indexers will be quicker to respond but another time it will be the other one.

That's why source onboarding is an important process. If those two lines are generally two parts of the same event happening on the source machine - it might mean that they should be ingested together as a single multiline event. Otherwise in order to reliably correlate multiple events regarding single something (operation, error, entity, whatever...) you need some form of identifier that appears in every of those events.

If you are absolutely sure that your event stream contains monotonic timestamps and multiple "packs of events" are not interleaved you can often use the time ordering. But as I said - multiple events indexed under the exactly same timestamp don't have a guaranteed search order.

I privately suspect that they might be returned in an index-time based "suborder" but I have no docs to support it.

[Thulasinathan_M]

Thanks @PickleRick. May be we have a something in place to display the preceeding & following lines of events which is always correct. I don't have admittance to see how this has been done and the team has the respective privilege are not much supportive. That's why trying to find a solution by search.

[richgalloway]

I second what @PickleRick said.  There isn't really a concept of "next" in Splunk.  Events return in (roughly) time order based on the search criteria.  Depending on how the data arrived in the indexers, events from multiple sources or hosts could be interleaved.  The immediately next event could be unrelated to the "word" event.

Tell us more the use case and perhaps we can come with another solution.

[PickleRick]

The question is a bit less straightforward that you could expect.

1. In general, Splunk works on one event at a time. You can't make it "relate" one event to another. You have to use splunk magic to either group events into aggregate rows of results or copy over data from one event to another.

2. Another thing is what you mean by "next".