topic Re: How to create a search for matching next line? in Splunk Search

How to create a search for matching next line?

Thulasinathan_M — Thu, 03 Aug 2023 21:09:18 GMT

Hi Splunk Experts,

I want to search for a word and then print the current matching line & the immediate next line. Kindly assist. Thanks in advance!!

Note: My events are Single-Line events.

Re: How to create a search for matching next line?

PickleRick — Thu, 03 Aug 2023 21:44:29 GMT

The question is a bit less straightforward that you could expect.

1. In general, Splunk works on one event at a time. You can't make it "relate" one event to another. You have to use splunk magic to either group events into aggregate rows of results or copy over data from one event to another.

2. Another thing is what you mean by "next".

Re: How to create a search for matching next line?

richgalloway — Fri, 04 Aug 2023 00:22:18 GMT

I second what @PickleRick said. There isn't really a concept of "next" in Splunk. Events return in (roughly) time order based on the search criteria. Depending on how the data arrived in the indexers, events from multiple sources or hosts could be interleaved. The immediately next event could be unrelated to the "word" event.

Tell us more the use case and perhaps we can come with another solution.

Re: How to create a search for matching next line?

Thulasinathan_M — Fri, 04 Aug 2023 03:50:32 GMT

Hi @PickleRick, @richgalloway
Yes, I agree with your statements. I don't want to relate events. Based on the search match events, I want that matched event along with next line. Which we usually see in the Show source.

Ex:
Timestamp: Location:AAA
Timestamp: Address:AAAAA.....

Timestamp: Location:BBB
Timestamp: Address:BBBBB.....

Timestamp: Location:CCC
Timestamp: Address:CCCCC.....

From the above example, If I search for specific Location='BBB', based on the matched event. I want to display below matched event and it's immediate next line which contains additional information I'm looking for, this can be achieved using regex/ rex patterns. But I'm missing something "rex field=_raw "[^\r|\n\r|\n](?<MatchedPatterns>.*[\r|\n\r|\n]?.*), it's not giving me the exact result I'm looking for, even if there are better solutions I would be happy to know about it.
Timestamp: Location:BBB
Timestamp: Address:BBBBB.....

Re: How to create a search for matching next line?

yuanliu — Fri, 04 Aug 2023 07:29:31 GMT

This doesn't answer the essential question that @PickleRick and @richgalloway asked. Let me ask in a different way: Are these illustrated lines in the same event or two separate events?

Timestamp: Location:AAA
Timestamp: Address:AAAAA.....

If they are in the same event, you need to explain the relevancy of the question because you automatically get the "second line".

If they are in separate events, as @PickleRick already explained, there is no such a thing as "next line". You have to tell Splunk what else you want. Is it the next increment in _time? Is it the next largest Timestamp (which can be calculated into numeric time)? Or something totally different? (It is important to realize that whether you want to relate events or not, you are relating the first event with the second event. You have to tell Splunk what that relationship is. Is there some other fields/keys that relate the second event to the first?)

In all likeliness, you seem to be asking for a transaction-like relationship, i.e., if I see a Location that meets my criteria, I want to see the next Address event. In pseudo code, something like

| sort - _time | transaction startswith=Location endswith=Address ``` add key fields if any ``` | where <whatever criteria on Location>

Re: How to create a search for matching next line?

ITWhisperer — Fri, 04 Aug 2023 07:40:41 GMT

Try something like this

| streamstats window=2 current=f last(_raw) as previous_raw | regex previous_raw="match string"

Re: How to create a search for matching next line?

Thulasinathan_M — Fri, 04 Aug 2023 09:32:03 GMT

Hi @ITWhisperer ,

Thanks, but this gives the same matching event, not adding the next immediate line.

Re: How to create a search for matching next line?

Thulasinathan_M — Fri, 04 Aug 2023 09:37:40 GMT

Hi @yuanliu, Generally I'm looking for what we see in the Events -> Event Actions -> Show Source.
I can't do it by _time, because we've multiple host & source which could overlap. Unfortunately so such fields/ keys to identify it.

Re: How to create a search for matching next line?

ITWhisperer — Fri, 04 Aug 2023 09:40:26 GMT

True but the data is together.

Another approach could be this

Re: How to create a search for matching next line?

Thulasinathan_M — Fri, 04 Aug 2023 10:10:09 GMT

Thanks @ITWhisperer , tried but still it's displaying only the Event matched line.

Re: How to create a search for matching next line?

PickleRick — Fri, 04 Aug 2023 10:35:52 GMT

This view shows you events in the (reverse? I don't remember) order as they are being returned by the search. But there is no guarantee that they will always be in the same order unless there is an explicit or implicit factor ordering them.

Results are by default (unless you sort them otherwise) returned in a reverse chronological order. But if you have two events with the exact same timestamp, I don't think there is ah officially defined and known mechanics for ordering them.

You have to remember that Splunk can consists of many servers and your two "consecutive" events can be indexed on different servers and returned in different order each time you run the search because on one occasion one of the indexers will be quicker to respond but another time it will be the other one.

That's why source onboarding is an important process. If those two lines are generally two parts of the same event happening on the source machine - it might mean that they should be ingested together as a single multiline event. Otherwise in order to reliably correlate multiple events regarding single something (operation, error, entity, whatever...) you need some form of identifier that appears in every of those events.

If you are absolutely sure that your event stream contains monotonic timestamps and multiple "packs of events" are not interleaved you can often use the time ordering. But as I said - multiple events indexed under the exactly same timestamp don't have a guaranteed search order.

I privately suspect that they might be returned in an index-time based "suborder" but I have no docs to support it.

Re: How to create a search for matching next line?

ITWhisperer — Fri, 04 Aug 2023 11:32:19 GMT

Here is a runanywhere example showing it working

Re: How to create a search for matching next line?

Thulasinathan_M — Sat, 05 Aug 2023 08:48:15 GMT

Thanks @ITWhisperer & apologies for delayed response.

If I do a search with 'Location:BBB', I won't see the 'Timestamp: Address:BBBBB.....' in search event result. That's why streamstats not working in my case?

Re: How to create a search for matching next line?

Thulasinathan_M — Sat, 05 Aug 2023 08:55:33 GMT

Thanks @PickleRick. May be we have a something in place to display the preceeding & following lines of events which is always correct. I don't have admittance to see how this has been done and the team has the respective privilege are not much supportive. That's why trying to find a solution by search.

Re: How to create a search for matching next line?

ITWhisperer — Sat, 05 Aug 2023 10:00:02 GMT

It may depend on when you are doing the search - please share your search which is failing to find the events you are looking for

Re: How to create a search for matching next line?

Thulasinathan_M — Sat, 05 Aug 2023 10:28:33 GMT

This is my realtime scenario. I search for a Generic exceptions like NPE, ArrayOutOfBoundExceptions, etc.

My Search: index=main .*NullPointerException.*

Search Results:
Event 1: From host:XXX source:xxx

16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException

Event 2: From host:XXX source:xxx

java.lang.NullPointerException

Event 3: From host:YYY source:yyy

java.lang.NullPointerException

*********************************************
Results I'm expecting:

Event 2: From host:XXX source:xxx

16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException java.lang.NullPointerException

Event 2: From host:XXX source:xxx

java.lang.NullPointerException at com.Khorn.TerrainControl.Generator.ObjectSpawner.populate(ObjectSpawner.java:128)

Event 3: From host:YYY source:yyy

java.lang.NullPointerException Caused at app15a.action.CreateCustomerAction.execute(CreateCustomerAction.java:54) at

**************************************************
Real Logs From host:XXX source:xxx

16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException java.lang.NullPointerException at com.Khorn.TerrainControl.Generator.ObjectSpawner.populate(ObjectSpawner.java:128) at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:178) at net.minecraft.server.Chunk.a(Chunk.java:820) at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:96)

Real Logs From host:YYY source:yyy

11:27:19,522 INFO app15a.action.CreateCustomerAction - an exception occurred! ============================== java.lang.NullPointerException Caused at app15a.action.CreateCustomerAction.execute(CreateCustomerAction.java:54) at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)

***********************************************

Tried alternative exhaustive search using below query but even this is bringing the same search result

index=main | rex field=_raw "(?<Ex>(.*NullPointerException.*[\r\n].+[^\r\n]))" | where len(Ex)!=0 | table Ex

Re: How to create a search for matching next line?

ITWhisperer — Sat, 05 Aug 2023 11:07:44 GMT

Here is a runanywhere example with your logs.

| makeresults | fields - _time | eval _raw="16:19:33 [WARNING] Failed to handle packet: java.lang.NullPointerException java.lang.NullPointerException at com.Khorn.TerrainControl.Generator.ObjectSpawner.populate(ObjectSpawner.java:128) at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:178) at net.minecraft.server.Chunk.a(Chunk.java:820) at net.minecraft.server.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:96) 11:27:19,522 INFO app15a.action.CreateCustomerAction - an exception occurred! ============================== java.lang.NullPointerException Caused at app15a.action.CreateCustomerAction.execute(CreateCustomerAction.java:54) at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)" | multikv noheader=t | fields _raw | streamstats list(_raw) as raw first(_raw) as first_raw window=2 | regex first_raw="NullPointerException" | where mvcount(raw) > 1 | fields - first_raw | mvexpand raw | rename raw as _raw

Things to note is the where command was added to remove the first event since it has no previous

Also, depending on what you are trying to see, the mvexpand (and rename) could be removed to show how the raw events are paired, so when NullPointerException appears on adjacent lines, it is clear why the line is repeated.

Finally, you might want to look at your ingestion so that all the lines from the same "event" are kept together in a single event e.g. breaking at the timestamp not every new line.

Re: How to create a search for matching next line?

Thulasinathan_M — Sat, 05 Aug 2023 11:42:35 GMT

Very Much Thanks to you @ITWhisperer. It worked perfectly, when I didn't add 'NullPointerException' in my main search. But the Events it searched against 2 mins window was (23,976,134) which took lot of time. If I expand the window to 15 mins. The search got aut-cancelled. 😞 Is there any other work-arounds I could do in this situation.

Re: How to create a search for matching next line?

ITWhisperer — Sat, 05 Aug 2023 12:56:54 GMT

Summary indexes are a good way of extracting useful information so that subsequent searches are faster, for example, you could run a report every minute to find these occurrences and put them in a summary index. You could then use the summary index to reframe the time search on your main index if you need more information from it.

The other option I already mentioned is to change your ingestion so that the multi-line faults are indexed as a single event rather than being spread across multiple events, which might improve your search time for these faults.

Re: How to create a search for matching next line?

PickleRick — Sat, 05 Aug 2023 13:29:08 GMT

OK. I know that it wasn't your question but these are not supposed to be separate events. These are simply subsequent lines of a single multi-line event and your source is not properly onboarded. You should fix your line breaking (and maybe timestamp recognition because I suspect it is broken as well). This will solve more problems than just "next line".