topic Re: How to ignore a section of a log in a query? in Splunk Search

How to ignore a section of a log in a query?

shah_nishay — Fri, 25 Jul 2014 16:56:27 GMT

I am parsing a file and would like to skip a section of the same
Below is the log :

I am interested in processing "timing" value from this log but do not need any timing that comes between "Starting new thread FileReplayThread1" and "Thread FileReplayThread2 exiting"
For the above mentioned example , I am only interested in timing=2301,52,53 and 96

How can I add this in the query ?

I donot want to remove this all together from indexer as another report on the same server also uses these timings and will need all of these values.

Re: How to ignore a section of a log in a query?

somesoni2 — Fri, 25 Jul 2014 17:33:21 GMT

Assuming each line is an event and the field extraction is not done (if fields are already extracted, remove the rex portion from the search and use the corresponding field names), try this

your base search | rex "\|\s*(?<LogType>.*)\s*\|\s*(?<ThreadId>.*)\s*\|\s*(?<Message>.*)" | transaction ThreadId startswith="Starting new thread" endswith=" exiting" keeporphans=t | where linecount=1

Re: How to ignore a section of a log in a query?

martin_mueller — Fri, 25 Jul 2014 17:38:52 GMT

Assuming your pairs of starting/exiting are complete within the time range you can do something like this:

  base search yielding the events you posted
| streamstats count(eval(searchmatch("Starting new thread FileReplayThread1"))) as startCount
              count(eval(searchmatch("Thread FileReplayThread2 exiting"))) as endCount
| where startCount==endCount AND isnotnull(timing)
| further processing goes here

This keeps a running count of both messages and only keeps events with a timing field if the running counts are equal, ie each start has seen an end or vice versa.

Re: How to ignore a section of a log in a query?

shah_nishay — Fri, 25 Jul 2014 20:59:13 GMT

this doesnt work, it still pulls up all "timings" events

Re: How to ignore a section of a log in a query?

shah_nishay — Fri, 25 Jul 2014 21:01:08 GMT

I wont have the Start new Thread and Thread exiting messages on all day's , it occurs only on few days and for those days I want to ignore the "timing" values that comes between these 2 lines. On some other day's logs there will only be "timing" messages without the "Start new Thread" and "exiting" lines and I need to use these data as well

Re: How to ignore a section of a log in a query?

somesoni2 — Fri, 25 Jul 2014 21:12:24 GMT

Have you tried this??

Re: How to ignore a section of a log in a query?

martin_mueller — Fri, 25 Jul 2014 21:24:28 GMT

No start and exit messages won't be a problem. A start without an exit or an exit without a start might though.