Archive
Highlighted

How to detect two consecutive lines with same pattern

Explorer

I wanted to detect the pattern with two consecutive lines with Received x messages , In ideal scenario it should be Received followed by Updated tried streamer but no luck

8/22/17
5:30:32.542 AM

2017-08-22 01:30:32.542 INFO 3815 --- [ThreadPoolService-safe-1-thread-6256] com.adobe.ids.kafka.KafkaRestFacade : Updated consumed offset to value '56622449' for consumer 'mpshistorychannelconsumer3prodarchiverb030beb9' in consumer group 'mpshistory'
datehour = 1
8/22/17
5:30:17.849 AM

2017-08-22 01:30:17.849 INFO 3815 --- [ThreadPoolService-safe-1-thread-6252] c.a.ids.consumer.BasePermissionConsumer : Received 1 messages in batch for consumer 'mps
historychannelconsumer3prodarchiverb030beb9'
datehour = 1
8/22/17
5:30:17.606 AM

2017-08-22 01:30:17.606 INFO 3815 --- [ThreadPoolService-safe-1-thread-6248] c.a.ids.consumer.BasePermissionConsumer : Received 1 messages in batch for consumer 'mps
historychannelconsumer3prodarchiverb030beb9'
datehour = 1
8/22/17
5:30:17.437 AM

2017-08-22 01:30:17.437 INFO 3815 --- [ThreadPoolService-safe-1-thread-6252] com.adobe.ids.kafka.KafkaRestFacade : Updated consumed offset to value '56622448' for consumer 'mps
historychannelconsumer3prodarchiverb030beb9' in consumer group 'mpshistory'
date
hour = 1
8/22/17
5:30:02.602 AM

2017-08-22 01:30:02.602 INFO 3815 --- [ThreadPoolService-safe-1-thread-6248] com.adobe.ids.kafka.KafkaRestFacade : Updated consumed offset to value '56622448' for consumer 'mpshistorychannelconsumer3prodarchiverb030beb9' in consumer group 'mpshistory'
datehour = 1
8/22/17
5:29:56.573 AM

2017-08-22 01:29:56.573 INFO 3815 --- [ThreadPoolService-safe-1-thread-6240] c.a.ids.consumer.BasePermissionConsumer : Received 1 messages in batch for consumer 'mps
historychannelconsumer3prodarchiverb030beb9'
datehour = 1
8/22/17
5:29:56.572 AM

2017-08-22 01:29:56.572 INFO 3815 --- [ThreadPoolService-safe-1-thread-6244] c.a.ids.consumer.BasePermissionConsumer : Received 0 messages in batch for consumer 'mps
historychannelconsumer3prodarchiverb030beb9'
datehour = 1
8/22/17
5:29:47.776 AM

2017-08-22 01:29:47.776 INFO 3815 --- [ThreadPoolService-safe-1-thread-6244] com.adobe.ids.kafka.KafkaRestFacade : Updated consumed offset to value '56622447' for consumer 'mps
historychannelconsumer3prodarchiverb030beb9' in consumer group 'mps_history'

Tags (1)
0 Karma
Highlighted

Re: How to detect two consecutive lines with same pattern

SplunkTrust
SplunkTrust

What are you trying to do with it? What is the use case? Do you want to eliminate the dups, or detect them?

 your search here
 some eval or rex to get what you want to test into checkfield
| streamstats current=f last(checkfield) as priorcheckfield

If you want to get rid of dups, add this...

| where isnull(priorcheckfield) OR checkfield!=priorcheckfield

If you want to keep both dups and only the dups, add this...

| eval SecondDup=if(checkfield=priorcheckfield,1 null())
| reverse
| streamstats current=T last(SecondDup) as BothDups window=2 
| where BothDups==1
0 Karma
Highlighted

Re: How to detect two consecutive lines with same pattern

SplunkTrust
SplunkTrust
0 Karma
Highlighted

Re: How to detect two consecutive lines with same pattern

Explorer

When i try do query with single consume it works as expected but when i do for all then it's not reprting the exact issue.

index=idsmpsprd "mpshistorychannelconsumer3" | transaction startswith=("Received") | search eventcount > 2 OR eventcount < 2

But these does not work

index=idsmpsprd "mpsprod" | rex field=raw "for consumer '(?P\S+)" | transaction startswith=("Received") by consumer_name | search eventcount > 2

0 Karma
Highlighted

Re: How to detect two consecutive lines with same pattern

SplunkTrust
SplunkTrust

I'll ask again. What are you trying to do? What is your use case?

0 Karma
Highlighted

Re: How to detect two consecutive lines with same pattern

Explorer

i am trying to find the pair of the line with Updated followed by Received with group by consumer group.
Our Java program get's the messages from Kafka and update offset back to Kafka so Updated followed by received should be sequence if somehow ( due to java schedule or let response time from kafka ) if missed the sequence we may missed the message or we ahead of the actual offset in kafka that causes us -ve lag.
Below worked for me

index=idsmpsprd "mpsprod'" earliest=-65m latest=-5m | rex field=raw "for consumer '(?P\S+)" | transaction startswith=("Updated") by consumer_name | search eventcount< 2 OR eventcount > 2

0 Karma
Highlighted

Re: How to detect two consecutive lines with same pattern

Explorer

these is worked for me
index=idsmpsprd "mpsprod'" earliest=-65m latest=-5m | rex field=raw "for consumer '(?P\S+)" | transaction startswith=("Updated") by consumer_name | search eventcount< 2 OR eventcount > 2

0 Karma
Highlighted

Re: How to detect two consecutive lines with same pattern

Explorer

I was able to get result using transaction command

search | earliest=-65m latest=-5m | rex field=raw "for consumer '(?P\S+)" | transaction startswith=("Updated") by consumername | search eventcount > 2

0 Karma