My rex search is returning all the rows instead of the one being searched. What am I doing wrong?
index=cloudwatchlogs loggroup="/aws-glue/jobs/xxxxx/*" meta_region="us-east-1" meta_env="TEST" meta_type="aws:jobs" | rex field="message.message" max_match=0 "Total rows from Raw Call meta:\s(?<msg1>\d+)\s" | rex field="message.message" max_match=0 "Total Meta rows written to S3 bucket:\s(?<msg2>\d+)\s" | rex field="message.message" max_match=0 "Total QCI Raw Data rows read from S3 bucket:\s(?<msg3>\d+)\s" | rex field="message.message" max_match=0 "Total root rows written to S3 bucket:\s(?<msg4>\d+)\s"
Sample data -
INFO:__main__:Total rows from Raw Call meta: 3995
INFO:__main__:Deleting duplicate rows
INFO:__main__:Total rows before Deleting duplicate rows: 3995
INFO:__main__:Listing duplicates, if any
INFO:__main__:Total Meta rows written to S3 bucket: 3995
INFO:__main__:Processing RAW QCI Data.
rex isnt a search filter function, its only for field extraction or data replacement.
See: rex - Splunk Documentation
In your search, your filter criteria is:
index=cloudwatchlogs loggroup="/aws-glue/jobs/xxxxx/*" meta_region="us-east-1" meta_env="TEST" meta_type="aws:jobs"
Is the sample data one event with 6 lines or 6 separate events ?
rex isnt a search filter function, its only for field extraction or data replacement.
See: rex - Splunk Documentation
In your search, your filter criteria is:
index=cloudwatchlogs loggroup="/aws-glue/jobs/xxxxx/*" meta_region="us-east-1" meta_env="TEST" meta_type="aws:jobs"
Is the sample data one event with 6 lines or 6 separate events ?
Ok. Have a look into using rex mode=sed in the documentation above and you should be able to replace any lines that don't end in a number with nothing.
Something like this:
| rex field=_raw mode=sed "s/[\n\r]+.*\D([\n\r]+|$)|$/\n/g"
Checks for any line that doesn't end number and replaces it with a new line character.