About holtb

holtb · ‎09-04-2012

It seems to work well with the modification to the LINE_BREAKER. I wrote the following rules for field extraction (from transforms.conf), can anyone comment on their efficiency, is there a better way to do this? [extract_tsm_success] REGEX = ^Successful\ incremental\ backup\ of\ \'(.+?)\'\n FORMAT = Filesystem::$1 [extract_tsm_oinspect] REGEX = ^Total number\ of\ objects\ inspected:\s+(.+?)\n FORMAT = ObjectsInspected::$1 [extract_tsm_obackup] REGEX = ^Total\ number\ of\ objects\ backed\ up:\s+(.+?)\n FORMAT = ObjectsBackedup::$1 [extract_tsm_oupdate] REGEX = ^Total\ number\ of\ objects\ updated:\s+(.+?)\n FORMAT = ObjectsUpdated::$1 [extract_tsm_orebound] REGEX = ^Total\ number\ of\ objects\ rebound:\s+(.+?)\n FORMAT = ObjectsRebound::$1 [extract_tsm_odeleted] REGEX = ^Total\ number\ of\ objects\ deleted:\s+(.+?)\n FORMAT = ObjectsDeleted::$1 [extract_tsm_oexpired] REGEX = ^Total\ number\ of\ objects\ expired:\s+(.+?)\n FORMAT = ObjectsExpired::$1 [extract_tsm_ofailed] REGEX = ^Total\ number\ of\ objects\ failed:\s+(.+?)\n FORMAT = ObjectsFailed::$1 [extract_tsm_bytesinspected] REGEX = ^Total\ number\ of\ bytes\ inspected:\s+(.+?)\n FORMAT = BytesInspected::$1 [extract_tsm_bytestransferred] REGEX = ^Total number of bytes transferred:\s+(.+?)\n FORMAT = BytesTransfered::$1 [extract_tsm_transfertime] REGEX = ^Data\ transfer\ time:\s+(.+?)\n FORMAT = TransferTime::$1 [extract_tsm_transferrate] REGEX = ^Network\ data\ transfer\ rate:\s+(.+?)\n FORMAT = TransferRate::$1 [extract_tsm_aggtransferrate] REGEX = ^Aggregate\ data\ transfer\ rate:\s+(.+?)\n FORMAT = AggregateTransferRate::$1 [extract_tsm_objcompression] REGEX = ^Objects\ compressed\ by:\s+(.+?)\n FORMAT = ObjectCompression::$1 [extract_tsm_reductionratio] REGEX = ^Total\ data\ reduction\ ratio:\s+(.+?)\n FORMAT = ReductionRatio::$1 [extract_tsm_timeelapsed] REGEX = ^Elapsed\ processing\ time:\s+(.+?)\n FORMAT = TimeElapsed::$1

holtb · ‎09-04-2012

Awesome, thanks. The LINE_BREAKER was stolen from another splunkbase when trying to take each logfile as a single event.

holtb · ‎09-04-2012

We're trying to analyze a complex multiline backup log and having some difficulties. I was hoping an expert here could help. The backup logs have two states, one of which is success, looking like below. Otherwise it has lines in it that look like this denoting eror: ANS1802E Incremental backup of '/mnt/wpimaging/acsm-home' finished with 103 failure ANS1017E Session rejected: TCP/IP connection failure What I would really like is to only see the summary and ANS....E lines, ignoring the ANS....I or ANS....W lines. It'd be great if all the fields of the summary were captured as fields so we could generate table of all the successful backups. So far we just have the following in our props.conf, any suggestions would be appreciated [TSM_REPORTS] MAX_EVENTS=2000000 LINE_BREAKER=((?!)) LEARN_MODEL = false BREAK_ONLY_BEFORE= IBM Tivoli Storage Manager BREAK_ONLY_BEFORE_DATE = False MUST_BREAK_AFTER= Elapsed processing time: TRUNCATE=0 Logfile: IBM Tivoli Storage Manager Command Line Backup-Archive Client Interface Client Version 6, Release 2, Level 2.3 Client date/time: 08/28/2012 15:15:02 (c) Copyright by IBM Corporation and other(s) 1990, 2010. All Rights Reserved. Normal File--> 348 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09.hdr [Sent] Normal File--> 491,520 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09.img [Sent] Normal File--> 9,262 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09.imp [Sent] Normal File--> 3,411 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09.mat [Sent] Normal File--> 985,608 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09.mnc [Sent] Normal File--> 986,136 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_norm.mnc [Sent] Normal File--> 348 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_norm_dist5.hdr [Sent] Normal File--> 983,040 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_norm_dist5.img [Sent] Normal File--> 428 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_norm_dist5.mat [Sent] Normal File--> 348 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_nuc.hdr [Sent] Normal File--> 983,040 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_nuc.img [Sent] Normal File--> 428 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_nuc.mat [Sent] Successful incremental backup of '/mnt/wpimaging/analysis' Total number of objects inspected: 8,260,012 Total number of objects backed up: 18,077 Total number of objects updated: 0 Total number of objects rebound: 0 Total number of objects deleted: 0 Total number of objects expired: 8 Total number of objects failed: 0 Total number of bytes inspected: 26.25 TB Total number of bytes transferred: 78.61 GB Data transfer time: 46,329.38 sec Network data transfer rate: 1,779.22 KB/sec Aggregate data transfer rate: 1,537.23 KB/sec Objects compressed by: 0% Total data reduction ratio: 99.71% Elapsed processing time: 14:53:42

holtb · ‎08-20-2012

I'm new to this wonderful app, so pardon my inexperience if this is easy...I have a very long search string, but I'd like to have these values created automatically on index. How could I convert this set of converts and calculations into a transform? source="/sge/default/default/common/accounting" | eval "Wait Time"=start_time-submit_time | eval "Run Time"=end_time-start_time | eval "Submit Time"=strftime(submit_time,"%Y %b %d %H:%M:%S") | eval "Start Time"=strftime(start_time,"%Y %b %d %H:%M:%S") | eval "End Time"=strftime(end_time,"%Y %b %d %H:%M:%S") | table uid job_name project qname "Submit Time" wait_time "Start Time" "End Time" run_time failed I tried doing it this way to no avail: [accttimeconvert] eval "Wait Time"=start_time-submit_time eval "Run Time"=end_time-start_time eval "Submit Time"=strftime(submit_time, "%Y/%m/%d %H:%M:%S") eval "Start Time"=strftime(start_time, "%Y/%m/%d %H:%M:%S") eval "End Time"=strftime(end_time, "%Y/%m/%d %H:%M:%S")

holtb · ‎08-08-2012

That's pretty neat, but I don't care about the key=value pairs in the middle, I'd rather keep that whole section intact. I just want to separate the many fields by : so I can search against them separately.

holtb · ‎08-08-2012

I'm trying to extract -all- the fields from a rather complex Oracle Grid Engine log file with a format like this: all.q:50s01fb:clusterusers:zarakhov:transc4desc.fasta_$SGE_TASK_ID.qsub:5161781:sge:0:1344452577:1344452604:1344452617:0:0:13:8.781664:0.452931:0.000000:0:0:0:0:332711:716:0:0.000000:0:0:0:0:12147:13606:normal:MPRICompGenomics:NONE:1:987:9.234595:1.061786:0.000000:-u zarakhov -q all.q -l h_rt=21600,h_vmem=2G,mem_free=2G,mem_reserve=2G,virtual_free=2G:0.000000:NONE:383160320.000000:0:0 Obviously, I'd like to separate on the colons and dump each of the variables into a sequentially labeled variable (fieldname1-fieldname27 or whatnot). I've found fairly easy ways to extract the fields one at a time, is there a mechanism for doing in a more efficient way? Like the split command in perl?

Posts	6
Solutions	0
Karma Given	1
Karma Received	3
Member Since	‎08-08-2012

Online Status	Offline
Date Last Visited	‎06-05-2020 02:03 AM

Help with complex multiline backup log

Convert timestamps in transforms.conf?

Field Extraction - Separate on Colon?

Re: Help with complex multiline backup log

Re: Help with complex multiline backup log

Help with complex multiline backup log

Convert timestamps in transforms.conf?

Re: Field Extraction - Separate on Colon?

Field Extraction - Separate on Colon?

Join the Conversation