Process to extract one field from a large, multi-line eventlog

[jgaylard]

I am trying to extract the exit_status from a large, multi-line event log (see below example). I need to set the props.conf file correctly but have been struggling to know how to approach it. An example of the log below - exit_status is on the last line of the event.

What would be the best approach for a file like this?

My attempt at props.conf in the universal forwarder,though commenting out this file and restarting the forwarder has no effect on the output in SplunkWeb of the receiver so for some reason these settings do not seem to have taken effect:

[source::/Libary/Logs/CCC.log]
SHOULD_LINEMERGE = true
ONLY_BREAK_AFTER_DATE = false
BREAK_ONLY_BEFORE = ^={20}\.+={20}$

Can anyone help with this?

NOTE: I have also included the beginning of the header for the next event.

==================== Carbon Copy Cloner v. 3.5.1 (1073): 2012-08-17 11:36:53 +1000 ====================

OS: Version 10.8 (Build 12A269)
Architecture:   x86_64
Mac model:  MacBookPro5,4
Number of CPUs: 2
CPU Speed:  2.53 GHz
Memory: 4 GB
Console user id:    502
CCC euid:   502
Task owner: jimg (502)


Task: Copying selected files (-psn_0_831691)
Source: Macintosh HD
    Source path: /
    Mount point: /
    Filesystem: hfs
    Capacity: 249.20 GB
    Used: 177.44 GB
    Available: 71.76 GB
    Mac OS X version: 10.8
    UUID: 398DF01C-06A1-353F-B526-C10A5776EFDC
    Device ID: /dev/disk0s2
    Device vendor: Unidentified Vendor
    Device model: FUJITSU MJA2250BH FFS G1                
    Device interface: SATA
    Partition format: IOGUIDPartitionScheme
    Case sensitive: No
    Filesystem owner: 0
    Ownership respected: Yes

Destination: NO NAME
    Destination path: /Volumes/NO NAME
    Mount point: /Volumes/NO NAME
    Filesystem: msdos
    Capacity: 4.00 GB
    Used: 374.28 MB
    Available: 3.63 GB
    Mac OS X version: Mac OS not installed
    UUID: 
    Device ID: /dev/disk3s1
    Device vendor: SanDisk
    Device model: Cruzer Edge
    Device interface: USB
    Partition format: IOFDiskPartitionScheme
    Case sensitive: No
    Filesystem owner: 0
    Ownership respected: No


Settings
    Archive deleted items, owner: jimg
    - Protect root-level items
    Archive modified items
    Prune until 15000 MB free space is available
    Archive the Recovery HD volume: No (already archived)


08/17 11:36:54  Preparing...
08/17 11:36:54  Authenticating...
08/17 11:37:01  "Macintosh HD" has ownership enabled.
08/17 11:37:01  "NO NAME" is not formatted HFS+, not enabling ownership
08/17 11:37:02  Spotlight state on destination: On
08/17 11:37:02  **** Support for Access Control Lists is disabled ****
08/17 11:37:02  **** Support for Hard Links is disabled ****
08/17 11:37:02  **** Support for Ownership and Group filesystem metadata is disabled ****
08/17 11:37:02  **** Support for Device files is disabled ****
08/17 11:37:02  **** Files larger than 4GB will be skipped because the destination does not support files that large ****
08/17 11:37:02  Archive Manager: Creating folder at /Volumes/NO NAME/_CCC Archives/2012-08-17 (August 17) 11-37-02
currentOSVersion: 1080, recHDArchiveVersion: 1080, recHDIsCurrent: Yes
08/17 11:37:03  Pruning archived content...
08/17 11:37:03  Sparing /Volumes/NO NAME/_CCC Archives/2012-08-17 (August 17) 11-36-30 and any remaining archives [Current free space: 3628204032 bytes]
08/17 11:37:03  Nothing to prune...
08/17 11:37:03  Initiating synchronization engine...
08/17 11:37:03  DEBUG: [sender] Running the x86_64 executable (10001)
08/17 11:37:03  DEBUG: [dfcc: sender] Effective UID is 0 for // (10001)
08/17 11:37:03  receiver: Disabling HFS compression support, /Volumes/NO NAME doesn't support it (use --protect-decmpfs to force protection of the com.apple.decmpfs extended attribute). (10001)
08/17 11:37:03  [sender] Fileflags mask for //: 0 (10001)
08/17 11:37:03  DEBUG: [dfcc: sender] Setting effective UID back to 0 for source (10001)
08/17 11:37:03  DEBUG: [receiver] Running the x86_64 executable (10001)
08/17 11:37:03  DEBUG: [dfcc: receiver] Effective UID is 0 for /Volumes/NO NAME (10001)
08/17 11:37:03  [receiver] Fileflags mask for /Volumes/NO NAME: 109 (10001)
08/17 11:37:03  DEBUG: Destination filesystem doesn't support _PC_NAME_CHARS_MAX, will use _PC_NAME_MAX instead (10001)
08/17 11:37:03  DEBUG: Max xattr size for the destination filesystem is 131072 bytes (10001)
08/17 11:37:03  DEBUG: [dfcc: receiver] Setting effective UID back to 0 for dest (10001)
08/17 11:37:03  Building a list of items to be considered for backup
08/17 11:37:03  Preparing to copy, total items to consider: 2543
08/17 11:37:03  Build time: 0.086
08/17 11:37:03  Comparing selected items on the source and destination...
08/17 11:37:04  Hiding files that should be hidden...
08/17 11:37:04  Deleting empty archive folders...
08/17 11:37:04  Summary statistics: 
    Data copied: 16.38 KB [Please see http://www.bombich.com/log.html for a comment about this figure]
    Total data in file set: 251.37 MB
     Regular files copied: 1
     Largest file encountered: 68.25 MB
     Directories: 792
     Regular files: 1682
     Symlinks: 69
     Devices: 0
     Special files: 0
     Hard links: 0
     Extended attributes (modified): 1972 (0.00 KB)
     (Unmodified extended attributes are not enumerated)

08/17 11:37:04  Copying selected files: Time elapsed: 00:00:10. Data copied: 16.38 KB
exit_status=0 elapsed_time=10 source="/" destination="/Volumes/NO NAME" end_time=2012-08-17T11:37:04+1000 start_time=2012-08-17T11:36:54+1000 data_copied=16384 task_name="Copying selected files" 
================================================================================


==================== Carbon Copy Cloner v. 3.5.1 (1073): 2012-08-17 11:54:25 +1000 ====================

[lguinn2 · Accepted solution]

First, breaking the data stream into events is part of the "parsing phase" in Splunk. Universal forwarders do not parse. Therefore, these props.conf entries belong in a props.conf file on the indexer, not the forwarder.

Second, parsing only happens once - when the data is indexed. Changing things in props.conf will only affect new data; existing data will not be changed. (Think of Splunk as a write-once data store.)

Third, once a forwarder has sent data, it will not send that data a second time.

Finally, a piece of good news: fields are extracted at search time dynamically. So creating a field happens on the indexer - and changing field definitions does not require you to re-index your data. And another good thing: Splunk automatically looks for patterns of name=value, and recreates a field called "name"

So you should already have a field named exit_status.

If the existing data has been broken into many events (as I suspect it has), you will need to re-index the data to fix it. This requires that you
- stop Splunk on the indexer and forwarder
- clean the index on the indexer
- reset the "fishbucket" on the forwarder
- restart the indexer and forwarder

[lguinn2]

[source::/Libary/Logs/CCC.log]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^={20}\.+={20}$
BREAK_ONLY_BEFORE_DATE=false

I don't think there is a "ONLY_BREAK_AFTER_DATE" setting. But that's the only thing that really concerns me.

[jgaylard]

Thanks for the answer. Very helpful.

To test I have been manually running a clone job every time I restart the forwarder/indexer (I have tried multiple combinations) to generate fresh data with the new settings.

I will try again with props.conf on the indexer, not the forwarder. Do my props.conf settings look right to you?