topic bacula events again in Getting Data In

bacula events again

sergemueller — Mon, 28 Sep 2020 09:41:11 GMT

Hi,
i've read and tried this but somehow it does not work for me.

i've put the recommended settings into:

$SPLUNKHOME$/etc/system/local/props.conf
and also tried

$SPLUNKHOME$/etc/apps/search/local/props.conf

but it never really works.
Some events start in the middle of the indented block(eg at Start Time)
some start at the Date, but i never got the whole idendet Block into one event.
Here's an example of the data:



17-Jun 10:05 backup1.ber2-dir JobId 1948: Start Backup JobId 1948, Job=test.2011-06-17_10.05.00_40

17-Jun 10:05 backup1.ber2-dir JobId 1948: Using Device "FileStorage_test"

17-Jun 10:05 backup1.ber2-sd JobId 1948: Volume "test_new02" previously written, moving to end of data.

17-Jun 10:05 backup1.ber2-sd JobId 1948: Ready to append to end of Volume "test_new02" size=31846981494

17-Jun 10:10 backup1.ber2-sd JobId 1948: Job write elapsed time = 00:05:29, Transfer rate = 6.540 M Bytes/second

17-Jun 10:10 backup1.ber2-dir JobId 1948: Bacula backup1.ber2-dir 5.0.3 (04Aug10): 17-Jun-2011 10:10:39

  Build OS:               x86_64-unknown-linux-gnu debian 5.0.6

  JobId:                  1948

  Job:                    test.2011-06-17_10.05.00_40

  Backup Level:           Incremental, since=2011-06-17 06:05:03

  Client:                 "server11227.example.com-fd" 5.0.2 (28Apr10) x86_64-unknown-linux-gnu,redhat,Enterprise release

  FileSet:                "test Set" 2011-01-16 11:29:38

  Pool:                   "File_test" (From Job resource)

  Catalog:                "MyCatalog" (From Client resource)

  Storage:                "File_test" (From Job resource)

  Scheduled time:         17-Jun-2011 10:05:00

  Start time:             17-Jun-2011 10:05:03

  End time:               17-Jun-2011 10:10:39

  Elapsed time:           5 mins 36 secs

  Priority:               10

  FD Files Written:       5

  SD Files Written:       5

  FD Bytes Written:       2,151,817,284 (2.151 GB)

  SD Bytes Written:       2,151,817,820 (2.151 GB)

  Rate:                   6404.2 KB/s

  Software Compression:   None

  VSS:                    no

  Encryption:             no

  Accurate:               no

  Volume name(s):         test_new02

  Volume Session Id:      32

  Volume Session Time:    1308035975

  Last Volume Bytes:      34,000,395,480 (34.00 GB)

  Non-fatal FD errors:    0

  SD Errors:              0

  FD termination status:  OK

  SD termination status:  OK

  Termination:            Backup OK

17-Jun 10:10 backup1.ber2-dir JobId 1948: Begin pruning Jobs older than 6 months .
17-Jun 10:10 backup1.ber2-dir JobId 1948: No Jobs found to prune.
17-Jun 10:10 backup1.ber2-dir JobId 1948: Begin pruning Jobs.
17-Jun 10:10 backup1.ber2-dir JobId 1948: No Files found to prune.
17-Jun 10:10 backup1.ber2-dir JobId 1948: End auto prune.

Any help would be appreciated

Edit: Ah well i forgot:

Excerp from my props.conf:



[bacula]

BREAK_ONLY_BEFORE=^\d{2}-[A-Za-z]{3}\s+\d{2}:\d{2}\s+

SHOULD_LINEMERGE = true

TIME_FORMAT=%d-%b %H:%M

TIME_PREFIX=^

MAX_TIMESTAMP_LOOKAHEAD=13

Re: bacula events again

ziegfried — Fri, 17 Jun 2011 17:27:01 GMT

You could try specifying a line breaker instead of using line merging:

[bacula]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?!\s+)

This should break events by vertical whitespace not followed by whitespace characters.

Re: bacula events again

sergemueller — Mon, 20 Jun 2011 07:52:56 GMT

doesn really work,

now it breaks between "end time:" and volume session id

i dont really getit

Re: bacula events again

sergemueller — Mon, 20 Jun 2011 09:03:37 GMT

it also appears, that the TIME_PREFIX is not recognized, since it evaluates times in the middel of the line (The One after " Start Time:"), but the sourcetype is recognized (i tested it with an extraction)

Re: bacula events again

aquilino1 — Tue, 15 Nov 2011 00:20:07 GMT

You might want to double-check the first line of your bacula log file and make sure that it doesn't have a regular syslog style timestamp. If it does, this may be throwing off proper detection of the timestamp event for your bacula sourcetype on initial import. I had to delete the first line in my bacula log file and then re-index the data in order to correct the problem.

Also, at the risk of someone telling me that I'm doing my field extractions all wrong (I'm still learning), here is my bacula_logs sourcetype entry in props.conf in case it's helpful to anyone. So far it seems to be working well. If anyone has a suggestion for how I might improve upon it though, I'm open to ideas. 😃

[bacula_logs]
LEARN_MODEL=true
TIME_FORMAT = %d-%b %H:%M
EXTRACT-jobstatus = Termination:\s+\W*(?<jobstatus>(Backup OK|Backup Error|Backup Canceled))
EXTRACT-backupclient-1 = Client: \s*\W*(?<backupclient>[a-z0-9\-A-Z]*)
EXTRACT-backupclient-2 = gethostbyname\(\) for host "(?<backupclient>[a-z\.0-9\-A-Z]*)"
EXTRACT-backuplevel = Backup Level: \s*(?<backuplevel>\w*)
EXTRACT-compression = Software Compression: \s*(?<compression>\S+\s*%)
EXTRACT-sourcehost = \d+-\w\w\w \d+:\d+ \s*(?<sourcehost>\S*)
EXTRACT-jobid = JobId [:]*(?<jobid>\d+)
EXTRACT-byteswritten = SD Bytes Written:\s+(?<byteswritten>\S*)
EXTRACT-origination = ^\d\d-\w\w\w \d\d:\d\d (?<origination>\S+)
EXTRACT-fatal-error = Fatal error: (?<fatal_error>.*)
EXTRACT-fd-status = FD termination status:  (?<fd_status>.*?(?=SD term))
EXTRACT-sd-status = SD termination status:  (?<sd_status>.*?(?=Termination:))
EXTRACT-jobname = \s+Job: \s+(?<jobname>\S+)

Re: bacula events again

kristian_kolb — Tue, 15 Nov 2011 09:28:54 GMT

Hi,

did this message board filter out your backslashes or does your props.conf regex actually look like this?

[bacula]
BREAK_ONLY_BEFORE=^d{2}-[A-Za-z]{3}s+d{2}:d{2}s+

Other than that, I think you should be able to break at the correct places with (just being a little more specific on the numbers used for matching date/time);

[bacula]
LINE_BREAKER=([\r\n]+)[0-3][0-9]-[A-Za-z]{3}\s+[0-2][0-9]:[0-5][0-9]\s+
SHOULD_LINEMERGE = False
TIME_FORMAT=%d-%b %H:%M
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=13

Hope this helps,

Kristian