topic Re: how to ignore footer from forwarding in Getting Data In

how to ignore footer from forwarding

sathiyamoorthy — Tue, 08 Jul 2014 16:43:51 GMT

Using this which ignores header (and also retrieves field names from header): http://docs.splunk.com/Documentation/Splunk/6.1.2/Data/Extractfieldsfromfileheadersatindextime

But how to ignore footer from forwarding, now it is ending up with this error as footers doesn't have date/time.

07-08-2014 16:35:11.591 +0000 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jul  5 15:54:07 2014). Context: source::/usr/PATHOFFILE/filename_c105.log.20140704-163507-00|host::105|file_type_access|50

Re: how to ignore footer from forwarding

jkat54 — Tue, 08 Jul 2014 19:43:20 GMT

Lets say the footer looks like this:

# Hey, Im a footer #

Use this in your props.conf:

TRANSFORMS-null = sourcetype_NullQueue

Use this in your transforms.conf:

[sourcetype_NullQueue]
REGEX=^#\sHey\,Im.a.footer.#$
DEST_KEY = queue
FORMAT = nullQueue

Then restart and test.

Re: how to ignore footer from forwarding

sathiyamoorthy — Wed, 09 Jul 2014 09:20:48 GMT

I'd like both header and footer to be removed, and have the following in props.conf

[sourcetype_access]
HEADER_FIELD_LINE_NUMBER = 2
FIELD_HEADER_REGEX = ^#Fields:\s(.*)
FIELD_DELIMITER = \t
TIMESTAMP_FIELDS = date, time
PREAMBLE_REGEX = #.*
TRANSFORMS-to_trash1 = remove_comments

in transforms.conf

[remove_comments]
DEST_KEY = queue
REGEX = ^(?:#)
FORMAT = nullQueue

It removes only the header but not footer. I tried using REGEX instead of PREAMBLE_REGEX but that doesn't help. What should be done to remove both header and footer?

Re: how to ignore footer from forwarding

sathiyamoorthy — Wed, 09 Jul 2014 09:21:49 GMT

Please refer my comment, I'd like both header and footer to be removed, and use the fields from header.

Re: how to ignore footer from forwarding

alekksi — Wed, 09 Jul 2014 10:20:06 GMT

We don't know what your footer looks like, so it's hard to say. What jkat54 has said is how I get rid of headers/footers.

Re: how to ignore footer from forwarding

jkat54 — Thu, 10 Jul 2014 13:15:50 GMT

Yes, please give examples of your header and footer.