Using this which ignores header (and also retrieves field names from header): http://docs.splunk.com/Documentation/Splunk/6.1.2/Data/Extractfieldsfromfileheadersatindextime
But how to ignore footer from forwarding, now it is ending up with this error as footers doesn't have date/time.
07-08-2014 16:35:11.591 +0000 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sat Jul 5 15:54:07 2014). Context: source::/usr/PATHOFFILE/filename_c105.log.20140704-163507-00|host::105|file_type_access|50
Lets say the footer looks like this:
# Hey, Im a footer #
Use this in your props.conf:
TRANSFORMS-null = sourcetype_NullQueue
Use this in your transforms.conf:
[sourcetype_NullQueue] REGEX=^#\sHey\,Im.a.footer.#$ DEST_KEY = queue FORMAT = nullQueue
Then restart and test.
I'd like both header and footer to be removed, and have the following in props.conf
[sourcetype_access] HEADER_FIELD_LINE_NUMBER = 2 FIELD_HEADER_REGEX = ^#Fields:\s(.*) FIELD_DELIMITER = \t TIMESTAMP_FIELDS = date, time PREAMBLE_REGEX = #.* TRANSFORMS-to_trash1 = remove_comments
[remove_comments] DEST_KEY = queue REGEX = ^(?:#) FORMAT = nullQueue
It removes only the header but not footer. I tried using REGEX instead of PREAMBLE_REGEX but that doesn't help. What should be done to remove both header and footer?