topic Re: Help sending header and footer of CSV to nullqueue in Splunk Search

Help sending header and footer of CSV to nullqueue

msarro — Mon, 22 Aug 2011 18:10:17 GMT

Hey everyone.
The source files I am currently working with each contain a large amount of records. The problem is they follow a weird format. They begin with some numbers and symbols on a line. There is then a blank line. Then the actual body data starts.

After the body data, there is a blank line.
Finally, there is a footer line made of up some numbers and symbols.

Here is an example.

001;06.0.0;2011-08-01 09:31:02;CA114

DATA
...

10000;2011-08-01 09:34:18

I'm not sure how to ignore the header and footer lines. Any help would be very much appreciated.

Re: Help sending header and footer of CSV to nullqueue

ftk — Mon, 22 Aug 2011 20:12:16 GMT

Are any of the pieces in the header/footer static? If so which?

Re: Help sending header and footer of CSV to nullqueue

msarro — Tue, 23 Aug 2011 15:53:17 GMT

Realized that the header always starts with 001;, and the footer line always starts with 10000. Looks like it is working now. Thanks guys!

Re: Help sending header and footer of CSV to nullqueue

ftk — Tue, 23 Aug 2011 16:41:30 GMT

Maybe you can post an answer to this question with what you did in order to make it work, so that other users can benefit from it? Thanks!

Re: Help sending header and footer of CSV to nullqueue

msarro — Thu, 25 Aug 2011 13:19:46 GMT

In props.conf (items to the left and right of = can be changed to suit your needs):

TRANSFORMS-PBTS-set1=setnull_pbts_head_cdr
TRANSFORMS-PBTS-set2=setnull_pbts_foot_cdr

In transforms.conf (added two stanzas, the regex just removes lines that start with 001 and 10000):

[setnull_pbts_head_cdr]
REGEX=^001;.*$
DEST_KEY=queue
FORMAT=nullQueue

[setnull_pbts_foot_cdr]
REGEX=^10000;.*$
DEST_KEY=queue
FORMAT=nullQueue