Splunk Search

Help sending header and footer of CSV to nullqueue

msarro
Builder

Hey everyone.
The source files I am currently working with each contain a large amount of records. The problem is they follow a weird format. They begin with some numbers and symbols on a line. There is then a blank line. Then the actual body data starts.

After the body data, there is a blank line.
Finally, there is a footer line made of up some numbers and symbols.

Here is an example.

001;06.0.0;2011-08-01 09:31:02;CA114

DATA
...

10000;2011-08-01 09:34:18

I'm not sure how to ignore the header and footer lines. Any help would be very much appreciated.

Tags (2)
0 Karma
1 Solution

msarro
Builder

In props.conf (items to the left and right of = can be changed to suit your needs):

TRANSFORMS-PBTS-set1=setnull_pbts_head_cdr
TRANSFORMS-PBTS-set2=setnull_pbts_foot_cdr

In transforms.conf (added two stanzas, the regex just removes lines that start with 001 and 10000):

[setnull_pbts_head_cdr]
REGEX=^001;.*$
DEST_KEY=queue
FORMAT=nullQueue

[setnull_pbts_foot_cdr]
REGEX=^10000;.*$
DEST_KEY=queue
FORMAT=nullQueue

View solution in original post

0 Karma

msarro
Builder

In props.conf (items to the left and right of = can be changed to suit your needs):

TRANSFORMS-PBTS-set1=setnull_pbts_head_cdr
TRANSFORMS-PBTS-set2=setnull_pbts_foot_cdr

In transforms.conf (added two stanzas, the regex just removes lines that start with 001 and 10000):

[setnull_pbts_head_cdr]
REGEX=^001;.*$
DEST_KEY=queue
FORMAT=nullQueue

[setnull_pbts_foot_cdr]
REGEX=^10000;.*$
DEST_KEY=queue
FORMAT=nullQueue
0 Karma

ftk
Motivator

Maybe you can post an answer to this question with what you did in order to make it work, so that other users can benefit from it? Thanks!

0 Karma

msarro
Builder

Realized that the header always starts with 001;, and the footer line always starts with 10000. Looks like it is working now. Thanks guys!

0 Karma

ftk
Motivator

Are any of the pieces in the header/footer static? If so which?

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...