Splunk Search

Help sending header and footer of CSV to nullqueue

msarro
Builder

Hey everyone.
The source files I am currently working with each contain a large amount of records. The problem is they follow a weird format. They begin with some numbers and symbols on a line. There is then a blank line. Then the actual body data starts.

After the body data, there is a blank line.
Finally, there is a footer line made of up some numbers and symbols.

Here is an example.

001;06.0.0;2011-08-01 09:31:02;CA114

DATA
...

10000;2011-08-01 09:34:18

I'm not sure how to ignore the header and footer lines. Any help would be very much appreciated.

Tags (2)
0 Karma
1 Solution

msarro
Builder

In props.conf (items to the left and right of = can be changed to suit your needs):

TRANSFORMS-PBTS-set1=setnull_pbts_head_cdr
TRANSFORMS-PBTS-set2=setnull_pbts_foot_cdr

In transforms.conf (added two stanzas, the regex just removes lines that start with 001 and 10000):

[setnull_pbts_head_cdr]
REGEX=^001;.*$
DEST_KEY=queue
FORMAT=nullQueue

[setnull_pbts_foot_cdr]
REGEX=^10000;.*$
DEST_KEY=queue
FORMAT=nullQueue

View solution in original post

0 Karma

msarro
Builder

In props.conf (items to the left and right of = can be changed to suit your needs):

TRANSFORMS-PBTS-set1=setnull_pbts_head_cdr
TRANSFORMS-PBTS-set2=setnull_pbts_foot_cdr

In transforms.conf (added two stanzas, the regex just removes lines that start with 001 and 10000):

[setnull_pbts_head_cdr]
REGEX=^001;.*$
DEST_KEY=queue
FORMAT=nullQueue

[setnull_pbts_foot_cdr]
REGEX=^10000;.*$
DEST_KEY=queue
FORMAT=nullQueue
0 Karma

ftk
Motivator

Maybe you can post an answer to this question with what you did in order to make it work, so that other users can benefit from it? Thanks!

0 Karma

msarro
Builder

Realized that the header always starts with 001;, and the footer line always starts with 10000. Looks like it is working now. Thanks guys!

0 Karma

ftk
Motivator

Are any of the pieces in the header/footer static? If so which?

0 Karma
Get Updates on the Splunk Community!

Why You Can't Miss .conf25: Unleashing the Power of Agentic AI with Splunk & Cisco

The Defining Technology Movement of Our Lifetime The advent of agentic AI is arguably the defining technology ...

Deep Dive into Federated Analytics: Unlocking the Full Power of Your Security Data

In today’s complex digital landscape, security teams face increasing pressure to protect sprawling data across ...

Your summer travels continue with new course releases

Summer in the Northern hemisphere is in full swing, and is often a time to travel and explore. If your summer ...