Getting Data In

Multiple transforms on one data source

msarro
Builder

I am working with CSV files. When they come in, the first row contains a version header which I want to get rid of. That is followed by a start record, which I need to get rid of (it contains no information other than the time the record collection was started) and the file ends with an end record, which I also want to get rid of.

Here are the relevant stanzas in my props.conf file:

[AS-CDR]
REPORT-ascdrparse=as_cdr_parse
TRANSFORMS-null028=setnull_as_head_cdr
TRANSFORMS-null029=setnull_as_start_cdr
TRANSFORMS-null030=setnull_as_end_cdr
SHOULD_LINEMERGE=false
KV_MODE=none

Here are the relevant stanzas in my transforms.conf file:

[as_cdr_parse]
DELIMS","
FIELDS="Record ID", "Service Provider", "Type", "User Number", "Group Number", "Direction", "Calling Number", "Calling Presentation Indicator", "Called Number", "Start Time", "User Time Zone", "Answer Indicator", "Answer Time", "Release Time", "Termination Cause", "Network Type", "Carrier Identification Code", "Dialed Digits", "Call Category", "Network Call Type", "Network Translated Number", "Network Translated Group", "Releasing Party", "Route", "Network Call ID", "Codec", "Access Device Address", "Access Call ID", "Spare 1", "Failover Correlation ID", "Spare 2", "Group", "Department", "Account Code", "Authorization Code", "Original Called Number", "Original Called Presentation Indicator", "Original Called Reason", "Redirecting Number", "Redirecting Presentation Indicator", "Redirecting Reason", "Charge Indicator", "Type Of Network", "Voice Portal Calling: Invocation Time", "Local Call ID", "Remote Call ID", "Calling Party Category", "Instant Conference: Invocation Time", "Instant Conference: Call ID", "Instant Conference: TO", "Instant Conference: FROM", "Instant Conference: Conference ID", "Instant Conference: Role", "Instant Conference: Bridge", "Instant Conference: Owner", "Instant Conference: Owner DN", "Instant Conference: Title", "Instant Conference: Project Code", "Key", "Creator", "Originator Network", "Terminator Network", "Account Code Per Call: Invocation Time", "Account Code Per Call: FAC Result", "ACB Activation: Invocation Time", "ACB Activation: FAC Result", "ACB Deactivation: Invocation Time", "ACB Deactivation: FAC Result", "Call Park: Invocation Time", "Call Park: FAC Result", "Call Park Retrieve: Invocation Time", "Call Park Retrieve: FAC Result", "Call Pickup: Invocation Time", "Call Pickup: FAC Result", "Directed Call Pickup: Invocation Time", "Directed Call Pickup: FAC Result", "Directed CPU Barge In: Invocation Time", "Directed CPU Barge In: FAC Result", "Cancel Call Waiting Per Call: Invocation Time", "Cancel Call Waiting Per Call: FAC Result", "CFA Activation: Invocation Time", "CFA Activation: FAC Result", "CFA Deactivation: Invocation Time", "CFA Deactivation: FAC Result", "CFB Activation: Invocation Time", "CFB Activation: FAC Result", "CFB Deactivation: Invocation Time", "CFB Deactivation: FAC Result", "CFNA Activation: Invocation Time", "CFNA Activation: FAC Result", "CFNA Deactivation: Invocation Time", "CFNA Deactivation: FAC Result", "CLID Delivery Per Call: Invocation Time", "CLID Delivery Per Call: FAC Result", "CLID Blocking Per Call: Invocation Time", "CLID Blocking Per Call: FAC Result", "Customer Originated Trace: Invocation Time", "Customer Originated Trace: FAC Result", "Direct VM Transfer: Invocation Time", "Direct VM Transfer: FAC Result", "DND Activation: Invocation Time", "DND Activation: FAC Result", "DND Deactivation: Invocation Time", "DND Deactivation: FAC Result", "SAC Lock: Invocation Time", "SAC Lock: FAC Result", "SAC Unlock: Invocation Time", "SAC: Unlock: FAC Result", "Flash Call Hold: Invocation Time", "Flash Call Hold: FAC Result", "Last Number Redial: Invocation Time", "Last Number Redial: FAC Result", "Return Call: Invocation Time", "Return Call: FAC Result", "sd100 Programming: Invocation Time", "sd100 Programming: FAC Result", "sd8 Programming: Invocation Time", "sd8 Programming: FAC Result", "Clear MWI: Invocation Time", "Clear MWI: FAC Result", "User Id", "Other Party Name", "Other Party Name Presentation Indicator", "MOH Per Call Deactivation: Invocation Time", "MOH Per Call Deactivation: FAC Result", "Push To Talk: Invocation Time", "Push To Talk: FAC Result", "Hoteling: Invocation Time", "Hoteling: Group", "Hoteling: User ID", "Hoteling: User Number", "Hoteling: Group Number", "Diversion Inhibitor: FAC Invocation Time", "Diversion Inhibitor: FAC Result", "Trunk Group Name", "Instant Conference: Recording Duration", "Instant Group Call: Invocation Time", "Instant Group Call: Push To Talk", "Instant Group Call: Related Call ID", "Custom Ringback: Invocation Time", "CLID Permitted", "Auto Hold Retrieve: Invocation Time", "Auto Hold Retrieve: Action", "Access Network Info", "Charging Function Addresses", "Charge Number", "Related Call ID", "Related Call ID Reason", "Transfer: Invocation Time", "Transfer: Result", "Transfer: Related Call ID", "Transfer: Type", "Conference: Start Time", "Conference: Stop Time", "Conference: Conf. ID", "Conference: Type", "Codec Usage", "VM Busy Activation: Invocation Time", "VM Busy Activation: FAC Result", "VM Busy Deactivation: Invocation Time", "VM Busy Deactivation: FAC Result", "VM No Answer Activation: Invocation Time", "VM No Answer Activation: FAC Result", "VM No Answer Deactivation: Invocation Time", "VM No Answer Deactivation: FAC Result", "VM Always Activation: Invocation Time", "VM Always Activation: FAC Result", "VM Always Deactivation: Invocation Time", "VM Always Deactivation: FAC Result", "No Answer Timer Set: Invocation Time", "No Answer Timer Set: FAC Result", "CLID Blocking Activation Invocation Time", "CLID Blocking Activation: FAC Result", "CLID Blocking Deactivation: Invocation Time", "CLID Blocking Deactivation: FAC Result", "Call Waiting Activation: Invocation Time", "Call Waiting Activation: FAC Result", "Call Waiting Deactivation: Invocation Time", "Call Waiting Deactivation: FAC Result", "Fax Messaging", "Two Stage Dialing Digits", "Trunk Group Info", "Recall Type", "CFNRc Activation: Invocation Time", "CFNRc Activation: FAC Result", "CFNRc Deactivation: Invocation Time", "CFNRc Deactivation: FAC Result", "q850 Cause", "Dialed Digits Context", "Called Number Context", "Network Translated Number Context", "Calling Number Context", "Original Called Number Context", "Redirecting Number Context", "Location Activation Result", "Location Deactivation Result", "Call Retrieve Result", "Routing Number", "Origination Method", "Call Parked: Invocation Time", "BroadworksAnywhere: Related Call ID", "ACR Activation: Invocation Time", "ACR Activation: FAC Result", "ACR Deactivation: Invocation Time", "ACR Deactivation: FAC Result", "Outside Access Code"

[setnull_as_head_cdr]
REGEX=^\v.*I\b
DEST_KEY=queue
FORMAT=nullQueue

[setnull_as_start_cdr]
REGEX=^\d*\.\d{4}-\d{6},,Start\b
DEST_KEY=queue
FORMAT=nullQueue

[setnull_as_end_cdr]
REGEX=^\d*\.\d{4}-\d{6},,End\b
DEST_KEY=queue
FORMAT=nullQueue

Any advice to get this working would be very much appreciated. Also, I know that the number of fields is excessive; sadly most of the fields are used and each indicates something we're going to need to create reports off of.

Here is an example file containing the line types i'm trying to get rid of: Test.csv:

version=14.9 encoding=US-ASCII
132883a12ac920101115183000.1400+000000,,Start
132983a12ac920101115184500.1100+000000,,End
0 Karma

southeringtonp
Motivator

At a quick look, there are some issues with your regex.

The sample data has '+' characters, but the regexes listed have '-'. Also, the initial characters include hex digits, which '\d' will not match.

props.conf:

# Replaces both setnull_as_start_cdr and setnull_as_end_cdr
[setnull_as_startend_cdr]
REGEX=^[0-9a-f]+\.\d{4}\+\d{6},,(Start|End)$
DEST_KEY=queue
FORMAT=nullQueue

transforms.conf:

[AS-CDR]
TRANSFORMS-null029=setnull_as_startend_cdr
0 Karma

MillerTime
Splunk Employee
Splunk Employee

your bold titles are confused -- the first block of config (starting with # Replaces) should be "transforms.conf" and the second should be "props.conf"

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...