All Apps and Add-ons

GCP Billing CSV Not Parsing Correctly

jamin358
Explorer

This is more of a bug report than a question.
I am working with the Billing data coming from our new GCP environment and have found that CSV data where a string contains a comma is not being parsed properly.
I have discovered this from data where the Credit1 field says "External IPs will not be charged until April 1, 2020." It splits the row into two columns, pushing all data out by one and therefore losing the Description Field value from that event.

The data looks like this is Splunk
*{ [-]
Account ID: 111111-222222-333333
Cost: GBP
Credit1: "External IPs will not be charged until April 1
Credit1 Amount: 2020."
Credit1 Currency: -0.156831
Currency: 0.156831
Description:
End Time: 2020-02-20T00:00:00-08:00
Line Item: com.google.cloud/services/compute-engine/ExternalIp
Measurement1: com.google.cloud/services/compute-engine/ExternalIp
Measurement1 Total Consumption: 183661
Measurement1 Units: seconds
Project: 12344566778
Project ID: 12344566778
Project Labels: project
Project Name: project
Project Number: GBP
Start Time: 2020-02-19T00:00:00-08:00
} *

This data parsing isnt done in the props of the sourcetype but in the python code. Line 260 of Splunk_TA_google-cloudplatform/bin/splunk_ta_gcp/modinputs/billing.py
I'm not an expert in GCP data, so I dont know what the best course of action would be to fix this bug without breaking other things.

My assumption is that no data values will have "word,word" but "word, word" with a space, so would the fix here to change the split from being just a comma to a comma and no space afterwards. ie in python line = line.split(',[^\s]')

0 Karma
Get Updates on the Splunk Community!

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...

SOCin’ it to you at Splunk University

Splunk University is expanding its instructor-led learning portfolio with dedicated Security tracks at .conf25 ...

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...