All Apps and Add-ons

How can one get Google BigQuery data in Splunk?

Explorer

Hello ,

How can one get Google Big Query data in Splunk?
At the moment we use a script that extracts necessary parameters out of Google Analytics into a csv file that is then indexed in Splunk.

Should we use the Big Query API and REST API Modular Input (https://apps.splunk.com/app/1546/) ?

If yes, can you give me advice on how to configure this connection?

Thank you.

Splunk Employee
Splunk Employee

There's a new Add-on that might be of interest:

Splunk-GABQ-Addon on GitHub
https://github.com/AusDTO/Splunk-GABQ-Addon

0 Karma

Thank you for the share. This code was a great head start.

I've been able to harvest the majority of the 50 fields.

I've got a few that giving me grief.
Here's one: messageinfo.triggeredrule_info.consequence.action - holds an integer

focused in this code

def buildStruct(dataRow, fieldList):
                        returnVal = {}
                        for field in fieldList:
                                ew.log(EventWriter.INFO, "Debug field: %s " % field)
                                if '.' in field:
                                        record, recfield = field.split(".", 1)
                                        ew.log(EventWriter.INFO, "Debug datarow record: %s and recfield %s " % (dataRow,  recfield))
                                        if len(dataRow) != 0:
                                                returnVal[field] = buildStruct(dataRow[record], [recfield])
                                else:
                                        if type(dataRow) is list:
                                                ew.log(EventWriter.INFO, "Debug hit the dataRow is a list " )
                                                ew.log(EventWriter.INFO, "Debug dataRow: %s " % dataRow)
                                                if len(dataRow) == 1:
                                                        returnVal[field] = dataRow[0][field]
                                                else:
                                                        returnVal[field] = []
                                                        for x in dataRow:
                                                                ew.log(EventWriter.INFO, "Thayne Debug dataRow: %s " % x)
                                                                returnVal[field].append(x[field])
                                        else:
                                                returnVal[field] = dataRow[field]
                        return returnVal

Debug datarow record: [{u'consequence': [{u'action': u'17', u'subconsequence': [], u'reason': u'Triggered by CONTENTCOMPLIANCE rule. Rule description: GA GS Keyword Test'}], u'spamlabelmodifier': None, u'policyholderaddress': u'dipak.samanta@dev.megadiamond.com', u'stringmatch': [{u'predefineddetectorname': None, u'matchedstring': u'\nInternal use only\r', u'type': u'1', u'source': u'1', u'matchexpression': u'(?i)(\W|^)(Not\sfor\sDistribution|Do\sNot\sDistribute|Internal\sUse\sOnly|IUO|Confidential|(?i', u'attachmentname': None}, {u'predefineddetectorname': None, u'matchedstring': u'GS\r', u'type': u'1', u'source': u'1', u'matchexpression': u'(?i)(\W|^)(Gold\s*Story|GS|Gold)(\W|$)', u'attachmentname': None}], u'rulename': u'GA GS Keyword Test', u'ruletype': u'8'}] and recfield action

/splunk/etc/apps/GoogleAnalyticsBQ/bin/gabq.py" File "/opt/splunk/etc/apps/GoogleAnalyticsBQ/bin/gabq.py", line 104, in buildStruct
07-27-2018 18:27:42.515 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/GoogleAnalyticsBQ/bin/gabq.py" returnVal[field] = buildStruct(dataRow[record], [recfield])

07-27-2018 18:27:42.515 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/GoogleAnalyticsBQ/bin/gabq.py" TypeError: list indices must be integers, not str

0 Karma

Path Finder

Has anyone tried this add-on? We're interested in querying bigquery tables similar to how we'd use dbconnect (without having to index the data).

0 Karma