Getting Data In

Is there a way to index Avro files in Splunk?

djain
Path Finder

Hi Splunkers,

I am in an odd pickle here.

So, I am ingesting data from Amazon Web Services (AWS) to my Splunk instance. I am able to see the data flowing in but the data is encrypted .avro, which from my knowledge, does not work with Splunk. Is there an add on or another way to decrypt and index these Avro files like JSON?

I apologize in advance if this question feels repetitive. I didn't find another answer for it.

Tags (4)
0 Karma

hramos_splunk
Splunk Employee
Splunk Employee

There is not an add-on that can natively ingest Avro files. Splunk is not able to read it because .avro files are in a binary format that Splunk can't read.

Alternatively, you can change the format to a text format prior to ingestion in order for Splunk to read it. For more information in this approach please check the answers post link below:

https://answers.splunk.com/answers/83891/indexing-an-avro-file.html

The example made is as follow:

Avro to “json” conversion:

Install Avro Tools:

wget 'https://archive.apache.org/dist/avro/avro-1.7.5/py/avro-1.7.5.tar.gz'
tar xvf avro-1.7.6.tar.gz
cd avro-1.7.6
sudo python setup.py (https://setup.py/) install

Install Avro Tools using PIP:

pip install avro

Conversion from Avro to json

avro cat "/avro_file_path/*.avro" -- format json >"output_file_path/output.json"

Data input >> Files & Directories >> Moniter "output_file_path/output.json"

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...