All Apps and Add-ons

Indexing an Avro file

trad511
New Member

Does anyone have experience indexing an Avro file?

I have Avro data stored in HDFS, but have been unable to find a good way to have Splunk read the binary Avro format without using custom code or other transforms.

Before I go down the custom code path figured I'd ask for other's experiences.

I have tried:
* HadoopConnector - Underlying hadoop fs commands (at least on my CDH4 system) return binary data.
* Flume - Doesn't seem to be able to read in an Avro file source.
* Hue + Splunk w/ http GET - this gets ugly quickly and is super inefficient.

Thanks

0 Karma

sumanssah
Communicator

Install Avro Tools:

wget 'https://archive.apache.org/dist/avro/avro-1.7.5/py/avro-1.7.5.tar.gz'
tar xvf avro-1.7.6.tar.gz
cd avro-1.7.6
sudo python setup.py install

Install Avro Tools using PIP:

pip install avro

Conversion from Avro to json

avro cat "/avro_file_path/*.avro" -- format json >"output_file_path/output.json"

Data input >> Files & Directories >> Moniter "output_file_path/output.json"

0 Karma

barakreeves
Splunk Employee
Splunk Employee

Sorry for the late reply but this is what I have found...you may already know this:

There are tools that will convert the Avro binary format to text:
http://avro.apache.org/docs/1.4.1/api/java/org/apache/avro/tool/package-summary.html
-- BinaryFragmentToJsonTool
-- ToTextTool
-- DataFileReadTool

The link below is a good blog article discussing this
http://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/

Avro is not the only tool that write to binary format which then needs a converter (or translator) to make it readable in Splunk; some network devices that Splunk monitors act similarly. The point being is that you are not alone.

Hopefully these built in tools will work for you so you don't have to write and support code.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...