Getting Data In

Splunk as a generic parsing utility

amethon
Engager

We provide mobile data analytics reporting to mobile operators and we are increasingly being asked to take input in the form of proprietary transaction logs from a range of vendor solutions e.g. WAP gateways, mobile data optimization nodes, Cisco RDRs, DPI nodes, passive probes, etc.

Our solution has a standard input log format so we are having to write parsing scripts to convert these different 3rd party logs to our format.

Can we use Splunk as a generic parsing tool to facilitate the ingestion of all of these different logging formats?

Thanks,

Michael Stone
Amethon Solutions

Tags (2)

kristian_kolb
Ultra Champion

Well, I'd assume that if the logs in question are text files - then yes - that is pretty much what Splunk does. It will read and index events from any text file, and allow you to run searches, produce reports etc.

Bear in mind that you would most likely have to write your own "parsing" code in order to fully benefit from using Splunk. E.g. Splunk will happliy index events in a proprietary format like;

2011-09-22 09:11:34,431 BOBBY RRR 000988:2231 11,22 ZZZZ <VFV>,8888 LPL/GTG

but you'd probably want to make that data into more usable information by "parsing" it, so that you interpret "BOBBY" as the hostname generating the logs, "RRR" as the type of device, "000988:2231" as the transactionID, "11,22" as the kB transferred, "ZZZZ" as cell tower ID etc etc etc.

This is not hard to do, as long as the structure of events from a (type of) file is fairly consistent.

This allows to search for events and produce reports such as "all customers in the Toronto area making more than 5 phone calls per hour on average" or "total bandwitdh used for Twitter messages from iPhones last week". As long as that kind of information is in the logs. Mathematical/statistical operations can be performed on any numeric data.

There is (as far as I know) no hard limit in the amount of different types of log a splunk instance can handle, so ... well I could go on and on... you should just download the Trial version for free and try it out.

Familiarize yourself with the concept of 'sourcetypes' and 'field extraction' and you should be on your way of making sense of the logs.

UPDATE: I re-read your question and realized that you weren't looking for an alternative to your existing solution. I'm not too familiar with using Splunk for that purpose, but it may be possible, although Splunk is not really a file conversion tool.

Hope this helps,

Kristian

0 Karma

kristian_kolb
Ultra Champion

I have not used Splunk for that purpose, and I am not sure if you could do that. I updated my previous answer to reflect this.

Sorry if I gave you false hopes. Hopefully somebody more knowledgeable will give you a better answer. However, you could always start using Splunk instead of your existing solution... 🙂

/Kristian

0 Karma

amethon
Engager

Thanks for the answer Kristian!

Once the logs have been indexed, can they then be written to a new flat file format i.e. our standard log format?

FYI, have downloaded the demo and have been playing with the data input and field extraction features.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...