topic Re: formatted CDR files in Splunk Search

formatted CDR files

MaryCampbell — Wed, 09 Jan 2013 16:31:32 GMT

Hi
I am very new to Splunk but have been asked to look into the possibility to using Splunk to replace an existing system used query our CDR files.

From what I can tell Splunk has most of what we would need.
The one thing I cannot seem to find is:
Our files are comma separated but follow a pattern similar to ANumber,BNumber,deliveryDate,etc.
Our searches would be :
return all files where ANumber = ? and BNumber = ? and deliveryDate > ? etc

Our current system collects the files and inserts them into a database. It is told which field is what and creates a column based on this. The querys are then created to match the columns.

I cannot see a way of collecting files in Splunk where you can index the files by telling it a format field1,field2 etc

I hope this all makes sense. I am sure there is a way but that I am unable to see it.

Thanking you in advance
Mary

Re: formatted CDR files

DaveSavage — Wed, 09 Jan 2013 16:52:42 GMT

Mary - the good news is that this is possible. You would probably split the fields out (even using the GUI) to create a Regex, defining the field. Searches are easily constructed for ANumber etc.
If volume isn't going to cause you a problem with your license (CDRs are big, I know) then you could index the lot. Defining a new index would be advisable.
For Regex I'll post the links, ditto the tutorial if they are useful?
Br
D

Re: formatted CDR files

DaveSavage — Wed, 09 Jan 2013 16:57:10 GMT

Splunk tutorial: http://docs.splunk.com/Documentation/Splunk/latest/Tutorial/WelcometotheSplunktutorial
Exploring Splunk: http://www.splunk.com/goto/book a really good read.

Re: formatted CDR files

DaveSavage — Wed, 09 Jan 2013 16:59:18 GMT

...and this is unmissable 😉
http://www.youtube.com/watch?v=Yf5gTNiotnM

Re: formatted CDR files

sideview — Wed, 09 Jan 2013 21:54:36 GMT

What system are your CDR files from? Sideview makes apps for both Cisco CallManager's CDRs and Shoretel CDR. And if you can send us sample logs we may very well be able to expand into your particular product. 😃 If you can help us with sample data to get a new app started we will happily grant you a free perpetual license to use the resulting product.

Re: formatted CDR files

lguinn2 — Thu, 10 Jan 2013 10:53:44 GMT

To build on Dave's answer:

Actually, if the fields are comma-separated, there is an even easier way to tell Splunk how to identity them. Assume that you create a sourcetype called CDR for your data.

In props.conf

[CDR]
TRANSFORM-ecf1=extract-CDR-fields

In transforms.conf

[extract-CDR-fields]
DELIMS = ","
FIELDS = ANumber,BNumber,deliveryDate,etc.

These conf files can be put in $SPLUNK_HOME/etc/system/local

As Dave pointed out, you should create a separate index for your CDR data. I would suggest this:

Create the new index.
Set up the props.conf and transforms.conf files
Upload some sample CDR into the index. Be sure to specify the CDR sourcetype manually (just type it in) and choose your new index. This can all be done from the Splunk GUI.
Play with the data. If it looks wrong, use the clean command to clean out your index and try again.
When the data looks right - clean out the index one more time and then start indexing your real data!

Re: formatted CDR files

DaveSavage — Thu, 10 Jan 2013 13:42:22 GMT

Nicee ;-)...thanks LG

Re: formatted CDR files

DaveSavage — Thu, 10 Jan 2013 13:53:18 GMT

I like the sound of that, and concept sideview 😉
@Mary as you are fairly new to this, blitz through the Exploring Splunk (Splunk Query Language) manual then go play with the data. Quite quickly and with only a v small amount of effort you will have top numbers calling, those called, duration etc. Interested in cost, by department?! Next stop, reference a tariff table, extend your searches and pipe the results to graphs and tables. I appreciate that the sound of all this may have you running for the hills...but I suspect not.
Myabe Call Loggers are dead. Think of the savings.