Data Format Transforamtion

[kkamatchisundar]

Hello All

I am just started using Splunk for my project and very new to it .
In my project, there is a requirement to receive the data from different sources and convert them into common format and store into our external stoage (db or hadoop)

I read the Splunk docs. In that there is feature called transform. So can anyone please provide few example on this

Raw format : DateTime Description Event Type Source Count
Desired format : DateTime Source Type Event Description Count

Replace the default value if any value is missing.

Also is there any way to store the received data into exernal storage system (db or hadoop).Instead store into Splunk database.

Also is there any way to avoid the indexing on received data?

So please anyone help me on that

Thanks in Advance

[lukejadamec]

1) In your example there is no need to use transform to change the order of the data. Once the data is indexed and the fields extracted, you would create searches on the data and output the data in your desired order.

2) Replacing empty value fields is typically done at search time. You can use the fillnull function to specify the value to substitute for null value fields. You can use the eval function to change fields that contain values.

3) There are ways to write data to databases from Splunk. See this post: http://answers.splunk.com/answers/55134/automatically-forward-splunk-data-to-database

4) No, if Splunk does not index the data, then there is no way to export or search the data. If there is data that you do not want to index because it is ‘useless’ then there are ways to drop that data before it is indexed.

[ShaneNewman]

Raw format : DateTime Description Event Type Source Count
Desired format : DateTime Source Type Event Description Count

| table DateTime Source Type Event Description Count

Replace the default value if any value is missing.

|fillnull value=your_value field=your_field

Also is there any way to store the received data into exernal storage system (db or hadoop).Instead store into Splunk database.

Yes, you can install the DBX app and do SQL inserts. The method for doing this, without indexing, is tricky. You essentially use Splunk as an ETL tool.

Also is there any way to avoid the indexing on received data?

Yes, if you just want to "not index" certain data, and don't need to move it somewhere... you can send it to the nullqueue via the props.conf file.