Getting Data In

Filtering events from Hadoop unstructured data

sansri7680
Path Finder

I am trying to read log files from Hadoop cluster. These are unstructured files which otherwise can be filtered after indexing using Regex searches. But my input data is huge and the throughput requirement is also very high. The result is only a small portion of the input. Hence is it possible to filter the input data before being indexed by Hunk so that I can avoid searching unnecessary data

Tags (3)
0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

Currently Hunk optimizes data access if the data is partitioned and Hunk is properly configured to recognize those partitions. Two types of partitioning exist: (a) time based, this is when the data is structured hierarchically using some time partitioning and (b) field based partitioning.

For example if your data is organized as follows

/some/path/20140108/server1/...
/some/path/20140108/server2/...
/some/path/20140109/server1/...
/some/path/20140109/server2/...

You can configure Hunk to recognize the third segment in the path as the data and the fourth segment as the server field. You can look at the details of how to do that here

Currently Hunk does not have the ability to optimize data access based on the file content, because we don't create an index - we just access/process the data in it's raw form.

Does this help?

0 Karma
Get Updates on the Splunk Community!

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more with ITSI’s ...

Accelerate Service Onboarding, Decomposition, Troubleshooting - and more! Faster Time to ValueManaging and ...

New Release | Splunk Enterprise 9.3

Admins and Analyst can benefit from:  Seamlessly route data to your local file system to save on storage ...

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...