Getting Data In

Single log to multiple indexes

aaronnicoli
Path Finder

Hi there,

I am in the process of planning a roll out of splunk to our network, however, I am stuck on the indexes. I understand the role of an index is to provide different datasets with different retention periods and control the way said data flows through it's hot/warm/cold/frozen life cycle.

However, the issue that I have is simple, I have log files that contain multiple event types within the same log file, meaning I wish to have varying events from one log file be separated and placed into different indexes.

ie. log_file - contains log data for ftp and system events.

I wish to have events within this file following the ftp event style indexed to index_ftp. Also, I wish for events with any other event style to be indexed into index_system.

Is this possible to achieve? If it is, they how does one create such a configuration?

Thanks, Aaron.

Tags (2)
0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

Yes it is, but do you need these in different indexes? Do you in fact require different retention policies for different lines in the same log file? It sounds to me like these are simply different sourcetypes, or even just different (dynamically defined) event types.

Update:

You can do this: http://www.splunk.com/base/Documentation/latest/Admin/Routeandfilterdata#Filter_event_data_and_send_...

Except that you set the "index" key to send to the desired index, rather than the "queue" key.

View solution in original post

0 Karma

gfriedmann
Communicator

I'm thinking about using this technique to parse out massive firewall-permit events and Windows Security log events into their own indexes. I think this will allow for different retention times AND maybe speed up searches for terms that are not in these indexes.

In essence, i don't want all my crazy security auditing to impact more operationally interesting log messages.

Aha! This page seems to be exactly what is called for: http://www.splunk.com/base/Documentation/latest/Admin/Setupmultipleindexes#Route_events_to_specific_...

Update: IF you do this for a TCP source, like me, the stanza for a tcp source in props.conf is "source::tcp:5144" , not "source::tcp://5144" which was my first attempt.

0 Karma

aaronnicoli
Path Finder

Sweet, thank you for your input.
Very appreciated.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Yes it is, but do you need these in different indexes? Do you in fact require different retention policies for different lines in the same log file? It sounds to me like these are simply different sourcetypes, or even just different (dynamically defined) event types.

Update:

You can do this: http://www.splunk.com/base/Documentation/latest/Admin/Routeandfilterdata#Filter_event_data_and_send_...

Except that you set the "index" key to send to the desired index, rather than the "queue" key.

0 Karma

aaronnicoli
Path Finder

I kind of agree with you, this is a request from higher up 🙂
Anyway, thanks so much for the help, at least I can now say yes it is doable but not desired or for that matter required.. 😛
Thanks again,

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

okay so you do need different indexes, though i guess i'm surprised that you have different log data in the same file, and i'm not sure how you plan to identify what is security-related vs not.

0 Karma

aaronnicoli
Path Finder

Basically we are trying keep certain events (or line items) from individual RHEL system logs for different periods.

ie. so that we can specify anything security related within the log be kept for 52weeks and anything else be kept for only 4 weeks.

What would you say is the best way of going about this process?

Aaron.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...