topic Re: Group data forwarded by a list of hosts into different buckets in Getting Data In

Group data forwarded by a list of hosts into different buckets

mhessick — Thu, 30 Dec 2010 07:46:39 GMT

I've got 4 splunk instances running, with 3 light forwarders sending application logs to my main 'server' instance (i've configured this via forwarding in the management console of my server instance)

It's working great, but I need some way to group or separate the incoming data into different buckets of logs.

For example, I'd like to have the logs from my collection of development environments going into a development index, that only the development user is allow to see and search.

I want to do the same thing for a collection of other environments and users.

What's the easiest approach to this? The buckets are qualified by the hostnames the logs are coming from.

Thanks

Re: Group data forwarded by a list of hosts into different buckets

Simon_Shelston — Thu, 30 Dec 2010 07:53:20 GMT

Can you describe you use case a bit more? What do you mean by buckets? Do you need different indexes for this data? Or do you just need to have separation of data at search time?

Re: Group data forwarded by a list of hosts into different buckets

mhessick — Thu, 30 Dec 2010 08:04:34 GMT

A bucket is just a generic term some place that's separate from other places. I suppose either a separation at search time or a different index would both work.

As long as users could be restricted from viewing an index or logs . . . by their role.

Re: Group data forwarded by a list of hosts into different buckets

mhessick — Thu, 30 Dec 2010 08:12:05 GMT

As long as the aggregation of the incoming logs doesn't go into 1 big searchable index . . .that's kind of what i'm looking for. I want to assign a role to be able to view only a specific index and direct incoming logs from a set of hostnames to a specified index.

Re: Group data forwarded by a list of hosts into different buckets

araitz — Thu, 30 Dec 2010 08:31:54 GMT

Set up different indexes (for example 'dev'):

http://www.splunk.com/base/Documentation/4.1.6/Admin/Setupmultipleindexes

Specify what index data should go to what index via inputs.conf on the forwarder:

http://www.splunk.com/base/Documentation/4.1.6/Admin/Inputsconf

Example:

[monitor:///path/to/devfiles/]
sourcetype=foo
index=dev

Alternately, you can control the destination index on the indexer if the data is coming from a lightweight forwarder. You can do this OR you can set the index on the forwarder - you do not need to do both.

http://www.splunk.com/base/Documentation/4.1.6/Admin/Propsconf:

[host::some_dev_host]
TRANSFORMS-foo=route_to_dev_index

http://www.splunk.com/base/Documentation/4.1.6/Admin/Transformsconf

[route_to_dev_index]
REGEX=.
DEST_KEY=_MetaData:Index
FORMAT=dev

Create roles and constrain the roles to the proper indexes:

http://www.splunk.com/base/Documentation/4.1.6/Admin/Addusersandassignroles#Add_and_edit_roles_using_Splunk_Web

Add users to those roles, and you are done.

Re: Group data forwarded by a list of hosts into different buckets

mhessick — Fri, 31 Dec 2010 04:11:54 GMT

This works great. I had previously specified the indexes on the server (because the indexes only existed there)

When I changed that and specified the indexes on the forwarders, everything worked.

Thanks a ton.

-Mike

Re: Group data forwarded by a list of hosts into different buckets

araitz — Fri, 31 Dec 2010 05:28:32 GMT

Great! You can do it on the server too, I will add that example.