Group data forwarded by a list of hosts into different buckets

[mhessick]

I've got 4 splunk instances running, with 3 light forwarders sending application logs to my main 'server' instance (i've configured this via forwarding in the management console of my server instance)

It's working great, but I need some way to group or separate the incoming data into different buckets of logs.

For example, I'd like to have the logs from my collection of development environments going into a development index, that only the development user is allow to see and search.

I want to do the same thing for a collection of other environments and users.

What's the easiest approach to this? The buckets are qualified by the hostnames the logs are coming from.

Thanks

[araitz]

Set up different indexes (for example 'dev'):

http://www.splunk.com/base/Documentation/4.1.6/Admin/Setupmultipleindexes

Specify what index data should go to what index via inputs.conf on the forwarder:

http://www.splunk.com/base/Documentation/4.1.6/Admin/Inputsconf

Example:

[monitor:///path/to/devfiles/]
sourcetype=foo
index=dev

Alternately, you can control the destination index on the indexer if the data is coming from a lightweight forwarder. You can do this OR you can set the index on the forwarder - you do not need to do both.

http://www.splunk.com/base/Documentation/4.1.6/Admin/Propsconf:

[host::some_dev_host]
TRANSFORMS-foo=route_to_dev_index

http://www.splunk.com/base/Documentation/4.1.6/Admin/Transformsconf

[route_to_dev_index]
REGEX=.
DEST_KEY=_MetaData:Index
FORMAT=dev

Create roles and constrain the roles to the proper indexes:

http://www.splunk.com/base/Documentation/4.1.6/Admin/Addusersandassignroles#Add_and_edit_roles_using...

Add users to those roles, and you are done.

[araitz]

Great! You can do it on the server too, I will add that example.

[mhessick]

This works great. I had previously specified the indexes on the server (because the indexes only existed there)

When I changed that and specified the indexes on the forwarders, everything worked.

Thanks a ton.

-Mike

[mhessick]

As long as the aggregation of the incoming logs doesn't go into 1 big searchable index . . .that's kind of what i'm looking for. I want to assign a role to be able to view only a specific index and direct incoming logs from a set of hostnames to a specified index.

[mhessick]

A bucket is just a generic term some place that's separate from other places. I suppose either a separation at search time or a different index would both work.

As long as users could be restricted from viewing an index or logs . . . by their role.

[Simon_Shelston]

Can you describe you use case a bit more? What do you mean by buckets? Do you need different indexes for this data? Or do you just need to have separation of data at search time?