topic Re: dispatch logs incoming from one source to multiple indexes in Getting Data In

dispatch logs incoming from one source to multiple indexes

AdrienW — Tue, 18 Dec 2012 13:25:07 GMT

Dear all,

I have syslogs of several firewalls incoming from a TCP port.

I would like to dispath the logs of each firewall (by their SN) in their proper index.

I saw it's possible to do that configuring transforms.conf and props.conf

Here is the example I found :

props.conf

[syslog]

TRANSFORMS-syslog-NSM=syslog-NSM

transforms.conf

[syslog-NSM]

DEST_KEY = _MetaData:Index

REGEX = NSM

FORMAT = index1

Does someone can adapt it to fit with my requirements?
I really don't know how to do it...

my index : index1 // index2 // ...
SN : FT001 // FT002 // ....

Thanks !

Re: dispatch logs incoming from one source to multiple indexes

kristian_kolb — Wed, 19 Dec 2012 09:56:14 GMT

That looks about correct, but your current REGEX will just match the literal string NSM in your event. I believe that you can set up a single transform, assuming your data looks something like

2012-12-12 12:12:12,12 SN=blah1 action=allow src=1.2.3.4 dst=2.3.4.5 ...
2012-12-13 13:13:13,13 SN=blah2 action=deny src=3.4.5.6 dst=1.0.0.2 ...

props.conf

[your_sourcetype]
TRANSFORMS-set_index = fw_index

transforms.conf

[fw_index]
DEST_KEY = _MetaData:Index
REGEX = \s+SN=(\S+)\s
FORMAT = $1

Thus the first example event would land in the index blah1, and the second in index blah2, if they exist - otherwise they'll end up in main.

Have a look at the examples in the docs for transforms.conf and for managing indexes.

UPDATE:

Then you'll have to hard-code it, with several transforms statements from props.conf, i.e.;

props.conf

[your sourcetype]
TRANSFORMS-fw_index = banana, apple, pear

and in transforms.conf

[banana]
DEST_KEY = _MetaData:Index
REGEX = FGT1234567
FORMAT = banana

[apple]
DEST_KEY = _MetaData:Index
REGEX = FGT45453324
FORMAT = apple

Hope this helps,

Kristian

Re: dispatch logs incoming from one source to multiple indexes

AdrienW — Thu, 03 Jan 2013 14:44:45 GMT

Thanks for your answer, I'm near to resolve this, but I need help one more time.

My logs are like this :

Jan  3 15:32:52 10.0.1.254 date=2013-01-03 time=15:03:19 devname=FGT60B-CFP device_id=FGT60B3908672004
Jan  3 15:38:21 10.0.1.254 date=2013-01-03 time=15:36:43 devname=FGT60B-EDC device_id=FGT60B3908668256

As you can see, there is a special SN for each hardware and I need to put it in index like "banana" and "apple" (common names).

Is it possible to add a condition in the regex or anything else that means :

FGT60B3908672004 => banana
FGT60B3908668256 => apple

all logs have the same source and sourcetype, and I can't take the host value to dispatch via the props.conf.

Re: dispatch logs incoming from one source to multiple indexes

kristian_kolb — Thu, 03 Jan 2013 16:32:02 GMT

see update above

Re: dispatch logs incoming from one source to multiple indexes

NewMilenium — Fri, 04 Jan 2013 13:08:36 GMT

Hello,

first, thank you for your answers, Kristian.
I am quite new to splunk; I did what you said, and still it won't work.
The situation is; I've got 2 different UDP sources coming to one port, let's say P1. I must separate the 2 sources so that each one goes to its index.
Here is what i've done so far;

1) created the 2 indexes "banana" and "apple",

2) created the 2 files "props.conf" and "transforms.conf" in /opt/splunk/etc/system/local ,

3) put the configuration above you talked about with the right serial numbers (and only that, assuming the default configs props.conf and transforms.conf in opt/splunk/etc/system/default are still used before overrided with those 2 files)

4) edited "inputs.conf" in /opt/splunk/etc/system/local which had

[default]

host = localhost

into

[default]

host = localhost

[udp]

connection_host=ip

queue = parsingqueue

because I saw here this could be necessary (and it wasn't working anyway) and that in the "default" inputs.conf, there was connection_host=ip .

I am still working on it, and cannot find what's wrong. Maybe because I can't avoid to precise what index is used in Manager>Data inputs>UDP>P1 (which is set to "main").

Re: dispatch logs incoming from one source to multiple indexes

kristian_kolb — Fri, 04 Jan 2013 13:28:24 GMT

First of all, you have no port specification in the udp stanza header. It should be like;

[udp://514]

http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitornetworkports#Add_a_network_input_using_inputs.conf

On a side note, if you have control over the sending parties, you could set them to send you data on different ports and specify the indexes in inputs.conf directly.

[udp://514] index=banana

[upd://515]
index=apple

Default values are just that. You don't need to set them again.

Re: dispatch logs incoming from one source to multiple indexes

NewMilenium — Fri, 04 Jan 2013 14:19:09 GMT

Well, I changed "local" inputs.conf accordingly, and logs are still put into "main" , sadly.
We don't have control over the sending parties; the order is "only one port : 514". That's why I/we must differentiate each log using the device_id.

Re: dispatch logs incoming from one source to multiple indexes

kristian_kolb — Mon, 07 Jan 2013 18:27:55 GMT

are you sure that your regexes work correctly?

Re: dispatch logs incoming from one source to multiple indexes

NewMilenium — Wed, 09 Jan 2013 14:36:57 GMT

No, I'm not sure...

Here are the files involved.

/opt/splunk/etc/system/local/props.conf :

[fortigate]

TRANSFORMS-fw_index = apple, banana

And here is the /opt/splunk/etc/system/local/transforms.conf :

[apple]

DEST_KEY = _MetaData:Index

REGEX = FG100C3G09619593

FORMAT = index::apple

[banana]

DEST_KEY = _MetaData:Index

REGEX = FGT50B3G10612115

FORMAT = index::banana

There are other files that were already here before I touch anything, in /opt/splunk/etc/system/local : alert_actions.conf, authorize.conf, eventtypes.conf, inputs.conf, migration.conf, README, server.conf, tenants.conf, web.conf. I prefer to mention it because "local" folder is supposed to contain files "adding and replacing" the default ones.

You can notice I changed [fortigate_traffic] to [fortigate]. I did, because this is the only way I managed to finally get an error message. Well, two, obviously, written in black on a yellow font up the splunk window. Here they are :

received event for unconfigured/disabled/deleted index='index::apple' with source='source::udp:514' host='host::10.0.1.254' sourcetype='sourcetype::fortigate_traffic' (2 missing total)

received event for unconfigured/disabled/deleted index='index::banana' with source='source::udp:514' host='host::10.0.1.254' sourcetype='sourcetype::fortigate_traffic' (1 missing total)

And yes, I created the 2 indexes called "apple" and "banana" in splunk, with "Manager>Indexes>New".

I seriously can't find what's wrong...

Re: dispatch logs incoming from one source to multiple indexes

kristian_kolb — Wed, 09 Jan 2013 15:02:41 GMT

Oops, I may have made a small mistake; it seems like the correct format for FORMAT is , NOT index::

http://docs.splunk.com/Documentation/Splunk/5.0.1/Indexer/Setupmultipleindexes

Sorry about that.

Re: dispatch logs incoming from one source to multiple indexes

NewMilenium — Wed, 09 Jan 2013 15:09:55 GMT

Oh my, it works. Wonderful job, Kristian. Thanks so much!