Dear all,
I have syslogs of several firewalls incoming from a TCP port.
I would like to dispath the logs of each firewall (by their SN) in their proper index.
I saw it's possible to do that configuring transforms.conf and props.conf
Here is the example I found :
props.conf
[syslog]
TRANSFORMS-syslog-NSM=syslog-NSM
transforms.conf
[syslog-NSM]
DEST_KEY = _MetaData:Index
REGEX = NSM
FORMAT = index1
Does someone can adapt it to fit with my requirements?
I really don't know how to do it...
my index : index1 // index2 // ...
SN : FT001 // FT002 // ....
Thanks !
No, I'm not sure...
Here are the files involved.
/opt/splunk/etc/system/local/props.conf :
[fortigate]
TRANSFORMS-fw_index = apple, banana
And here is the /opt/splunk/etc/system/local/transforms.conf :
[apple]
DEST_KEY = _MetaData:Index
REGEX = FG100C3G09619593
FORMAT = index::apple
[banana]
DEST_KEY = _MetaData:Index
REGEX = FGT50B3G10612115
FORMAT = index::banana
There are other files that were already here before I touch anything, in /opt/splunk/etc/system/local : alert_actions.conf, authorize.conf, eventtypes.conf, inputs.conf, migration.conf, README, server.conf, tenants.conf, web.conf. I prefer to mention it because "local" folder is supposed to contain files "adding and replacing" the default ones.
You can notice I changed [fortigate_traffic] to [fortigate]. I did, because this is the only way I managed to finally get an error message. Well, two, obviously, written in black on a yellow font up the splunk window. Here they are :
received event for unconfigured/disabled/deleted index='index::apple' with source='source::udp:514' host='host::10.0.1.254' sourcetype='sourcetype::fortigate_traffic' (2 missing total)
received event for unconfigured/disabled/deleted index='index::banana' with source='source::udp:514' host='host::10.0.1.254' sourcetype='sourcetype::fortigate_traffic' (1 missing total)
And yes, I created the 2 indexes called "apple" and "banana" in splunk, with "Manager>Indexes>New".
I seriously can't find what's wrong...
Oh my, it works. Wonderful job, Kristian. Thanks so much!
Oops, I may have made a small mistake; it seems like the correct format for FORMAT is
, NOT index::
http://docs.splunk.com/Documentation/Splunk/5.0.1/Indexer/Setupmultipleindexes
Sorry about that.
Hello,
first, thank you for your answers, Kristian.
I am quite new to splunk; I did what you said, and still it won't work.
The situation is; I've got 2 different UDP sources coming to one port, let's say P1. I must separate the 2 sources so that each one goes to its index.
Here is what i've done so far;
1) created the 2 indexes "banana" and "apple",
2) created the 2 files "props.conf" and "transforms.conf" in /opt/splunk/etc/system/local ,
3) put the configuration above you talked about with the right serial numbers (and only that, assuming the default configs props.conf and transforms.conf in opt/splunk/etc/system/default are still used before overrided with those 2 files)
4) edited "inputs.conf" in /opt/splunk/etc/system/local which had
[default]
host = localhost
into
[default]
host = localhost
[udp]
connection_host=ip
queue = parsingqueue
because I saw here this could be necessary (and it wasn't working anyway) and that in the "default" inputs.conf, there was connection_host=ip .
I am still working on it, and cannot find what's wrong. Maybe because I can't avoid to precise what index is used in Manager>Data inputs>UDP>P1 (which is set to "main").
are you sure that your regexes work correctly?
Well, I changed "local" inputs.conf accordingly, and logs are still put into "main" , sadly.
We don't have control over the sending parties; the order is "only one port : 514". That's why I/we must differentiate each log using the device_id.
First of all, you have no port specification in the udp stanza header. It should be like;
[udp://514]
On a side note, if you have control over the sending parties, you could set them to send you data on different ports and specify the indexes in inputs.conf directly.
[udp://514]
index=banana
[upd://515]
index=apple
Default values are just that. You don't need to set them again.
Thanks for your answer, I'm near to resolve this, but I need help one more time.
My logs are like this :
Jan 3 15:32:52 10.0.1.254 date=2013-01-03 time=15:03:19 devname=FGT60B-CFP device_id=FGT60B3908672004
Jan 3 15:38:21 10.0.1.254 date=2013-01-03 time=15:36:43 devname=FGT60B-EDC device_id=FGT60B3908668256
As you can see, there is a special SN for each hardware and I need to put it in index like "banana" and "apple" (common names).
Is it possible to add a condition in the regex or anything else that means :
FGT60B3908672004 => banana
FGT60B3908668256 => apple
all logs have the same source and sourcetype, and I can't take the host value to dispatch via the props.conf.
see update above
/k
That looks about correct, but your current REGEX will just match the literal string NSM in your event. I believe that you can set up a single transform, assuming your data looks something like
2012-12-12 12:12:12,12 SN=blah1 action=allow src=1.2.3.4 dst=2.3.4.5 ...
2012-12-13 13:13:13,13 SN=blah2 action=deny src=3.4.5.6 dst=1.0.0.2 ...
props.conf
[your_sourcetype]
TRANSFORMS-set_index = fw_index
transforms.conf
[fw_index]
DEST_KEY = _MetaData:Index
REGEX = \s+SN=(\S+)\s
FORMAT = $1
Thus the first example event would land in the index blah1
, and the second in index blah2
, if they exist - otherwise they'll end up in main
.
Have a look at the examples in the docs for transforms.conf and for managing indexes.
UPDATE:
Then you'll have to hard-code it, with several transforms statements from props.conf, i.e.;
props.conf
[your sourcetype]
TRANSFORMS-fw_index = banana, apple, pear
and in transforms.conf
[banana]
DEST_KEY = _MetaData:Index
REGEX = FGT1234567
FORMAT = banana
[apple]
DEST_KEY = _MetaData:Index
REGEX = FGT45453324
FORMAT = apple
Hope this helps,
Kristian