Getting Data In

dispatch logs incoming from one source to multiple indexes

AdrienW
Explorer

Dear all,

I have syslogs of several firewalls incoming from a TCP port.

I would like to dispath the logs of each firewall (by their SN) in their proper index.

I saw it's possible to do that configuring transforms.conf and props.conf

Here is the example I found :

props.conf

[syslog]

TRANSFORMS-syslog-NSM=syslog-NSM

transforms.conf

[syslog-NSM]

DEST_KEY = _MetaData:Index

REGEX = NSM

FORMAT = index1

Does someone can adapt it to fit with my requirements?
I really don't know how to do it...

my index : index1 // index2 // ...
SN : FT001 // FT002 // ....

Thanks !

Tags (1)

NewMilenium
Path Finder

No, I'm not sure...

Here are the files involved.

/opt/splunk/etc/system/local/props.conf :

[fortigate]

TRANSFORMS-fw_index = apple, banana

And here is the /opt/splunk/etc/system/local/transforms.conf :

[apple]

DEST_KEY = _MetaData:Index

REGEX = FG100C3G09619593

FORMAT = index::apple

[banana]

DEST_KEY = _MetaData:Index

REGEX = FGT50B3G10612115

FORMAT = index::banana

There are other files that were already here before I touch anything, in /opt/splunk/etc/system/local : alert_actions.conf, authorize.conf, eventtypes.conf, inputs.conf, migration.conf, README, server.conf, tenants.conf, web.conf. I prefer to mention it because "local" folder is supposed to contain files "adding and replacing" the default ones.

You can notice I changed [fortigate_traffic] to [fortigate]. I did, because this is the only way I managed to finally get an error message. Well, two, obviously, written in black on a yellow font up the splunk window. Here they are :

received event for unconfigured/disabled/deleted index='index::apple' with source='source::udp:514' host='host::10.0.1.254' sourcetype='sourcetype::fortigate_traffic' (2 missing total)

received event for unconfigured/disabled/deleted index='index::banana' with source='source::udp:514' host='host::10.0.1.254' sourcetype='sourcetype::fortigate_traffic' (1 missing total)

And yes, I created the 2 indexes called "apple" and "banana" in splunk, with "Manager>Indexes>New".

I seriously can't find what's wrong...

0 Karma

NewMilenium
Path Finder

Oh my, it works. Wonderful job, Kristian. Thanks so much!

0 Karma

kristian_kolb
Ultra Champion

Oops, I may have made a small mistake; it seems like the correct format for FORMAT is , NOT index::

http://docs.splunk.com/Documentation/Splunk/5.0.1/Indexer/Setupmultipleindexes

Sorry about that.

0 Karma

NewMilenium
Path Finder

Hello,

first, thank you for your answers, Kristian.
I am quite new to splunk; I did what you said, and still it won't work.
The situation is; I've got 2 different UDP sources coming to one port, let's say P1. I must separate the 2 sources so that each one goes to its index.
Here is what i've done so far;

1) created the 2 indexes "banana" and "apple",

2) created the 2 files "props.conf" and "transforms.conf" in /opt/splunk/etc/system/local ,

3) put the configuration above you talked about with the right serial numbers (and only that, assuming the default configs props.conf and transforms.conf in opt/splunk/etc/system/default are still used before overrided with those 2 files)

4) edited "inputs.conf" in /opt/splunk/etc/system/local which had

[default]

host = localhost

into

[default]

host = localhost

[udp]

connection_host=ip

queue = parsingqueue

because I saw here this could be necessary (and it wasn't working anyway) and that in the "default" inputs.conf, there was connection_host=ip .

I am still working on it, and cannot find what's wrong. Maybe because I can't avoid to precise what index is used in Manager>Data inputs>UDP>P1 (which is set to "main").

0 Karma

kristian_kolb
Ultra Champion

are you sure that your regexes work correctly?

0 Karma

NewMilenium
Path Finder

Well, I changed "local" inputs.conf accordingly, and logs are still put into "main" , sadly.
We don't have control over the sending parties; the order is "only one port : 514". That's why I/we must differentiate each log using the device_id.

0 Karma

kristian_kolb
Ultra Champion

First of all, you have no port specification in the udp stanza header. It should be like;

[udp://514]

http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitornetworkports#Add_a_network_input_usin...

On a side note, if you have control over the sending parties, you could set them to send you data on different ports and specify the indexes in inputs.conf directly.

[udp://514]
index=banana

[upd://515]
index=apple

Default values are just that. You don't need to set them again.

0 Karma

AdrienW
Explorer

Thanks for your answer, I'm near to resolve this, but I need help one more time.

My logs are like this :

Jan  3 15:32:52 10.0.1.254 date=2013-01-03 time=15:03:19 devname=FGT60B-CFP device_id=FGT60B3908672004
Jan  3 15:38:21 10.0.1.254 date=2013-01-03 time=15:36:43 devname=FGT60B-EDC device_id=FGT60B3908668256

As you can see, there is a special SN for each hardware and I need to put it in index like "banana" and "apple" (common names).

Is it possible to add a condition in the regex or anything else that means :

FGT60B3908672004 => banana
FGT60B3908668256 => apple

all logs have the same source and sourcetype, and I can't take the host value to dispatch via the props.conf.

kristian_kolb
Ultra Champion

see update above

/k

0 Karma

kristian_kolb
Ultra Champion

That looks about correct, but your current REGEX will just match the literal string NSM in your event. I believe that you can set up a single transform, assuming your data looks something like

2012-12-12 12:12:12,12 SN=blah1 action=allow src=1.2.3.4 dst=2.3.4.5 ...
2012-12-13 13:13:13,13 SN=blah2 action=deny src=3.4.5.6 dst=1.0.0.2 ...

props.conf

[your_sourcetype]
TRANSFORMS-set_index = fw_index

transforms.conf

[fw_index]
DEST_KEY = _MetaData:Index
REGEX = \s+SN=(\S+)\s
FORMAT = $1

Thus the first example event would land in the index blah1, and the second in index blah2, if they exist - otherwise they'll end up in main.

Have a look at the examples in the docs for transforms.conf and for managing indexes.


UPDATE:

Then you'll have to hard-code it, with several transforms statements from props.conf, i.e.;

props.conf

[your sourcetype]
TRANSFORMS-fw_index = banana, apple, pear

and in transforms.conf

[banana]
DEST_KEY = _MetaData:Index
REGEX = FGT1234567
FORMAT = banana

[apple]
DEST_KEY = _MetaData:Index
REGEX = FGT45453324
FORMAT = apple

Hope this helps,

Kristian

Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...