Where does Splunk for Fortigate gets data from? / How do I configure indexes Splunk for Fortigate gets data from?

[NewMilenium]

I think question is all in the title. I'm asking it because I got 3 indexes it could get data from, and it actually only gets from 2... after much investigation, I got no idea why it won't get logs from the third one.
I proved this by getting an example log of the third index: a fortigate_traffic type log of January the 18th, 13:10. This log isn't showed in Splunk for Fortigate Traffic dashboard if I set a "custom time" : between January the 18th 12:00 and 18:00, nor if I click "View Full Report" and look manually at the logs selected with the search.

But if I manually configure a Splunk for Fortigate search and put "index=XXXX", then it obviously finds the said logs.

Any idea, anyone?

[NewMilenium]

Let me sum up a second;
I thank dmaislin_splunk for his answers, now I know "where Splunk for Fortigate gets his data".

My problem now is that I don't know how to both get my data in the index I want AND force their type to fortigate (with those conditions: all my sources come from the same host, on the same port, I separate them with the device_id). My configuration of transforms.conf seems bad, and I can't get it right, even after tons of tests.
I need something that would be like

[force_fortigate_sourcetype]
DEST_KEY = MetaData:Sourcetype, _MetaData:Index
REGEX = [device_ID]|[TheOtherDevice_ID]
FORMAT = Sourcetype::fortigate, Index::index3

[dmaislin_splunk]

I don't know your experience level, but, you have lots of options.

1. Create a different UDP data input on a dedicated port just for fortigate data, i.e. 51400
2. Learn how to use props.conf and transforms.conf to change the sourcetype based on the host or whatever your like. Please read: http://docs.splunk.com/Documentation/Splunk/latest/Data/Advancedsourcetypeoverrides

[NewMilenium]

Hello,
I'm on the sleeping IRC channel.
All my sources are coming from the same host, I cannot index them using host; so I'm using device_id.
I'm testing tons of transforms.conf configurations to set the sourcetype to fortigate AND the index to index3. I've not found how to do it yet - and I now perfectly know this splunk documentation page: http://docs.splunk.com/Documentation/Splunk/5.0.1/Admin/Transformsconf

I'd need something like
[force_fortigate_sourcetype]
DEST_KEY = MetaData:Sourcetype, _MetaData:Index
REGEX = [deviceID]|[OtherDeviceID]
FORMAT = Sourcetype::fortigate, Index::index3

[dmaislin_splunk]

You could setup props.conf like this too:

[host::1.2.3.4]
index = index1
sourcetype = fortigate

I am headed out for travel, but stop by our IRC chatroom and introduce yourself.

http://www.splunk.com/view/SP-CAAACDF

OR

http://chat.efnet.org:9090
Channel: #splunk
Nick: Whatever you like.

[NewMilenium]

Okay, now I got a little result;
I removed "index3" from the list under stanza [fortigate] in props.conf, now I receive data from source3 in index "main", which actually sounds logical as in transforms.conf there is nothing in stanza [force_fortigate_sourcetype] to precise where to put the data. How would I put it again in index3? I don't get much how those "DEST_KEY" and "FORMAT" work (yes, AFTER reading documentation), as they can define sourcetype or index... I guess not both..?..

[NewMilenium]

So, here is what I changed;
removed the 2 stanza [index3] in transforms.conf,
put the stanza [force_fortigate_sourcetype] with what you said, but REGEX = [aDevice_ID]|[anotherDevice_ID] .
Added in props.conf :
[syslog]
TRANSFORMS-force_sourcetype_for_fortigate = force_fortigate_sourcetype

By the way, another thing: the source1 and source2 are fine with Splunk for Fortigate dashboards, but of type fortigate_traffic too (just like source3) in my Search examples.

[dmaislin_splunk]

Real examples from the actual transforms and props are better.

[dmaislin_splunk]

Also, you example above lists the same stanza of [index3] twice in the transforms.conf. That would not work. Maybe it's only because you mistyped in the example.

[dmaislin_splunk]

The application is written to expect a sourcetype of fortigate, not fortigate_traffic. From the README.txt file in the app:

Important: When you configure the input port, you must set the sourcetype of the firewall data to fortigate. Otherwise, the app will not work.

[NewMilenium]

I'm sorry to use "answer question" as a way to post, but this wouldn't fit in a "comment".

So, here I am;

• I did the change in transforms.conf,
• I receive the logs in the right index, then,
• It's not found by Splunk for Fortigate, for example in Traffic dashboard.

Let's say this :
I got 3 types of fortigate data coming into port 514. Source 1, source 2, source 3; they are separated into index1, index2, index3. Index1 and index2 are perfectly working, and Splunk for Fortigate shows their results in its dashboards. Data of index3 should be shown as well, and it isn't.

When I go in "full report" in Splunk for Fortigate, then change the search into "index=index3", it finds the data, which has the type "fortigate_traffic" by the way. When I do sourcetype="fortigate_traffic" , it shows only data from index1 and index2. If i do sourcetype="fortigate_traffic" AND index="index3", it shows the same result than index="index3".

Here is how my files look. The setnull is used because there are 2 other sources I'd like not to index, otherwise the 500 Mb/day would be blown up. The first [index3] you see is the one I just added according to the documentation I read that you headed me to.

props.conf :
[fortigate]
TRANSFORMS-fw_index = index1, index2, index3
TRANSFORMS-null = setnull

transforms.conf :
[setnull]
REGEX = [hereisaDevice_ID]|[hereisanotherDevice_ID]
DEST_KEY = queue
FORMAT = nullQueue

[index3]
FORMAT = sourcetype::fortigate
DEST_KEY = MetaData:Sourcetype

[index1]
DEST_KEY = _MetaData:Index
REGEX = [hereisanotherDevice_ID]
FORMAT = index1

[index2]
DEST_KEY = _MetaData:Index
REGEX = [yetanotherDevice_ID]
FORMAT = index2

[index3]
DEST_KEY = _MetaData:Index
REGEX = [hereisagainaDevice_ID]|[anotherDevice_IDagain]
FORMAT = index3

(There are even more things I could explain that are related to this, but it would be seriously even longer and more complicated, and I think the problem I showed here is "enough" already.) So, what am I doing wrong? I seriously don't get it.

edit : oh. Time for me to post, you posted already. Reading and trying what you said. Thanks!

[dmaislin_splunk]

From: http://splunk-base.splunk.com/answers/6917/different-sourcetypes-for-different-syslog-hosts

Something like this for those events from the fortigate:

--props.conf--
[syslog] <-- important part. host=fortigate hasn't been set yet, so use syslog or the hostname of the forwarder
TRANSFORMS-force_sourcetype_for_fortigate = force_fortigate_sourcetype

--transforms.conf--
[force_fortigate_sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (fortigate-hostname.domain|1.2.3.4) <-- some unique string that only appears in fortigate events
FORMAT = sourcetype::fortigate

[NewMilenium]

Maybe I expressed myself in a wrong way, so,
1. I cannot and must not change the port for any data incoming on splunk,
2. This sounds like a solution, I'm going to try to apply this right now.

Thanks a lot for your help!

[dmaislin_splunk]

It is getting the data from syslog. Just configure your firewall to syslog to Splunk on whatever port you want, then setup a data input on Splunk to receive UDP or TCP traffic on that port with a sourcetype of fortigate. In the app home directory there is a README.txt if you download it and untar it there is more information there.

*** Configuring ***

To get the firewall data into Splunk:

• Configure a port on the Splunk server to listen for UDP traffic. If you do not know how to do this, refer to the online documentation here:

http://www.splunk.com/base/Documentation/latest/admin/MonitorNetworkPorts

Important: When you configure the input port, you must set the sourcetype of the firewall data to fortigate. Otherwise, the app will not work.

If you are using UDP input, you will also need to add:

no_appending_timestamp = true

In the UDP stanza in your inputs.conf file. For example:

[udp://514]
connection_host = ip
sourcetype = fortigate
no_appending_timestamp = true

- Next, configure the firewall device to direct log traffic to the Splunk server on the network port that you specified.

[NewMilenium]

Thank you for your answer.

But with such a configuration, isn't ALL the data coming to the set port going to be recognized as sourtype=fortigate? If yes, it will be a problem, because one of the obligations I got is: all data is received on port 514, and some of the data isn't only of fortigate type. So far, I used props.conf and transforms.conf to index my data using the device_id of each firewall, successfully.