topic Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once? in All Apps and Add-ons

SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

jtsapos — Wed, 19 Nov 2014 15:16:24 GMT

According to the SPLUNK App for CEF documentation:

3) Use the guided search wizard included in the Splunk App for CEF to define what the output will look like in CEF by selecting a data model, mapping data model attributes to fields where necessary (a good amount of this work will be done automatically), creating any new static fields you need, and defining the name of the syslog receiver that will receive the data.

4) Using the search description that you defined with the Splunk App for CEF, Splunk Enterprise writes the data in CEF to the syslog receiver you specified, for use by HP ArcSight or another compatible tool.

You mention that the SPLUNK app for CEF provides a continuous export of the data from SPLUNK which sounds great, but the question I have on this is "Do you have to map every event one by one first or is there some way to just get a full export of the SPLUNK data all at once?"

Thanks in advance.

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

danje57 — Tue, 17 Feb 2015 12:47:21 GMT

No answer?
It seeams that the application is bad and the document doesn't contain lot of information regarding the configuration of the data model

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

jtsapos — Wed, 18 Feb 2015 14:33:32 GMT

Not very encouraging then, is it?

I'm surprised we cannot get an answer for this SPLUNK app from the SPLUNK team...

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

LukeMurphey — Wed, 18 Feb 2015 22:14:16 GMT

I suspect the problem is that people don't understand what this means: "Do you have to map every event one by one first or is there some way to just get a full export of the SPLUNK data all at once?"

Could you clarify? The app doesn't export a single event but all events within the selected data-model. By event, do you mean data-model?

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

jtsapos — Thu, 19 Feb 2015 14:42:37 GMT

By event I mean every event with an event ID. From the description above, it looks like data model is translated into events?

In any case though, it looks like you create a "search description" within the SPLUNK app for CEF and using that search description SPLUNK Enterprise writes the data in CEF to the Syslog receiver you specify.

What about other receivers such as Sourcefire, Tripwire, Symantec, Mcafee, etc.?

SPLUNK app for CEF looks like it may work for Syslog data but I don't know about other different log types.

Also, I have info from others out in the field regarding the SPLUNK to ArcSight integration:

"Two things it doesn't seem to address is the timing issues and health monitoring. It's aim is to fix the formatting issues so you can use a standard ArcSight connector in a standard configuration but it doesn't seem to address any other the other issues with relaying events through Splunk. Two of the biggest problems with relaying your events through Splunk is the timing issues it creates with late events and the lack of device/health monitoring that you would normally have if you were getting it straight from a An ArcSight connector. I can elaborate further if needed but it is my understanding that the Splunk CEF app only addresses formatting it doesn't address delivery, event time or health monitoring issues which plague that configuration."

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

danje57 — Thu, 19 Feb 2015 15:10:35 GMT

I think that the documentation is not enough detailled.

1) Splunk App for CEF (SACEF)

Normally the goal of this application, if I'm correct, will be to translate data from incoming data received by Splunk, using a Data Modal, wich is a query in the Splunk database, that select event we want to translate into CEF format. This CEF format is then sent to a syslog server or an ArcSight Connector.

The main problem I think here, is that we need to create a Data Model for all logs to send via CEF format.

I mean, we have to indicate to SACEF all datasource to translate into the CEF format. This mean that we need to modify the Data Model for each new sourcetype we want to translate in to CEF.

I'm correct?

2) Receiving CEF into Splunk

Now, I'll be surprise how to configure Splunk to receive and understand logs from CEF flow, such as from ArcSight Connector or ArcSight Logger?

I found this Splunk App ==> https://apps.splunk.com/app/487/

But again a think the document is not enough detailled ... 😕 Where to install it? (on the Indexer? on the head search? on the Universal Forwarder?

Lot of application or network security devices work with CEF syslog flow. I'm surprise that Splunk doesn't have a document that describe how to configure it...

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

jtsapos — Thu, 19 Feb 2015 15:23:58 GMT

From your comment above: "I mean, we have to indicate to SACEF all data sources to translate into the CEF format. This mean that we need to modify the Data Model for each new sourcetype we want to translate in to CEF."

Even if you can somehow get these data models configured correctly you still have timing and delivery issues as well as health monitoring issues...

For security monitoring you cannot sacrifice accurate event times or correct delivery of events. You will have loss of data integrity which is unacceptable when reporting security events.

Is there any way that SPLUNK can look at some of these data integrity issues when trying to integrate with ArcSight?

Does the SPLUNK team agree or disagree that there are data integrity issues with the SPLUNK ---> ArcSight integration?

Any feedback is appreciated.. Thanks

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

danje57 — Thu, 19 Feb 2015 15:52:11 GMT

Yes, I agree.

I read a doc that explain how to use a heavy forwarder for such thing... but it doesn't support CEF...

I don't know if a Splunk Professional Service read the forum to explain how to deal with this difficulties...

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

LukeMurphey — Fri, 20 Feb 2015 07:46:26 GMT

I'm confused on why you would use Splunk App for CEF as well as the CIM extraction utilities. Are you saying that you just want to ingest CEF events into Splunk and then send them on to another device (in the same format)?

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

LukeMurphey — Fri, 20 Feb 2015 07:47:11 GMT

BTW: if that is the case, I think a much simpler solution exists. I'll provide an answer if you confirm that is the scenario you are looking to fulfill.

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

danje57 — Fri, 20 Feb 2015 08:35:34 GMT

Hi Luke,

Yes, this is a first use case.

Another one should be sending logs that are not in cef format. Transforming logs that are not in cef and send it to arcsight in a cef format.

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

jtsapos — Fri, 20 Feb 2015 12:26:02 GMT

Yes, as danje57 explains below.

The goal is to take the SPLUNK output and send it to ArcSight for correlation and monitoring.

Do you have a better way to do this without using the SPLUNK cef app?

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

LukeMurphey — Sat, 21 Feb 2015 09:19:01 GMT

There are two scenarios here that are similar but need a different solution. Basically, the difference between the two scenarios is whether or not the events are already in CEF format. If they are received in CEF format, then the Splunk App for CEF is unnecessary since no translation is necessary to make the events into CEF format.

Scenario 1: CEF data needs translation

In this scenario, logs need to be forwarded to a device that accepts CEF but the log themselves are not CEF. Thus, they need to be translated into CEF. It would look something like this:

Recommended solution:

Use the Splunk App for CEF. You need to use the app for CEF because the files are not in CEF format and thus needs to be translated into CEF. CEF does not support arbitrary fields (like Splunk allows). Thus, you need to know all of the fields that you want to export and assign them to CEF fields. This is why the app uses data-models. Data-models require you to know what the fields are beforehand and knowing the fields beforehand is necessary in order to make a valid CEF event.

Scenario 2: CEF data does not need translation (because it started out in CEF format)

In this scenario, the logs are generated in CEF and provided to Splunk but they need to be forwarded to a device that accepts CEF. Thus, no translation is necessary. It would look something like this:

Recommended solution:

Use outputs.conf to forward the events on to the destination device. In this case, the Splunk App for CEF is unnecessary because no translation step is required since the events are already CEF events.

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

LukeMurphey — Sat, 21 Feb 2015 09:20:13 GMT

Ok, I think I see what you mean now. Just added an answer outlining the two use cases.

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

jcoates_splunk — Sun, 22 Feb 2015 00:50:30 GMT

Hi, thanks -- that helps me understand the question.

1) The app works by modeling data that has been received into Splunk and then emitting a CEF formatted stream. Its functionality hasn't got anything to do with event IDs, at least as we understand them.
2) If your receiver doesn't want CEF over syslog, then you probably don't need this app, and can use Splunk's native syslog output or outputcsv or pull from one of our APIs instead. You may still find the app handy as an example of how you can manipulate data for such an output though.
3) I agree that the app is not intended or designed for that purpose. A better way to handle such a use case is to do it in Splunk directly and pass the results to the next system down the pipeline.

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

jcoates_splunk — Sun, 22 Feb 2015 01:05:56 GMT

Thank you, I've updated the docs to try and clarify the point of confusion about what App for CEF does.

It's not accurate to say that you need to alter your data model for every sourcetype that you want to send... rather you might add a transform to that sourcetype. Much of the time, you'll find that this is already done because the Add-on or App that helped gather the data is Common Information Model compliant. An example might help:

You install the Windows Add-on, the CIM, and the App for CEF... now you've got several data models for concepts like Authentication and Change Analysis, and you can use the App for CEF to just grab any Change Analysis events and send them to on your SIEM.
Then you install the Cisco ASA Add-on, and since it's CIM-compliant too, any reconfigurations to your ASA devices also get caught by the Change Analysis model and sent to the SIEM.
Then you install the Oracle Database Add-on, same thing...
Then you point it at your home-grown middleware... and you just need to tag the change analysis events to have the same thing happen.

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

jcoates_splunk — Sun, 22 Feb 2015 01:07:41 GMT

Spot on and thank you.

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

jtsapos — Mon, 23 Feb 2015 15:07:38 GMT

Luke,
Probably scenario 1 is what we are looking at when talking about logs from IDS, Firewalls, Mcafee, Windows, etc. So translation is probably necessary however even with this we still have the problem of delivery, timing and performance monitoring.

I have feedback from others out in the field regarding the SPLUNK to ArcSight integration:

Two of the biggest problems with relaying your events through Splunk is the timing issues it creates with late events and the lack of device/health monitoring that you would normally have if you were getting it straight from a An ArcSight connector.

I can elaborate further if needed but it is my understanding that the Splunk CEF app only addresses formatting it doesn't address delivery, event time or health monitoring issues which plague that configuration."

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

jtsapos — Mon, 23 Feb 2015 15:28:59 GMT

With all these additional steps required to finally get something over to ArcSight for correlation, you can imagine there must be some loss of data integrity somewhere along the way.

I have feedback from others out in the field regarding the SPLUNK to ArcSight integration:

For security monitoring you cannot sacrifice accurate event times or correct delivery of events. You will have loss of data integrity which is unacceptable when reporting security events.

Is there any way that SPLUNK can look at some of these data integrity issues when trying to integrate with ArcSight?

Re: SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

jtsapos — Tue, 24 Feb 2015 16:05:30 GMT

Luke,
Scenario 2 assumes that the SPLUNK output is a common event format for all systems.

This is not exactly correct because the SPLUNK CEF is different than the ArcSight CEF so you still need special formatting from the SPLUNK output EVEN IF the original Log source was in a CEF format.

At this point CEF becomes subjective as to which is the correct CEF.

ArcSight Inc. is considered the originators of the Common Event Format with their flagship SEIM product ArcSight ESM. From there came a lot of copycats of which SPLUNK was one of them.

SPLUNK CEF does not follow the same standard as ArcSight CEF who originated it. Therefore, special formatting needs to be done in order to make the SPLUNK out put match the ArcSight CEF format thus the reason that the SPLUNK app for CEF was developed.

The delivery, timing and performance monitoring issues still exist however even when the SPLUNK output is formatted for CEF data.

The question I have is can SPLUNK app for CEF create a true, clean data stream that can be channeled to Arcsight without all the above mentioned issues?