SPLUNK App for CEF: Do you have to map every event one by one or is there a way to get a full export of Splunk data all at once?

[jtsapos]

According to the SPLUNK App for CEF documentation:

3) Use the guided search wizard included in the Splunk App for CEF to define what the output will look like in CEF by selecting a data model, mapping data model attributes to fields where necessary (a good amount of this work will be done automatically), creating any new static fields you need, and defining the name of the syslog receiver that will receive the data.

4) Using the search description that you defined with the Splunk App for CEF, Splunk Enterprise writes the data in CEF to the syslog receiver you specified, for use by HP ArcSight or another compatible tool.

You mention that the SPLUNK app for CEF provides a continuous export of the data from SPLUNK which sounds great, but the question I have on this is "Do you have to map every event one by one first or is there some way to just get a full export of the SPLUNK data all at once?"

Thanks in advance.

[LukeMurphey]

There are two scenarios here that are similar but need a different solution. Basically, the difference between the two scenarios is whether or not the events are already in CEF format. If they are received in CEF format, then the Splunk App for CEF is unnecessary since no translation is necessary to make the events into CEF format.

Scenario 1: CEF data needs translation

In this scenario, logs need to be forwarded to a device that accepts CEF but the log themselves are not CEF. Thus, they need to be translated into CEF. It would look something like this:

Recommended solution:

Use the Splunk App for CEF. You need to use the app for CEF because the files are not in CEF format and thus needs to be translated into CEF. CEF does not support arbitrary fields (like Splunk allows). Thus, you need to know all of the fields that you want to export and assign them to CEF fields. This is why the app uses data-models. Data-models require you to know what the fields are beforehand and knowing the fields beforehand is necessary in order to make a valid CEF event.

Scenario 2: CEF data does not need translation (because it started out in CEF format)

In this scenario, the logs are generated in CEF and provided to Splunk but they need to be forwarded to a device that accepts CEF. Thus, no translation is necessary. It would look something like this:

Recommended solution:

Use outputs.conf to forward the events on to the destination device. In this case, the Splunk App for CEF is unnecessary because no translation step is required since the events are already CEF events.

[jtsapos]

Luke,
Probably scenario 1 is what we are looking at when talking about logs from IDS, Firewalls, Mcafee, Windows, etc. So translation is probably necessary however even with this we still have the problem of delivery, timing and performance monitoring.

I have feedback from others out in the field regarding the SPLUNK to ArcSight integration:

"Two things it doesn't seem to address is the timing issues and health monitoring. It's aim is to fix the formatting issues so you can use a standard ArcSight connector in a standard configuration but it doesn't seem to address any other the other issues with relaying events through Splunk.

Two of the biggest problems with relaying your events through Splunk is the timing issues it creates with late events and the lack of device/health monitoring that you would normally have if you were getting it straight from a An ArcSight connector.

I can elaborate further if needed but it is my understanding that the Splunk CEF app only addresses formatting it doesn't address delivery, event time or health monitoring issues which plague that configuration."

[jcoates_splunk]

Spot on and thank you.

[danje57]

Hi Luke,

Yes, this is a first use case.

Another one should be sending logs that are not in cef format. Transforming logs that are not in cef and send it to arcsight in a cef format.

[vhdevera]

Hi Luke & danje57,

I am doing a similar setup log source (not CEF) -> SPlunk -> Log output (CEF) and could not make this work... do you guys have any updates or have seen such setup that worked?

[tlmayes]

Anybody on this thread ever been able to get the Splunk App for CEF to work properly? I have been working with Splunk support for two months now, and they can't make it work?

[LukeMurphey]

@tlmayes: What version of CEF are you using?

[tlmayes]

Version 2.0.0

[LukeMurphey]

@tlmayes: Could you create a new post with the particulars of the issue you are seeing?

[tlmayes]

To your question about creating a new post, already there. Posted back in May

https://answers.splunk.com/answers/529140/cef-output-forwarding-everything-from-all-indexes.html

[tlmayes]

Ultimately trying to forward events from a Splunk index to ArcSight, which requires CEF format. Going through the installation process on stand-alone host, everything works fine, i.e. the "dummy" ArcSight server receives the exact CEF formatted event as expected.

Duplicating the process on a clustered production environment: Create the datamodel/dataset, map the fields, etc, import the created App to each of the clustered indexers, restart the indexers (required), and wait..... this time the dummy ArcSight server receives 100% of everything (and no CEF formatted data).

The only difference from my perspective is clustering vs. no clustering, and separation between indexers and SH's (test was on standalone all-in-one).

[jtsapos]

Regarding your comment above: "The App's implementation of CEF is defined by a user-editable file at lookups/cef_inventory.csv so that you can address any drift introduced by your organization or HP;"

Are you telling me that this .csv file is being used to correct delivery, timing and performance monitoring issues that have been plaguing this configuration today out in the field?

Do you have evidence of that? Do you know of any SPLUNK ----> ArcSight implementations that have been stood up and working now in the field without any of these timing or delivery issues mentioned above?

[danje57]

CSV will not solve these issue. But with it you can modify CEF format on the fly

[jtsapos]

If you can modify the SPLUNK CEF output to match the ArcSight CEF output it's good but if the date/time stamps and delivery is off, then you are looking at data integrity issues.

This is not good for correct data results especially in the security field where requests are made daily for information based on date/time periods.

In short, SPLUNK App for CEF does not appear to be all the way there yet. It doesn't appear to be 100% reliable for what it's supposed to be doing.

[LukeMurphey]

Ok, I think I see what you mean now. Just added an answer outlining the two use cases.

[jtsapos]

Luke,
Scenario 2 assumes that the SPLUNK output is a common event format for all systems.

This is not exactly correct because the SPLUNK CEF is different than the ArcSight CEF so you still need special formatting from the SPLUNK output EVEN IF the original Log source was in a CEF format.

At this point CEF becomes subjective as to which is the correct CEF.

ArcSight Inc. is considered the originators of the Common Event Format with their flagship SEIM product ArcSight ESM. From there came a lot of copycats of which SPLUNK was one of them.

SPLUNK CEF does not follow the same standard as ArcSight CEF who originated it. Therefore, special formatting needs to be done in order to make the SPLUNK out put match the ArcSight CEF format thus the reason that the SPLUNK app for CEF was developed.

The delivery, timing and performance monitoring issues still exist however even when the SPLUNK output is formatted for CEF data.

The question I have is can SPLUNK app for CEF create a true, clean data stream that can be channeled to Arcsight without all the above mentioned issues?

[jtsapos]

Is there any way that SPLUNK can look at some of these data integrity issues when trying to integrate with ArcSight?

[jcoates_splunk]

Hi,

we've included a CEF mapping as provided by Hewlett Packard's publicly posted standard, but can certainly understand that actual implementation drifts from published standards, particularly when the product allows local customization. The App's implementation of CEF is defined by a user-editable file at lookups/cef_inventory.csv so that you can address any drift introduced by your organization or HP; no restart of Splunk is necessary.

[danje57]

I think that the documentation is not enough detailled.

1) Splunk App for CEF (SACEF)

Normally the goal of this application, if I'm correct, will be to translate data from incoming data received by Splunk, using a Data Modal, wich is a query in the Splunk database, that select event we want to translate into CEF format. This CEF format is then sent to a syslog server or an ArcSight Connector.

The main problem I think here, is that we need to create a Data Model for all logs to send via CEF format.

I mean, we have to indicate to SACEF all datasource to translate into the CEF format. This mean that we need to modify the Data Model for each new sourcetype we want to translate in to CEF.

I'm correct?

2) Receiving CEF into Splunk

Now, I'll be surprise how to configure Splunk to receive and understand logs from CEF flow, such as from ArcSight Connector or ArcSight Logger?

I found this Splunk App ==> https://apps.splunk.com/app/487/

But again a think the document is not enough detailled ... 😕 Where to install it? (on the Indexer? on the head search? on the Universal Forwarder?

Lot of application or network security devices work with CEF syslog flow. I'm surprise that Splunk doesn't have a document that describe how to configure it...

[jtsapos]

Yes, as danje57 explains below.

The goal is to take the SPLUNK output and send it to ArcSight for correlation and monitoring.

Do you have a better way to do this without using the SPLUNK cef app?