How to connect Apigee Edge to Splunk

[wswartz_splunk]

Need help configuring a secure connection between Google Apigee Edge and Splunk.  What parameters need to be set on the Apigee end and how does one configure the Splunk side?

William

[wswartz_splunk · Accepted solution]

Good question, William.  Please see below:

Do a web search on the term, “apigee splunk,” and eventually you’ll end up here: https://community.apigee.com/articles/13298/log-messages-into-splunk.html (I’ll refer to this as “the Article.”)  This Apigee Community page is a great place to start for information on configuring Apigee to work with Splunk.  It describes three methods of connecting -using Splunk’s HTTP Event Collector (HEC), using TCP, or using JavaScript.  I’ll cover the first two methods here.  While setting up HEC to work with Apigee is rather straightforward, and I’ll cover this briefly below, the TCP method is a bit trickier and requires some special configuration in Splunk. 

Option 1: HEC

Utilizing the HTTP Event Collector is by far the easiest method for connecting Apigee to Splunk.  The link for setting up HEC in the Article is no longer available but you can find the Splunk documentation here: https://docs.splunk.com/Documentation/Splunk/8.1.2/Data/UsetheHTTPEventCollector. 

Splunk configuration:

In summary, the following steps are required to set up an HEC token:

1. On your receiving system (indexer or heavy forwarder), consider creating a new application and index to contain your Apigee config.  I created an app called Apigee along with a corresponding “apigee” index.
2. In the Splunk UI, go to Settings, Data Inputs.
3. Under Local Inputs, select HTTP Event Collector.
4. Click the Global Settings button and select Enable SSL to utilize HTTPS.  Also, make note of the HTTP Port Number.  The default is 8088 but can be changed as required. Click Save.
5. Back on the main HEC page, select New Token
   1. Give your token a name (I used “apigee).  Click Next.
   2. On the Input Settings page, click Select and select the sourcetype as “_json.”
   3. Select an App Context as appropriate.
   4. Select an existing index to store the Apigee events – I used the index created above.
   5. Click Review, review your settings, then click Submit.
   6. Your HEC token will be displayed; make note of it.  You can also get the token from the Data Inputs, HTTP Event Collector page.

Apigee configuration:

This assumes you have Apigee set up with a working API Proxy.

1. Go to Develop, API Proxies, and select the proxy you’d like to use.
2. Click the Develop tab.
3. Under the Navigator pane, select the “+” next to Policies to add a new policy.
4. A set of policies appears on the left.  Scroll to the bottom of the list and under the Extension section, select the Service Callout policy.  Enter a Display Name for the policy (I used “Splunk_HTTP”), select the HTTP radio button and enter the HTTP Target.  The target will be the publicly accessible IP address and port number for your HEC  - for instance: https:// 198.51.100.1:8088.  Click Add.
5. Under the Code section of the policy, add the following lines (replace the line xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with your HEC token):
   
   

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ServiceCallout async="false" continueOnError="false" enabled="true" name="Splunk_HTTP">
    <DisplayName>Splunk_HTTP</DisplayName>
    <Properties/>
    <Request clearPayload="true" variable="myRequest">
        <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
        <Set>
            <Headers>
                <Header name="Authorization">Splunk xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx</Header>
            </Headers>
            <Payload contentType="application/json" variablePrefix="@" variableSuffix="#">
                {"type": "apigee-log","status": "@message.status.code#"}
            </Payload>
            <Path>services/collector/raw</Path>
            <Verb>POST</Verb>
        </Set>
    </Request>
    <Response>calloutResponse</Response>
    <HTTPTargetConnection>
        <URL>https://198.51.100.1:8088</URL>
    </HTTPTargetConnection>
</ServiceCallout>
​

Here's a screenshot of the Apigee configuration:


Notes:

1. The Payload section is where you can customize what data is being sent to Splunk.  The Article shows a simple message to be sent: {"event":"Logged from EDGE"}.  This can be customized using Apigee Flow Variables.  In my example, I used the “message.status.code” flow variable to return the status code of the event.  More on flow variables may be found here: https://docs.apigee.com/api-platform/reference/variables-reference#request.
2. For the URL, it is possible to change this to use an unsecure HTTP connection instead of HTTPS for testing purposes.  I found this valuable in diagnosing the connection as I could grab a packet capture from my firewall to see how the request was coming in.  If you would like to do this, make sure to uncheck the Enable SSL box in the HEC Global Settings.  Be sure to set it back when your testing is complete!
3. Finally, you need to add this policy to the flow.  For my example, I added the policy to the Proxy Endpoint PostFlow.

If all goes well, you should have a working connection.  When I access my API proxy, the following event is sent to Splunk:

As you can see, using HEC is pretty straightforward.  Now let’s set up Apigee to use a TCP connection from Apigee to Splunk. 

Option 2: TCP:

For this configuration, we’ll need to create a receive port in Splunk and add a new API policy in Apigee.  Let’s start with the Splunk configuration.

Splunk configuration:

1. On your receiving system (indexer or heavy forwarder), as noted above, consider creating a new application and index to contain your Apigee config. 
2. Add the following to an appropriate inputs.conf file.  This can be in your application or the main file located at $SPLUNK_HOME/etc/system/local:

[tcp-ssl://6514]
index = apigee
 
[SSL]
serverCert = /opt/splunk/etc/auth/server.pem
sslPassword = password
​

Note: The above example utilizes the internal Splunk self-signed certificate, the password of which is simply “password.”  While this may be fine for testing, it is recommended you utilize a signed certificate or a self-signed certificate that you create. 

Details on inputs.conf may be found here: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

3. Restart Splunk for the new TCP input to take effect.

Apigee configuration:

In your API proxy

1. Go to Develop, API Proxies, and select the proxy you’d like to use.
2. Click the Develop tab.
3. Under the Navigator pane, select the “+” next to Policies to add a new policy.
4. A set of policies appears on the left.  Scroll to the bottom of the list.  This time we’re going to select the Message Logging extension.  Enter a Display Name for the policy (I used “Splunk_TCP”).  This will create a default Message Logging template that we will need to adjust.  Change the <Host> and <Port> sections to suit your config.  The <Message> section can contain any text, including Flow Variables (see example below).
5. Finally, you need to add this policy to the flow.  For my example, I added the policy to the Proxy Endpoint PostFlow.

Here’s what the finished config should look like:

Once I access my API, the following event is sent to Splunk:

That’s it!  I hope this was helpful for anyone wanting to integrate Apigee with Splunk.

[micheal]

In Truck Simulator Ultimate, connecting platforms like Apigee Edge to Splunk is similar to integrating tracking tools for your fleet. This connection allows you to monitor and analyze API traffic data in real-time, just as tracking fuel and route efficiency improves logistics. It’s a powerful way to optimize operations smoothly. See More