Log4J with splunk-library-javalogging-1.8.0.jar causes - Incorrect index code":7,"invalid-event-number

[obrientim]

I was previously using Splunk jar 1.7.2 which uses Apache Http client, I was able to setup log4j config to hit the splunk HEC endpoint.

<SplunkHttp name="splunk"
      url="https://ENDPOINT.splunkcloud.com"
      token=“MYTOKEN”
      includeMDC="true"
      messageFormat="json"
      disableCertificateValidation="true">
        <PatternLayout pattern="%m%n"/> 
</SplunkHttp>

This worked great, my log object was a simple object converted to Json via gson

Logger.info(sp.toJson());

In splunk all the object fields parsed as message.myobjectfield.

All was good in the world, but then I needed to use a proxy for this endpoint in another environment. I was unable to find a way, splunk used .custom() to build the request\client and setting system properties did nothing. Setting JVM to use system proxy was too broad because there were other connections that should not use the proxy.

So I had a look at the latest splunk java logging 1.8 and it switched from using Apache Http to OkHttp, and seemed to imply that builder connection configuration was shared, so I think I can setup my proxy prior to using and it would work.

However, I didn't even get that far because my initial baseline test on the open environment with same configuration that worked on 1.7.2 no longer works on 1.8. On 1.8 okhttp returned error 400 Bad Request with body

{"text":"Incorrect index","code":7,"invalid-event-number":1}

Can someone provide a usage of 1.8 using HttpEventCollectorLog4jAppender with and without a proxy. What log4J config did you use and what did you log, I need an example of a multi-field object not just a curl plain text example as I know that already works.

[obrientim]

Ok, I have also solved the Proxy issue. I have other connections like databases, etc so I only want to funnel the splunk HEC post through the proxy.

This code runs on startup and it's called for all requests. I add proxy only for splunkcloud.com and NO_PROXY otherwise

        ProxySelector.setDefault(new ProxySelector() {
            @Override
            public List<Proxy> select(URI uri) {
                List<Proxy> list = new ArrayList<Proxy>();
                String host = uri.getHost().toLowerCase();
                if (host.endsWith("splunkcloud.com")) {
                    Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(httpProxy, httpProxyPort));
                    list.add(proxy);
                } else {
                    list.add(Proxy.NO_PROXY);
                }
                return list;
            }

            @Override
            public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {

            }
        });

[obrientim]

I updated the Log4J config for the SplunkHttp to type="Raw" and I no longer receive the error.
Note: Splunk fields change from message.myobjectfield to event.message.myobjectfield