Why error in handling indexed fields code 15 with HEC in C#?

[Jayne]

Hello everyone!

Currently I am integrating Splunk into our project, working with a local installation of Splunk Enterprise to test the waters and find my way around with Splunk itself. I am using the HttpEventCollectorSender class from the Splunk Package.

My issue is the following: No matter in which format I send a message with the HEC Sender, I will always get the following exception: 

Web Exception:
Server Reply: {"text":"Error in handling indexed fields","code":15,"invalid-event-number":0}
Response: StatusCode: 400, ReasonPhrase: 'Bad Request', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
{
  Date: Mon, 02 May 2022 10:39:30 GMT
  X-Content-Type-Options: nosniff
  Vary: Authorization
  Connection: close
  X-Frame-Options: SAMEORIGIN
  Server: Splunkd
  Content-Type: application/json; charset=utf-8
  Content-Length: 78
}
HResult: -2146233088

The code that I use for sending is almost line by line from the HEC Tutorial from Splunk (Added some more Send-commands at the bottom to try out different formats)

var middleware = new HttpEventCollectorResendMiddleware(0);

var ecSender = new HttpEventCollectorSender(
                new Uri("https://splunkserverdefaultcert:8088/"),
                <token>,
                null,
                HttpEventCollectorSender.SendMode.Sequential,
                0,
                0,
                0,
                middleware.Plugin
            );

ecSender.Send(Guid.NewGuid().ToString(), "INFO", null, <message>);
ecSender.Send(Guid.NewGuid().ToString(), "INFO", <message>);
ecSender.Send(data: <message>);
ecSender.Send(message: <message>);
ecSender.Send(Guid.NewGuid().ToString(), "INFO", null, data: new { testProperty = "testing" });
ecSender.Send(data: new { testProperty = "testing" });

ecSender.FlushAsync().Start();

No matter how I format the message that I send, I will get the error that I mentioned above.

Since the error seems to indicate a formatting issue, I already tried different formats of sending the message. Looking into the errors that are getting logged I can see how the actual message that is getting sent looks, so I can confirm that the following formats do not work:

{"time":"1651492587,089","event":{"data":"This is an event"}}

{"time":"1651492587,089","event":{"message":"This is an event"}}

{"time":"1651494076,162","event":{"id":"00588efd-f403-4cf7-95ce-4ef2a28b0f93","severity":"INFO","data":"This is an event"}}

{"time":"1651494076,162","event":{"id":"00588efd-f403-4cf7-95ce-4ef2a28b0f93","severity":"INFO","message":"This is an event"}}

However, if I just do it with curl as follows, everything seems to work perfectly fine!

https://splunkserverdefaultcert:8088/services/collector/event/1.0 -k -H "Authorization: Splunk <token>" -d "{\"time\":\"1651492587\",\"event\":{\"data\":\"This_is_an_event\"}}"

Do you know what could be causing this error, and what I am doing wrong? Edit: I can now say that this happens also with other Splunk servers, not only with my local one. Curl works, but the HEC Service Implementation always throws the error mentioned above.

If you have any ideas, I would be really thankful for some input! 🙂

[Jayne · Accepted solution]

With the help of co-workers I have found whats causing the issue and created a solution (or rather a workaround) for it.

The actual issue:

The HEC Sender uses an HEC Info object to store events and the timestamp of the message. According to this github-repo this is the way that the timestamp is serialized:

        public HttpEventCollectorEventInfo(DateTime datetime, string id, string severity, string message, object data, Metadata metadata)
        {
            double epochTime = (datetime - new DateTime(1970, 1, 1)).TotalSeconds;
            // truncate to 3 digits after floating point
            Timestamp = epochTime.ToString("#.000", Thread.CurrentThread.CurrentCulture);
            this.metadata = metadata ?? new Metadata();
            Event = new LoggerEvent(id, severity, message, data);
        }

Very important - it uses Thread.CurrentThread.CurrentCulture to format the string. Our app uses the german culture, where we use a comma instead of a dot for the decimal separator. The Splunk Server seems to be confused by that and thus always throws out the beforementioned error. This practically means that the HEC Sender class is completely unusable with a german culture context. There is no way to directly influence the  string formatting, nor can I disable the generation of the timestamp inside the HEC Info Class. For me this seems to be either a bug on the side of the Splunk Server or the HEC Sender.

To test this hypothesis, I used curl to replicate this error and sent one message with the comma as the decimal separator and with the dot as the decimal separator:

curl "https://<server>/services/collector" -H "Authorization: Splunk <token>" -d "{\"time\":\"1651494076,162\",\"event\":\"testing\"}"

// Response: {"text":"Error in handling indexed fields","code":15,"invalid-event-number":0}

curl "https://<server>/services/collector" -H "Authorization: Splunk <token>" -d "{\"time\":\"1651494076.162\",\"event\":\"testing\"}"

// Response: {"text":"Success","code":0}

Solution / Workaround:

My current workaround is temporarily changing the Culture when I use the HEC Sender. I use a helpful construction that I found here to nest it in a using-context. At the moment my code looks roughly like this:

using (CultureInfo.GetCultureInfo("en-US").AsCurrent())
{
    ecSender.Send(Guid.NewGuid().ToString(), "INFO", null, <message>);
}

With the following helper implementation that automatically stores and changes the culture for me:

    public static class CultureInfoExtensions
    {
        public static IDisposable AsCurrent(this CultureInfo culture)
        {
            return new CultureRunner(culture);
        }
    }

    public class CultureRunner : IDisposable
    {
        readonly CultureInfo originalCulture;
        readonly CultureInfo originalUICulture;

        public CultureRunner(CultureInfo culture)
        {
            if (culture == null)
                throw new ArgumentNullException(nameof(culture));

            originalCulture = Thread.CurrentThread.CurrentCulture;
            originalUICulture = Thread.CurrentThread.CurrentUICulture;

            Thread.CurrentThread.CurrentCulture = culture;
            Thread.CurrentThread.CurrentUICulture = culture;
        }

        public void Dispose()
        {
            Thread.CurrentThread.CurrentCulture = originalCulture;
            Thread.CurrentThread.CurrentUICulture = originalUICulture;
        }
    }

I'm changing not only CurrentThread.CurrentCulture property but also the CurrentUICulture, to keep both synced for the time of the change. If anyone has a better idea or a better workaround, please let me know! I hope I could help someone with this.