Source type for HTTP requests / responses

[advseb]

I'm tasked to log all HTTP requests plus responses in a custom web application to Splunk. This should include the complete request including all HTTP headers plus some additional fields for categorizing the data.

I'm a bit overwhelmed by Splunk's terminology, but I guess I first need to identify the correct source type for this kind of data. I'm a bit surprised that there seems to be no built-in source type for it, at least I was not able to identify one: http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/Listofpretrainedsourcetypes

So do I need to define my own source type? Can't imagine nobody should have logged HTTP requests plus responses before.

[adonio]

is it Apache? try this:
https://splunkbase.splunk.com/app/3186/#/details
is it IIS? try this:
https://splunkbase.splunk.com/app/3185/
is it nginx? try this:
https://splunkbase.splunk.com/app/3258/
is it a custom application?
show us some sample data and we will help you build the right extractions

[advseb]

Unfortunately, It is a custom (read legacy) application not based on any current framework. I have full control over the output format as I have to manually generate it.

In contrast to e.g. Apache access logs, the logs should not just contain timestamp and URL of the request, but also all HTTP header fields plus request body plus belonging response. So if I would need to come up with some kind of JSON respresentation of the data, I would say it is similar to:

{
    "url": "http://www.example.com/?parm1=abc&param2=def",
    "method": "POST",
    "responseCode": 200,
    "request": {
        "headers": {
            "User-Agent": "Mozilla...",
            "Accept": "*/*",
            "Accept-Encoding": "gzip, deflate, br",
            "...": "..."

        },
        "body": "some arbitrary string, e.g. containing XML or JSON data"
    },
    "response": {
        "headers": {
            "Cache-Control": "max-age=14400",
            "Content-Length": "396",
            "...": "..."
        },
        "body": "some arbitrary string, e.g. containing XML or JSON data"
    }
}

But I was hoping that I don't need to reinvent the wheel and there is already some format for it, which I can generate.

[adonio]

in that case use sourcetype=_json or xml sourcetype, those are prebuilt in splunk.
try first manually by placing some events in a sample file.
navigate to settings (top right corner) -> add data -> upload -> browse to your sample file and add it -> click next -> from dropdown sourcetype option pick structured -> pick sourectype as _json
check on te right side of the screen preview. -> if good, add index -> submit and start searching

[adonio]

hello there,
what is the data format? in splunk its refered as "sourcetype".
what technology / application generates the data?
did you take a look at the HTTP Event Collector?
http://dev.splunk.com/view/event-collector/SP-CAAAE6M

[advseb]

My data format are HTTP requests and responses. So I want to log all HTTP requests received by my web app including the responses I generate. I'm not asking how to send events via HTTP to Splunk.