@gblock, Re: Using a TCP Input from the browser. Yes, for all the reasons you cite, that's pretty much just a party trick; certainly, not something I'd want to implement in a production environment. (That party trick works from curl, too.) The unwanted event, containing the lines I'd told LINE_BREAKER to discard, irks me. This morning, I used a TCP client to send an HTTP request - and also text that "looks" like an HTTP request, with \r\n -delimited preamble lines - to the Splunk TCP input. I still get that unwanted event. I've just asked about this in a separate question, "Why do the contents of the first capturing group in this LINE_BREAKER regex appear as a separate event?". ... View more

Thanks for converting the comment, Patrick 🙂 ... View more

Hi @gblock, Re: you can load balance TCP but it is expensive Any reason not to use HAProxy for TCP load balancing of Splunk TCP inputs? From the HAProxy website: HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for very high traffic web sites and powers quite a number of the world's most visited ones. Over the years it has become the de-facto standard opensource load balancer, is now shipped with most mainstream Linux distributions, and is often deployed by default in cloud platforms. Since it does not advertise itself, we only know it's used when the admins report it 🙂 ... View more

Re: Does Splunk support receiving a continual stream of input via an HTTP POST? No. Not a continual (endless) stream. You can "batch" (send multiple) events in a single HTTP POST, but there is a maximum limit to the size of an HTTP request. The limit is set by max_content_length in limits.conf . The default value, as of Splunk 6.4, is 1000000 bytes (~ 1 MB). Exceeding that limit results in the HTTP response error code 413 (request entity too large). The Splunk documentation that describes batching events (such as "About the JSON event protocol in HTTP Event Collector") does not mention this limit (at least, I can't find any such mention). I think it should. ... View more

@gblock, thanks very much for weighing in on this question, much appreciated. I'd hoped to catch your attention. I'm asking this question primarily on behalf of some developer colleagues who will soon be turning their attention to sending events to Splunk over an IP network. In particular, I want to present them with information to help them decide whether to use HEC or a TCP input. As mentioned in my question, I've already done some research, and have hands-on experience using both HEC and TCP inputs (albeit currently only on a small scale, on a single Splunk instance). To recap: my question is "Why would I use HEC when I can use TCP?". That is, as further clarified in the details of the question, why would I choose to use HEC in situations where I can use either HEC or TCP? I'm interpreting your answer in the context of that question. Point by point: There are many clients where TCP is not a viable option, such as sending from the browser. Yes, fair point. However - I sincerely don't mean to be adversarial or otherwise annoy you; I'm grateful for your time, and hope for more advice from you on your subsequent points - this point is not relevant to the specific context of this question, where TCP is a viable option. Incidentally, and more or less just for fun, this morning I played around sending events from a web browser (Chrome) to a Splunk TCP input. Yes, really (and, yes, I do have better things to do ;-): xhr = new XMLHttpRequest() xhr.open("POST", "http://localhost:6067") xhr.send("{\"my_field\": \"some_value\"}") with the following stanza in props.conf : [source::tcp:6067] KV_MODE = json LINE_BREAKER = ((^[^{][^\n]*\r\n)*)\{\"[^}]+\} SHOULD_LINEMERGE = false The LINE_BREAKER is intended to ditch the multiline HTTP request header. It kinda works: for each xhr.send , I get two events in Splunk: The event I want, {"my_field": "some_value"} , with myfield correctly presented as a field. An unwanted event, with a time stamp 10 seconds earlier (!), consisting only of the multiline HTTP header (which I thought I told LINE_BREAKER to discard!) I spent some time Googling about automatic HTTP request retries, and whether I can set an Ajax request to use HTTP 1.0 instead of 1.1, but gave up. Maybe I just specified an inappropriate regex? Interesting, but academic, thanks to HEC. Moving on. Scale. HEC is stateless and designed to easily scale out across a pool of instances behind a LB. Again, fair point. But, again, the point of this question is to decide between using HEC and a Splunk TCP input. A Splunk TCP input is also stateless. Right? And a Splunk TCP input easily scales out across a pool of instances behind a load balancer (LB), too. Or am I missing something here? The Splunk dev topic "High volume HTTP Event Collector data collection using distributed deployment" describes using a network traffic load balancer (such as NGINX) in front of several Splunk Enterprise indexers. Is there any reason why I can't do the same thing - use a TCP LB, such as NGINX or HAProxy - for Splunk TCP traffic? Performance. We've heavily optimized HEC to handle 100K events or more per instance. How does that compare with the performance of a Splunk TCP input? HTTP involves processing that TCP does not, such as parsing an HTTP request header and returning a response with a header (and, in the case of HEC, a JSON-format body). This is one reason for my original question: if I don't want or need the processing overhead of HTTP versus TCP, why use HEC? Outside of this processing that is specific to HTTP - and so, an overhead, when compared to TCP - I would have thought that the remainder of the event processing would be common to both HEC and Splunk TCP inputs. Or could be, if it isn't: that's one reason why I recently asked the question "Can I use the HTTP Event Collector JSON event protocol for TCP inputs?". As you mention in the next point, HEC has rich support for JSON out of the box. Does that "protocol" - for example, specifying the time in the metadata as a Unix Epoch value - improve the performance of HEC versus a Splunk TCP input? If so, why not offer that same JSON structure for TCP inputs? (Or are you deliberately deprecating TCP inputs in favor of HEC?) Aside: It occurred to me that perhaps you deliberately chose "EC" as the official abbreviation for HEC for this very reason: that you had plans to "roll out" the JSON-based EC metadata/data protocol across other input methods, including TCP. But nope, I was wrong, because you've recently clarified the official abbreviation as being HEC, not EC. Ease of use. HEC has really rich support for JSON out of the box, you don't have to mess with sourcetypes or bending over backwards with your JSON. Yes. I describe some of that "bending over" in my question. Much nicer with HEC, thanks. However, as I mentioned, I don't find this (ease of use) a compelling enough reason to choose HEC over TCP. Unless the "rich support for JSON" comes with a performance benefit (that you don't plan to make available to TCP inputs). Security.... Yes. However, in the use cases I expect to see - I didn't mention this in my question - I suspect (although I don't know for sure) that all of this traffic will occur behind a firewall on an intranet or over a VPN. I look forward to hearing more from you, especially regarding performance. Not wishing to put words in your mouth, but framing your answer in the context of my question, I think what you're telling me is: There's a bunch of reasons [why you would use HEC when you can use TCP]: ... Performance That is, in a nutshell: HEC offers better performance than using TCP inputs. I'd like to hear more about that. And if that's true, then perhaps the Splunk docs recommendation I cited in my question needs revisiting (or at least, qualifying): TCP ... is the recommended protocol for sending data from any remote host to your Splunk Enterprise server ... View more

@martin_mueller mentioned the following new visualization in a comment on a related question: Horizon Chart - Custom Visualization ... View more

Thanks very much for the link, Martin. Yes, that looks very useful: I'll try it out on Monday (I'm writing this on Friday night). Apologies for this belated acknowledgement. I don't know why I missed your comment; I just stumbled on it now as I was reviewing some of my old questions. I guess I must have overlooked the email notification, or deleted it by mistake. ... View more

Thanks! I'll wait until Monday to see if anyone pitches in with a different, and more compelling, answer (I would be surprised), but if not, I'll accept your answer. Thanks again, and have a good weekend. ... View more

I use cURL on Windows for ad hoc EC ingestion. To avoid escaping quotes, I save my JSON to a file, and refer to that file in the curl -d option by prefixing the path with an at sign (@). For example: -d @ec_input.json For details, see the curl man page. I also use a variety of homegrown PowerShell scripts (.ps1), batch files (.bat) - some of which are simply wrappers for curl - and Java programs to send JSON to EC. For example, I use Java to massage JSON lines-formatted event data with an ISO 8601-formatted time stamp field into EC "packets" with a Unix Epoch time metadata field. ... View more

Hi @woodcock, thanks for the suggestion: I would use HEC:xyz where HEC is the common name for HTTP Event Collector. How common? The first Splunk blog post tagged http-event-collector , "HTTP Event Collector, your DIRECT event pipe to Splunk 6.3", uses the abbreviation EC: HTTP Event Collector (EC) is a new, robust, token-based JSON API So does the Splunk dev topic "Introduction to Splunk HTTP Event Collector": Welcome to Splunk HTTP Event Collector (EC) So does the "Walkthrough" dev topic: the EC port ... an HTTP Event Collector authentication token ("EC token"). EC tokens are ... the EC event protocol ... But then, the latest Splunk blog post tagged `http-event-collector, "There is a “LOG”! Introducing Splunk Logging Driver in Docker 1.10.0", on 10 February 2016, refers to HEC: Built on the HTTP Event Collector (HEC) ... Enable HEC ... Create a New HEC Token And Googling for: "HTTP Event Collector (HEC)" site:splunk.com returns "about 38 results", whereas: "HTTP Event Collector (EC)" site:splunk.com returns "about 32 results". If any Splunk tech writers are reading this: what's the official abbreviation: EC or HEC? ... View more

Yeah, I read the Splunk docs topic on that ("Rename source types at search time") before asking this question. Problem is, that functionality is too limited to be useful in this situatoin: it only offers a one-to-one renaming, from the original sourcetype value to a different literal string value. Thanks for the suggestion, though. Thanks also for prodding me to try sourcetype and see what happens. I think that your answer, combined with this trail of comments, will prove useful to users with the same question, so I'm going to accept it. I'm considering asking a new question, spawned by the testing I've done here, to ask about (re)using the EC protocol for TCP inputs. ... View more

What I'd really like is to use the same JSON for TCP input as I use for the HTTP Event Collector. That is, to specify time and sourcetype as metadata keys, rather than having to write stanzas to configure timestamp recognition and override the source type per-event. ... View more

I realize that I could save myself a heap of trouble here by using a single sourcetype value for all of the different types of log records - all of which have different record structures - that are extracted by the platform-specific log extraction tool I referred to in my original question. And I could coin some new field with the unique values that would have been in sourcetype . But I think that would be a "cop out"; an un-Splunk-y thing to do; in neither the spirit nor the letter of the Splexicon definition of source type. ... View more

I'm now overriding the sourcetype. Here's my working transforms.conf stanza: [set_sourcetype_xyz] REGEX = \x22event_sourcetype\x22:\x22([^\x22]+)\x22 FORMAT = sourcetype::$1 DEST_KEY = MetaData:Sourcetype ( \x22 is an escaped double quote) Using sourcetype instead of event_sourcetype as a field name in the JSON input data also works, but you end up with an ingested event with a sourcetype field that has two identical values (for example, xyz_123 and xyz_123 ). I'm torn, and would appreciate advice on this. On the one hand, I'd prefer not to coin my own field name; on the other, I'm not comfortable with sourcetype having two values. That just looks weird to me, and I don't have enough experience with Splunk to know whether this will bite me in the a... ... View more

Well, that was interesting. I defined a TCP input in inputs.conf : [tcp://:6666] index = test sourcetype = xyz with a corresponding stanza in props.conf : [source::tcp:6666] INDEXED_EXTRACTIONS = JSON As an initial test, I used a Windows PowerShell script to send a few events in JSON format, and confirmed in Splunk Web that the following search: sourcetype=xyz displayed the events, with the field names and values extracted from the JSON. So far so good. Then I added: "sourcetype":"xyz_123" to the JSON, and sent that. The event appears in the Splunk Web Events tab with two values for the sourcetype field: xyz and xyz_123 . There's no new or renamed field: just the one sourcetype field with two values. That event appears if I use the search cited above, but if I change the search to: sourcetype=xyz_123 I get no results. I'm now about to try overriding the sourcetype as described in Splunk docs. Just for fun, I might try doing that using a sourcetype field, and see what happens: I wonder whether Splunk will "collapse" the two (now identical) values into one, or show them as separate values. Probably safer, though (since I don't understand the underlying code), to use a different field name. ... View more

Thanks, I'll try that first and report back. ... View more

Thanks, @somesoni2! Yeah, I rename series in charts that display a legend. For single-series charts, I typically don't bother (renaming or displaying a legend). If you can convert your comment to an answer, I'm happy to accept it. It's very useful for me to get feedback here on what I'm doing as I'm learning Splunk, especially from someone with high karma points (which I'm roughly equating to "experience"). Thanks again. ... View more

The solution I came up with (mentioned in my previous comment) was flawed, because xyseries does not produce the same "intelligent" X-axis labels as timechart . @Jeremiah provided the following improved solution (in response to a different question😞 sourcetype=my_log_type | bin _time | stats count by _time, conn_type | lookup connection_types.csv conn_type output description | timechart sum(count) as count by description ... View more

Thank you, @Jeremiah! That works for me. I've removed the span=1s option after reading the docs: bucket (and bin ) seem to share the same default spanning behavior as timechart . I've also replaced the bucket command name with bin , because - tell me if I'm wrong - the bin command seems to be the "primary" command (for which bucket is an alias): the Splunk docs topic for bucket refers the reader to the bin topic. I'd like to convert your comment into an answer so that I can accept it, but I can't see how to do that. I'm guessing I lack the authority - or karma points - for that option to appear in my user interface. Could you (or anyone reading this) please do that for me, or point me to where I can do that myself? So, pushing timechart to the end of the search solves my problem. I'm still curious, though: timechart seems to be "doing stuff under the covers" (perhaps: generating "internal use only fields" that Splunk "hides" from users?) that I do not (yet?) have the wit to see. ... View more