Hello from the future! I ran into this exact thing working with OpenTelemetry's HEC exporter.  It turns out if the "raw" field contains a "\n" it will increment the linecount to 2.  This is likely because Splunk's default line breaker behaviour is to drop any new lines as part of the capture group it uses "([\r\n]+)" so the logic assumes new lines would be removed. so if it somehow sneaks into raw (in my case it skips linebreaking due to hec event endpoint)  even tho you dont see it in the raw even on screen it looks to trigger the linecount to be 2.      ... View more

Yeah, I linked that in my original answer. Let us know when you try and how it goes! ... View more

Not yet, sorry, will have to try next week.  ... View more

I think it will just use reload. will try and replicate in my stack when i get the chance and let you know. Docs say it should just be a reload automagically for props/tranforms changes.  ... View more

I wouldn't give up yet...we have a lot of behind the scenes logic that takes care of distributing your configs in cloud and reducing the need for restarts, and the _reload option is available in cloud...are you certain you need a refresh? Generally props, i dont believe should require them anymore... https://docs.splunk.com/Documentation/SplunkCloud/9.0.2303/Admin/RollingRestart#Reload_or_restart_behavior_of_common_.conf_files https://docs.splunk.com/Documentation/Splunk/9.0.4/Admin/Configurationfilechangesthatrequirerestart#Index-time_settings are you hitting some sort of issue or error? I can always test your app in my stack if you want... ... View more

Splunk can definitely what you are trying to do, maybe in too many ways :).  I think finding the one that works for you is the key and I think you are close in your options 1 & 2.  (I'd ditch #3.) option #1 this is probably the simplest way.  I'd start here.  I am not sure I grok why the source field is a problem with using the data, but it can definitely be replaced using props/transforms versus hardcoding in the inputs. This should allow Splunk to use it to grab your desired hostname with "host_regex" in inputs.conf, then we can overwrite it in a props/transforms as the data flows through the pipeline. I suggest simply using an ingest eval stanza like:   ## props.conf [my:sourcetype] TRANSFORMS = source_override ## transforms.conf [source_override] INGEST_EVAL = source:="my_simple_source"   Option #2  This may provide even more flexibility in your logic for setting the fields you want. I would also use  "ingest_eval" here as well versus the general props/transforms as it provides some powerful logic that regex alone may not be suitable for.  https://docs.splunk.com/Documentation/Splunk/9.0.4/Data/IngestEval I believe the issue you may be hitting with losing the hostname is that your current approach relies on the hostname being present in the raw data, which may not happen when linebroken.  Instead with ingest_eval we can write some logic that allows us to manipulate the metadata you get to achieve your goal.  Technically you would just be re-implementing the "host_regex" logic here tho, so might be overkill, but might be useful if needed for advanced uses and wouldn't be limited to only the source field.    ## props.conf [my:sourcetype] TRANSFORMS = extract_host_from_source,source_override ## transforms.conf [extract_host_from_source] SOURCE_KEY = MetaData:Source REGEX = <some_regex_that_extracts_your_host_value> FORMAT = host::$1 DEST_KEY = MetaData:Host # once you have the hostname from source, now overwrite it! [source_override] INGEST_EVAL = source:="my_simple_source"   tldr: ingest_eval is mad powerful and allows you to refer to fields or metadata that exists already and apply powerful eval logic that goes well beyond what I showed here. I've even done conditional field overrides based on fields extracted, etc. It's super cool and leads into ingest actions world eventually.  Note: I wrote this pseudocode style, haven't tested them against data. If you have some sample data happy to try and tune them if you run into issues.  References: https://docs.splunk.com/Documentation/Splunk/9.0.4/Data/Overridedefaulthostassignments https://docs.splunk.com/Documentation/Splunk/9.0.4/Data/IngestEval  https://docs.splunk.com/Documentation/Splunk/9.0.4/Admin/Transformsconf#:~:text=INGEST_EVAL%20%3D%20%3Ccomma%2Dseparated,Optional.%0A*%20Default%3A%20empty   ... View more

I don't believe you can run /debug/refresh in cloud, at least not how I tried it  https://stackname .splunkcloud.com/en-US/debug/refresh That being said restarts aren't needed for much anymore these days. Are you certain you need to or is it habit? can you achieve what you need with the _reload endpoint instead? https://docs.splunk.com/Documentation/Splunk/9.0.4/RESTUM/RESTusing#Reload_endpoint   ... View more

kudos to you to mistrusting and verifying!!!  🙂  ... View more

I believe these can be safely ignored as "keep alive" calls from firehose/load balancers checking the connection but not sending data.  Putting in docs feedback on troubleshooting hec and firehose docs for future reference ... View more

Check out our otel image. It replaces fluentd-hec anyways. ... View more

Yes, check out our otel agent! ... View more

Use our otel agent and you can use its recombine operator to solve for multiline logs. ... View more

Greetings from the future! it is 2023 and the best way to handle docker logs is the OTel agent which has features to handle multiline log reassembly and much more! this replaces the guidance I provided to use fluentd-hec, which still works, but will be end of support january 2024. i do not recommend using the docker log driver or plugin at all.  if you read this and need help hit me up on splk.it/slack usergroups or check out the otel collector repo  https://github.com/signalfx/splunk-otel-collector     ... View more

No special config needed on splunk side..configuration of the docker image is all done via environment variables or default.yml that passes to ansible. Also you can load splunk apps in as well to handle splunk specific configs. See the advanced document I linked. ... View more

Hi, the UF does not have a UI. What exact image are you trying to use? Splunk/splunk or splunk/universalforwarder?   https://github.com/splunk/docker-splunk/blob/develop/docs/ADVANCED.md#runtime-configuration     ... View more

Im back in 2021 🙂 The "considerations around how SPL is crafted" I alluded to means you either need to search using indexed field syntax, like  k8s.cluster.name::foo   or you need to configure fields.conf with your indexed fields as described in docs https://docs.splunk.com/Documentation/Splunk/8.2.4/Data/Configureindex-timefieldextraction#Where_to_put_the_configuration_changes_in_a_distributed_environment Or finally, as we currently do in Cloud, presumably as a tradeoff for customer experience vs performance worries... always_include_indexedfield_lispy = <boolean> * Whether or not search always looks for a field that does not have "INDEXED = true" set in fields.conf using both the indexed and non- indexed forms. * If set to "true", when searching for <field>=<value>, the lexicon is searched for both "<field>::<value>" and "<value>". * If set to "false", when searching for <field>=<val>, the lexicon is searched only for "<value>". * Set to "true" if you have fields that are sometimes indexed and sometimes not indexed. * For field names that are always indexed, it is much better for performance to set "INDEXED = true" in fields.conf for that field instead. * Default: false https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Limitsconf#:~:text=always_include_indexedfield_lispy%20%3D%20%3Cboolean%3E%0A*%20Whether,instead.%0A*%20Default%3A%20false As always, being lazy is fun, but will cost perf. Setting the fields.conf is definitely preferable, especially when the fields don't change very much and just take a little bit of up front work/planning.  choose your weapon wisely, friends... ... View more

Looks like `hostnameOption` is Windows only, fyi.. hostnameOption = [ fullyqualifiedname | clustername | shortname ] * The type of information to use to determine how splunkd sets the 'host' value for a Windows Splunk platform instance when you specify an input stanza with 'host = $decideOnStartup'. * Applies only to Windows hosts, and only for input stanzas that use the "host = $decideOnStartup" setting and value. * Valid values are "fullyqualifiedname", "clustername", and "shortname". * The value returned for the 'host' field depends on Windows DNS, NETBIOS, and what the name of the host is. * 'fullyqualifiedname' uses Windows DNS to return the fully qualified host name as the value. * 'clustername' also uses Windows DNS, but sets the value to the domain and machine name. * 'shortname' returns the NETBIOS name of the machine. * Cannot be an empty string. * Default: shortname    ... View more

Awesome! Nice work! ... View more