Hi, I want to preface I understand that props isn't fully processed if you install it on the universal forwarder. My question is about the difference between the install of a Splunk Universal Forwarder vs Splunk converted to a Forwarder license. My setup is a fresh install of an ancient 6.5.2 SplunkForwarder and 6.5.2 Splunk in Forwarder Mode on two machines and mapped like this. Server1: Universal Forwarder --> Indexer Server2: Splunk (updated to run as a Forwarder) --> Indexer. On the indexer I have a props change (a trivial SEDCMD-test = s/a/o/g ) If I install the same serverclass on both servers that reads a /tmp/test.log and where I write some lines with letters a in them the Server 1's messages end up changed from a to o while Server 2's do not, they staay as a. I've tested it with multiple server installs (albeit on an old version 6.5.2). It seems to me that Splunk in forwarder, unlike a dedicated Splunk Universal Forwarder, applies some kind of tag that prevents downstream props/transforms changes to occur. A raw message coming from Universal Forwarder is then processed by the indexer's props/transforms while a message coming from Splunk in Forwarder mode does not. Note: I checked for any silliness. Both Servers send to the same indexer, have identical serverclass, sourcetype, inputs and outputs. And the props on the indexer only applies to a sourcetype not any specific host (btool matches on source servers) My questions are listed below. Can you confirm that there is some kind of cooked tag on events coming from Splunk (in forwarder mode) that tells downstream systems not to apply props/transforms and just write immediately to an index? Is there anything I can do on the server with Splunk in forwarder mode to behave exactly like a UniversalForwarder? Perhaps a etc conf change, or do I need to just uninstall it and setup Splunk UniversalForwarder from scratch (confirmed this works). How can I debug props/transforms issues between two servers. Turning on DEBUG mode didn't say anything useful in splunkd.log like "applying props [foo] on event 'Hello World'" Thanks! ... View more

[ANSWERED by to4kawa] props.conf should be [yoursourcetype] DATETIME_CONFIG = CURRENT SHOULD_LINEMERGE = false LINE_BREAKER = .*($) I have a Catch 22 issue. I want three things to happen. I want to monitor a log file and to combine all lines into a single event until the file is not updated for 2 seconds or more. I want to disable all time extraction, I want all the time set to current (ie _indextime). Being able to handle arbitrary data without any separators. Imagine my file is empty and then within a microsecond these three lines are added to my file. TEST1 Fri Apr 6 20:05:59 EDT 2020 TEST2 Fri Apr 3 20:04:30 EDT 2020 TEST3 Fri Apr 1 20:05:59 EDT 2020 I would like them all to be combined into a single event. And I want the timestamp to be set to current. If I just just start monitoring the file without any props it combines the events into a single one just fine. BUT it will try to parse the timestamps and they will be all over the place. If I mod props.conf and set DATETIME_CONFIG=CURRENT it will set the time to the current one but then split the events into single lines. So I am in a catch 22, I can have one or the other. Any ideas what inputs/props/transforms combo I can have that ignores all time stamps, and combines the events into a single one no matter what. I honestly want something that checks if file has not been modified for 2 seconds and then combine everything new that was added into ONE event. Thanks! ... View more

Hello, I can't seem to change maxout setting in limits.conf. I've tried changing every maxout or max_count param I could find but it is not taking effect. I tried changing other variables (like reduce_freq or ttl) and that seemed to work but not maxout or maxtime. This happens when I run sideview Search of "index=*" and then show results in a table. Here is the search log extract... 06-24-2019 17:51:40.147 INFO dispatchRunner - Search process mode: preforked (first search in process) 06-24-2019 17:51:40.147 INFO dispatchRunner - initing LicenseMgr in search process: nonPro=1 06-24-2019 17:51:40.147 INFO dispatchRunner - registering build time modules, count=1 06-24-2019 17:51:40.147 INFO dispatchRunner - registering search time components of build time module name=vix 06-24-2019 17:51:40.147 INFO dispatchRunner - Splunkd starting (build 67571ef4b87d). 06-24-2019 17:51:40.147 INFO dispatchRunner - System info: Linux, XXX, 3.10.0-957.5.1.el7.x86_64, #1 SMP Wed Dec 19 10:46:58 EST 2018, x86_64. 06-24-2019 17:51:40.148 INFO dispatchRunner - Detected 32 (virtual) CPUs, 16 CPU cores, and 128731MB RAM 06-24-2019 17:51:40.148 INFO dispatchRunner - Maximum number of threads (approximate): 16000 06-24-2019 17:51:40.148 INFO dispatchRunner - Arguments are: "search" "--id=1561413100.2" "--maxbuckets=0" "--ttl=600" "--maxout=10000" "--maxtime=8640000" "--lookups=1" "--reduce_freq=6" 06-24-2019 17:51:40.148 INFO dispatchRunner - Getting search configuration data from: /opt/splunk/etc/modules/parsing/config.xml 06-24-2019 17:51:40.152 INFO KVStoreBulletinBoardManager - MessageHandler:KVSTORE_FAILED removed ...<SNIP>... 06-24-2019 17:51:40.477 INFO DispatchThread - Job truncated due to max_count=10000 reached In the log above it is limiting results to 10000 . When I run the search index=_internal on the command line it works perfectly, I can override maxout. *splunk@XXX:[/opt/splunk/bin]> ./splunk search 'index=_internal' -maxout 10501 | wc -l 10501 * Here is the btool output.... splunk@xxx:[/opt/splunk/bin]> ./splunk btool limits list --debug | grep maxout /opt/splunk/etc/system/local/limits.conf subsearch_maxout = 50000 /opt/splunk/etc/system/local/limits.conf maxout = 5000 /opt/splunk/etc/system/local/limits.conf maxout = 5000 splunk@xxx:[/opt/splunk/bin]> ./splunk btool limits list --debug | grep max_count /opt/splunk/etc/system/local/limits.conf max_count = 10000000 /opt/splunk/etc/system/local/limits.conf alerts_max_count = 50000 /opt/splunk/etc/system/local/limits.conf max_count = 500000 /opt/splunk/etc/system/local/limits.conf max_count = 5000 splunk@xxx:[/opt/splunk/bin]> ./splunk btool limits list --debug | grep 10000 /opt/splunk/etc/system/local/limits.conf max_count = 10000000 /opt/splunk/etc/system/local/limits.conf max_number_of_ack_channel = 1000000 /opt/splunk/etc/system/local/limits.conf max_number_of_acked_requests_pending_query = 10000000 /opt/splunk/etc/system/local/limits.conf max_number_of_acked_requests_pending_query_per_ack_channel = 1000000 /opt/splunk/etc/system/local/limits.conf maxdatapoints = 100000000 /opt/splunk/etc/system/local/limits.conf max_memtable_bytes = 10000000 /opt/splunk/etc/system/local/limits.conf maxcount = 100000 /opt/splunk/etc/system/local/limits.conf batch_search_max_index_values = 10000000 /opt/splunk/etc/system/local/limits.conf batch_search_max_results_aggregator_queue_size = 100000000 /opt/splunk/etc/system/local/limits.conf batch_search_max_serialized_results_queue_size = 100000000 /opt/splunk/etc/system/local/limits.conf max_chunk_queue_size = 10000000 /opt/splunk/etc/system/local/limits.conf max_rawsize_perchunk = 100000000 /opt/splunk/etc/system/local/limits.conf result_queue_max_size = 100000000 /opt/splunk/etc/system/local/limits.conf max_valuemap_bytes = 100000 /opt/splunk/etc/system/local/limits.conf maxopenevents = 100000 /opt/splunk/etc/system/local/limits.conf chunk_size = 10000000 Nothing seems to match 10000. I went through the entire /etc stack and ensured there is no such setting there. Somehow no matter what I do it is picking up this 10000 maxout setting and I have no idea from where. Note, I am using sideview utils - not sure if it matters. Splunk version is 6.5.2 (so it is a bit old) - but this is a self contained UAT instance. ... View more