How to add data from the Linux machines to Splunk?

[pbnl]

hi all,

I'm completely new to Splunk and have some problems understanding the dataflow and what to configure where.
i have here a working environment with 2 indexers, 1 heavy forwarder which is the search head too. all running version 7.3.6 on ubuntu 20.04. additionally there a several dozen windows servers and ~50 linux servers. a lot of them have splunkforwarder installed and send data to the indexers. this was set up some years ago by some guys that left the company meanwhile.
my task now is to add data from the linux machines to splunk. as i have a working environment and a lot of stuff to see how it's done on other machines, it didn't sound too complicated. but...

the task: have on all linux servers the same task running which creates a log file in /var/log/
my solution: on a server that already sends data to splunk, i ran: splunk add monitor /var/log/mylog
the result: the data shows up in splunk. yepeee. easy.
then i went to a server that does not send data to splunk.
my solution: download and install splunkforwarder-7.3.6-47d8552a4d84-linux-2.6-amd64.deb
splunk add forward-server indexer1:9997
splunk add forward-server indexer2:9997
splunk add monitor /var/log/mylog
yepee. data shows up on the search head

next task: have a dashboard with the data and have some filter options
my solution: found a similar dashboard and tried to adopt it to my needs. not that easy, but i get it done. without the filters first.
and then the problems start: the logfile contains headers and lots of other junk i cannot filter out easily. during my search on how to delete events, i found out that i have multiline events. i learned about LINE_BREAKER and SHOULD_LINEMERGE and indexes and other config stuff.

and here the confusion starts: where do i have to configure what? 
after reading some docs and different solutions here in the forum, i decided to start from zero with one of the linux servers. i deleted the results from this server from the main index.
source=/var/log/mylog myserver | delete
removed the forwarders and monitor from the linux server
splunk remove forward-server indexer1:9997
splunk remove forward-server indexer2:9997
splunk remove monitor /var/log/mylog
i created a new index on the 2 indexers and on the search head with the GUI. lets call it myindex and i didn't change the defaults
i modified etc/users/admin/myapp/local/props.conf file on the search head, because that was the only place where i could find a reference to the monitor i've added.

[mylog-too_small]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
[mylog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)

adding forwarders and monitor again:
splunk add forward-server indexer1:9997
splunk add forward-server indexer2:9997
splunk add monitor /var/log/mylog
What the heck? no data shows up on the search head

What have I missed where?
and in what order are all these props.conf files applied?
I have some of them in different folders

any help or hint is welcome 🙂

[somesoni2]

I would start from this documentation page to how data progresses through various pipelines and Splunk instances. 

https://docs.splunk.com/Documentation/Splunk/latest/Data/WhatSplunkdoeswithyourdata

[richgalloway]

There is also this useful, if slightly dated, site: https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F