Parsing issue in distributed Environment

[Namdev]

Hello Team,parsing issue

I have built a distributed Splunk lab using a trial license. The lab consists of three indexers, one cluster manager, one search head, one instance serving as the Monitoring Console (MC), Deployment Server (DS), and License Manager (LM), along with two Universal Forwarders.

The forwarder is monitoring the /opt/log/routerlog directory, where I have placed two log files: cisco_ironport_web.log and cisco_ironport_mail.log. The logs are successfully forwarded to the indexers and then to the search head.

However, log parsing is not happening as expected. I have applied the same configuration of props.conf and transforms.conf on both the indexer cluster and the search head.

props.conf and transforms.conf file paths :

indexer path : /opt/splunk/etc/peer-apps/_cluster/local

Search head  path : /opt/splunk/etc/apps/search/local

configuration of props.conf and transforms.conf :

transforms.conf :

[extract_fields]

REGEX = ^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+(?P<src_ip>\d+\.\d+\.\d+\.\d+)\s+(?P<email>\S+@\S+)\s+(?P<domain>\S+)\s+(?P<url>\S+)

FORMAT = timestamp::$1 src_ip::$2 email::$3 domain::$4 url::$5

props.conf :

[custom_logs]

SHOULD_LINEMERGE = false

TIME_PREFIX = ^

TIME_FORMAT = %Y-%m-%d %H:%M:%S

MAX_TIMESTAMP_LOOKAHEAD = 19

TRANSFORMS-extract_fields = extract_fields

[isoutamo]

Hi

peer-apps are place where MN will deploy those apps. It’s not used on MN. There you should use manager-apps instead of it. There on your lab _cluster is ok for testing, but for any real environment you should use separate apps. 
Are you sure that your REGEX is correct? Can you give a sample to us from both log files? Use </> as a code block. That way we could be sure that examples are what you have!

Also could you add your inputs.conf also so we see what you have defined there?

[kiran_panchavat]

@Namdev 

Did you complete the following steps?

1. Copy the app to the $SPLUNK_HOME/etc/manager-apps directory on the cluster master node.
2. Push the app from the cluster master to the peer nodes by running the command:

   /opt/splunk/bin/splunk apply cluster-bundle

   This updates the cluster configurations on the peer nodes.
3. Verify on the indexers that the app is present in the /opt/splunk/etc/peer-apps directory.
4. If the app is not visible, refer to the official documentation for detailed instructions on how to push the app from the cluster master.  https://docs.splunk.com/Documentation/Splunk/9.4.0/Indexer/Manageappdeployment 

[Namdev]

I have tried this option, no luck. Can we have a call to discuss this ?

[kiran_panchavat]

@Namdev

I suggest starting with a standalone test instance. Create your props.conf and transforms.conf files in either the /opt/splunk/etc/system/local or app/local directory, then restart the Splunk instance. After that, open the web interface of the same instance, navigate to the "Add Data" option, and upload your sample log file. Apply your custom sourcetype, "custom_logs," and verify if it's working as expected. If everything looks good, proceed to update the same configuration in the cluster using the cluster master.

[kiran_panchavat]

@Namdev 

Did you deploy the props.conf and transforms.conf files through the cluster manager? You need to create an app on the cluster manager under /opt/splunk/etc/master-apps/ or /opt/splunk/etc/manager-apps/. Once the app is deployed, it should be propagated to the indexers, appearing under /opt/splunk/etc/peer-apps/ or /opt/splunk/etc/slave-apps/. Please verify if you have correctly created and deployed the app containing the props.conf and transforms.conf configurations.

Update common peer configurations and apps - Splunk Documentation

[Namdev]

Yes, I  tried using the app option also checked with the _cluster option where I placed the props.conf and transforms.conf files, and distributed them among the peers.