Getting Data In

Why is there event duplication via TCP port?

patriziadepaola
Explorer

Can anyone help me and clarify why Splunk duplicates events received from TCP port? The same type of events received on a UDP port are not duplicated.

I try to post an example:
Event received on UPD port 55553

{"CALLER_INFO":"{CORPORATE#010#NET43205#null#null}","INPUT":"{COMPANY#null#null#null#03978500720#null#null#null-null}","APPLICATION_SERVICE":"MAGNETO.EXTERNALSEARCH","ESITO":"OK","OUTPUT":"0","APPLICATION":"magneto","timestamp8601":"2017-05-18T09:07:02.400389+00:00","PID":"707633604","STEP":"TOTAL","program":"journal","CLASS":"LogUtil.traceStep","message":["2017-05-18T09:07:02.400389+00:00 prod-dcos6-12102016 journal: MSG;1501718321;707633604;MAGNETO.EXTERNALSEARCH;LogUtil.traceStep;18/05/2017 11:07:02,399;SPLUNK - magneto/externalsearch/magneto_externalsearch|TOTAL|OK|1301|{CORPORATE#010#NET43205#null#null}|{COMPANY#null#null#null#03978500720#null#null#null-null}|0\n","MSG;1501718321;707633604;MAGNETO.EXTERNALSEARCH;LogUtil.traceStep;18/05/2017 11:07:02,399;SPLUNK - magneto/externalsearch/magneto_externalsearch|TOTAL|OK|1301|{CORPORATE#010#NET43205#null#null}|{COMPANY#null#null#null#03978500720#null#null#null-null}|0\n"],"type":"rsyslog_produzione_dcos","logsource":"prod-dcos6-12102016","tags":["journal"],"SID":"1501718321","DATE":"18/05/2017 11:07:02,399","@timestamp":"2017-05-18T09:07:03.772Z","EXECUTION_TIME":1301,"@version":"1","LABEL":"SPLUNK","SERVICE":"externalsearch/magneto_externalsearch","LOGLEVEL":"MSG"}

Event received on TCP port 55555

{"CALLER_INFO":"{CORPORATE#010#NET43205#null#null}","INPUT":"1300013","APPLICATION_SERVICE":"MAGNETO.EXTERNALSEARCH","ESITO":"OK","OUTPUT":"AUTHORIZED","APPLICATION":"magneto","timestamp8601":"2017-05-19T12:28:45.940854+00:00","PID":"1528829935","STEP":"IS_AUTHORIZED_CONSUMPTION","program":"journal","CLASS":"LogUtil.traceStep","message":["2017-05-19T12:28:45.940854+00:00 prod-dcos6-12102016 journal: MSG;1937509434;1528829935;MAGNETO.EXTERNALSEARCH;LogUtil.traceStep;19/05/2017 14:28:45,940;SPLUNK - magneto/externalsearch/magneto_externalsearch|IS_AUTHORIZED_CONSUMPTION|OK|46|{CORPORATE#010#NET43205#null#null}|1300013|AUTHORIZED\n","MSG;1937509434;1528829935;MAGNETO.EXTERNALSEARCH;LogUtil.traceStep;19/05/2017 14:28:45,940;SPLUNK - magneto/externalsearch/magneto_externalsearch|IS_AUTHORIZED_CONSUMPTION|OK|46|{CORPORATE#010#NET43205#null#null}|1300013|AUTHORIZED\n"],"type":"rsyslog_produzione_dcos","logsource":"prod-dcos6-12102016","tags":["journal"],"SID":"1937509434","DATE":"19/05/2017 14:28:45,940","@timestamp":"2017-05-19T12:28:47.080Z","EXECUTION_TIME":46,"@version":"1","LABEL":"SPLUNK","SERVICE":"externalsearch/magneto_externalsearch","LOGLEVEL":"MSG"}{"CALLER_INFO":"{CORPORATE#010#NET43205#null#null}","INPUT":"{COMPANY#null#null#01893500890#null#null#null#null-null}","APPLICATION_SERVICE":"MAGNETO.EXTERNALSEARCH","ESITO":"OK","OUTPUT":"0","APPLICATION":"magneto","timestamp8601":"2017-05-19T12:28:46.449567+00:00","PID":"1528829935","STEP":"LIMINIRIS_REQUEST","program":"journal","CLASS":"LogUtil.traceStep","message":["2017-05-19T12:28:46.449567+00:00 prod-dcos6-12102016 journal: MSG;1937509434;1528829935;MAGNETO.EXTERNALSEARCH;LogUtil.traceStep;19/05/2017 14:28:46,448;SPLUNK - magneto/externalsearch/magneto_externalsearch|LIMINIRIS_REQUEST|OK|508|{CORPORATE#010#NET43205#null#null}|{COMPANY#null#null#01893500890#null#null#null#null-null}|0\n","MSG;1937509434;1528829935;MAGNETO.EXTERNALSEARCH;LogUtil.traceStep;19/05/2017 14:28:46,448;SPLUNK - magneto/externalsearch/magneto_externalsearch|LIMINIRIS_REQUEST|OK|508|{CORPORATE#010#NET43205#null#null}|{COMPANY#null#null#01893500890#null#null#null#null-null}|0\n"],"type":"rsyslog_produzione_dcos","logsource":"prod-dcos6-12102016","tags":["journal"],"SID":"1937509434","DATE":"19/05/2017 14:28:46,448","@timestamp":"2017-05-19T12:28:47.080Z","EXECUTION_TIME":508,"@version":"1","LABEL":"SPLUNK","SERVICE":"externalsearch/magneto_externalsearch","LOGLEVEL":"MSG"}{"CALLER_INFO":"{CORPORATE#010#NET43205#null#null}","INPUT":"{COMPANY#null#null#01893500890#null#null#null#null-null}","APPLICATION_SERVICE":"MAGNETO.EXTERNALSEARCH","ESITO":"OK","OUTPUT":"2137352876","APPLICATION":"magneto","timestamp8601":"2017-05-19T12:28:46.540997+00:00","PID":"1528829935","STEP":"BUILD_ACCOUNT","program":"journal","CLASS":"LogUtil.traceStep","message":["2017-05-19T12:28:46.540997+00:00 prod-dcos6-12102016 journal: MSG;1937509434;1528829935;MAGNETO.EXTERNALSEARCH;LogUtil.traceStep;19/05/2017 14:28:46,540;SPLUNK - magneto/externalsearch/magneto_externalsearch|BUILD_ACCOUNT|OK|91|{CORPORATE#010#NET43205#null#null}|{COMPANY#null#null#01893500890#null#null#null#null-null}|2137352876\n","MSG;1937509434;1528829935;MAGNETO.EXTERNALSEARCH;LogUtil.traceStep;19/05/2017 14:28:46,540;SPLUNK - magneto/externalsearch/magneto_externalsearch|BUILD_ACCOUNT|OK|91|{CORPORATE#010#NET43205#null#null}|{COMPANY#null#null#01893500890#null#null#null#null-null}|2137352876\n"],"type":"rsyslog_produzione_dcos","logsource":"prod-dcos6-12102016","tags":["journal"],"SID":"1937509434","DATE":"19/05/2017 14:28:46,540","@timestamp":"2017-05-19T12:28:47.100Z","EXECUTION_TIME":91,"@version":"1","LABEL":"SPLUNK","SERVICE":"externalsearch/magneto_externalsearch","LOGLEVEL":"MSG"}{"CALLER_INFO":"{CORPORATE#010#NET43205#null#null}","INPUT":"{COMPANY#null#null#01893500890#null#null#null#null-null}","APPLICATION_SERVICE":"MAGNETO.EXTERNALSEARCH","ESITO":"OK","OUTPUT":"0","APPLICATION":"magneto","timestamp8601":"2017-05-19T12:28:46.541236+00:00","PID":"1528829935","STEP":"TOTAL","program":"journal","CLASS":"LogUtil.traceStep","message":["2017-05-19T12:28:46.541236+00:00 prod-dcos6-12102016 journal: MSG;1937509434;1528829935;MAGNETO.EXTERNALSEARCH;LogUtil.traceStep;19/05/2017 14:28:46,540;SPLUNK - magneto/externalsearch/magneto_externalsearch|TOTAL|OK|647|{CORPORATE#010#NET43205#null#null}|{COMPANY#null#null#01893500890#null#null#null#null-null}|0\n","MSG;1937509434;1528829935;MAGNETO.EXTERNALSEARCH;LogUtil.traceStep;19/05/2017 14:28:46,540;SPLUNK - magneto/externalsearch/magneto_externalsearch|TOTAL|OK|647|{CORPORATE#010#NET43205#null#null}|{COMPANY#null#null#01893500890#null#null#null#null-null}|0\n"],"type":"rsyslog_produzione_dcos","logsource":"prod-dcos6-12102016","tags":["journal"],"SID":"1937509434","DATE":"19/05/2017 14:28:46,540","@timestamp":"2017-05-19T12:28:47.112Z","EXECUTION_TIME":647,"@version":"1","LABEL":"SPLUNK","SERVICE":"externalsearch/magneto_externalsearch","LOGLEVEL":"MSG"}

Has anyone seen anything like it before?

Tags (3)
0 Karma

DalJeanis
Legend

Those are not duplicates. Each chunk of the JSON has distinct attributes that are not identical to any other chunk.

The only thing you have to look at to verify I am correct is the number after "EXECUTION_TIME":. It is different in every block.

0 Karma
Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Dynamic formatting from XML events

This challenge was first posted on Slack #puzzles channelFor a previous puzzle, I needed a set of fixed-length ...

Enter the Agentic Era with Splunk AI Assistant for SPL 1.4

  🚀 Your data just got a serious AI upgrade — are you ready? Say hello to the Agentic Era with the ...

Stronger Security with Federated Search for S3, GCP SQL & Australian Threat ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...