topic Re: Forwarded log data is not being parsed correctly. Is there something wrong with my current configuration setup? in Getting Data In

Forwarded log data is not being parsed correctly. Is there something wrong with my current configuration setup?

shailesh030 — Mon, 28 Sep 2020 18:18:47 GMT

Hi!,

I have a splunk setup in which log files are being forwarded by an universal forwarder to an indexer and a search head is being used to perform the search
I have keeping the configuration files in etc/apps/app123/local in searchhead and indexer respectively.
Following are the contents of my configuration files:

On the universal forwarder:
inputs.conf (in apps/local)

[monitor:///home/abc/appLogs.txt]
sourcetype = applogs
blacklist = .(gz)$
index=main

On the search head:
props.conf

[applogs]
REPORT-parse_server=applogs
KV_MODE=none

transforms.conf

[applogs]
DELIMS = "~"
FIELDS = "Text","device_name","domain_name","OperationName","txn_id","time_stamp","FAULT","FaultCode"

On the indexer:
props.conf

[applogs]
TIMESTAMP_FIELDS = StartTimeStamp,ExitTimeStamp,App1StartTimeStamp
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=%b %d %H:%M:%S
TIME_PREFIX=^
TZ=UTC
TRUNCATE=300000
pulldown_type = 1

In the search head UI raw data, I can see the events being indexed with the correct sourcetype but they are not being mapped to fields given in transforms.conf

I have ran btool against each of the config files & no issues were found. The config files are only in apps & none in system/local so it can't be a precedence issue.
I also tried by putting all configurations (props + transforms) into props.conf and keeping them in etc/app/local in searchhead and indexer.

I am not able to figure out what am I missing or where am I going wrong.
Any help will be highly appreciated.

Re: Forwarded log data is not being parsed correctly. Is there something wrong with my current configuration setup?

RicoSuave — Tue, 02 Dec 2014 21:07:21 GMT

Have you tried setting KV_MODE = AUTO? Setting it to NONE disables field extraction for that sourcetype.

Re: Forwarded log data is not being parsed correctly. Is there something wrong with my current configuration setup?

shailesh030 — Tue, 02 Dec 2014 21:41:28 GMT

I changed props.conf on the search head to KV_MODE=AUTO, restarted splunkd but it still doesn't extract the fields.

Re: Forwarded log data is not being parsed correctly. Is there something wrong with my current configuration setup?

RicoSuave — Tue, 02 Dec 2014 21:55:16 GMT

Oh, try DELIMS = ","

Re: Forwarded log data is not being parsed correctly. Is there something wrong with my current configuration setup?

shailesh030 — Tue, 02 Dec 2014 22:12:30 GMT

But my log data is delimited by tilda "~" . Nevertheless, I tried changed the ~ to "," in DELIMS in transforms.conf and it still didn't work.

Re: Forwarded log data is not being parsed correctly. Is there something wrong with my current configuration setup?

RicoSuave — Tue, 02 Dec 2014 22:15:20 GMT

Can you post a sample of the applogs events?

Re: Forwarded log data is not being parsed correctly. Is there something wrong with my current configuration setup?

shailesh030 — Tue, 02 Dec 2014 22:24:25 GMT

Thanks Joetron .. here are some of the applogs events. Each event is in one line

Aug 4 07:02:43 ABC-XY12345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY12345~ALPHA~GCP~55403201~2014-08-04 07:02:43~FAULT~12345
Aug 4 07:02:44 ABC-XY22345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY22345~ALPHA~GCP~65403201~2014-08-04 07:02:44~FAULT~12346
Aug 4 07:02:45 ABC-XY32345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY32345~ALPHA~GCP~75403201~2014-08-04 07:02:45~FAULT~12347
Aug 4 07:02:46 ABC-XY42345-Default [XYZ][123][xsltmsg][info] #ABCD-IN#~XY42345~ALPHA~GCP~85403201~2014-08-04 07:02:46~FAULT~12348