topic Couldn't parse and extract mixed data (json and text) in Getting Data In

Couldn't parse and extract mixed data (json and text)

nareshinsvu — Fri, 30 Aug 2019 02:06:28 GMT

Hi,

I am not able to send my logfile into 2 sourcetypes (json and non-json). Below is my config. I know the fix might be a simple one. It's just that I am not getting anywhere near it. Need your expertise solution.

:::::::::::::::::inputs.conf:::::::::::::::: Only 1 monitor stanza is being picked. How can I parse the log file into two source types? Or any better solution?

[monitor://\\server1\sdata$\] 
sourcetype = Custom_W22 
index=0_nojson 
whitelist = (app1.log) 
recursive=false 
interval = 10 
crcSalt = <SOURCE> 


[monitor://\\server1\sdata$] 
sourcetype = myjson 
index=0_myjson 
whitelist = (\app1.log) 
recursive=false 
interval = 10 
crcSalt = <SOURCE>

::::::::::::props.conf::::::::::::::::::::::::::::::
-----Here I have used current DATETIME_CONFIG which ignores my timestamps. I can fix the timestamps after indexing. Otherwise, the parsing was slow---------

[Custom_W22] 
NO_BINARY_CHECK = true 
SHOULD_LINEMERGE = false 
category = Miscellaneous 
TRANSFORMS-set = discardAll,queue2resp
disabled = false 
pulldown_type = true 
DATETIME_CONFIG = CURRENT 
TRUNCATE = 100000 

[myjson] 
SEDCMD-strip_prefix = s/^[^{]+//g 
INDEXED_EXTRACTIONS=JSON 
NO_BINARY_CHECK = true 
category = Custom 
description = myjson custom 
disabled = false 
pulldown_type = true 
TRUNCATE = 100000 
MAX_EVENTS = 10000

::::::::::::::::::transforms.conf::::::::::::::::::::::::::

[discardAll] 
REGEX=. 
DEST_KEY=queue 
FORMAT=nullQueue 

[queue2resp] 
REGEX=(\<Tracker\s+\[pointer) 
DEST_KEY=queue 
FORMAT=indexQueue

Re: Couldn't parse and extract mixed data (json and text)

sandyIscream — Fri, 30 Aug 2019 04:27:13 GMT

@nareshinsvu Please go through the following document and let me know if it helps!

https://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Advancedsourcetypeoverrides

Re: Couldn't parse and extract mixed data (json and text)

nareshinsvu — Fri, 30 Aug 2019 05:44:26 GMT

Hi @sandyIscream - This doesn't help as myjson of my props doesn't need transforms.conf. So, I cant use transforms regex and seperate json events to a different index / sourcetype.

A working example of the config will actually help me to try and test

Re: Couldn't parse and extract mixed data (json and text)

nareshinsvu — Fri, 30 Aug 2019 05:50:59 GMT

More observations:
1)
Both the stanzas work fine if the source logfile is different. My requirement is that I need ONE logfile parsed to TWO sourcetypes.

2) While testing JSON data alone, found that "crcSalt = <SOURCE> " is not working. A new line addition at the tail of the log is re-indexing the whole log and duplicating my splunk events.
I am able to fix it by using below config. Need to know if there are any drawbacks with this approach in the future? Note: followTail is working fine even if the logfile gets archived and a new file with same name gets created.

crcSalt = <SOURCE>
followTail = 1

Re: Couldn't parse and extract mixed data (json and text)

nareshinsvu — Tue, 03 Sep 2019 06:21:56 GMT

:::::::::::::::::::Work around::::::::::::

Monitor JSON of 1.log - configure input, props and transforms at Forwarder end to push data to the Indexer
Monitor Non-json of same 1.log - configure input, props and transforms at Indexer end to pull data from Forwarder

I am not sure if this a recommended approach. Splunkexperts - please suggest