topic Re: UF is not forwarding the JSON data to indexers in Getting Data In

UF is not forwarding the JSON data to indexers

newsplunker1 — Wed, 30 Sep 2020 05:07:39 GMT

Hello,

I configured the UF to monitor a JSON file in a specific directory but its not forwarding it to the indexers

the output is working properly as there are files being sent to indexers

here is my input file
[monitor://C:\temp*.json]
index=test1
sourcetype=test_styp

my props
[test_styp]
INDEXED_EXTRACTIONS =json
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N+%4N
TIME_PREFIX="observedTime":"
MAX_TIMESTAMP_LOOKAHEAD=28

the splunk logs is stating the following " Adding watch on path splunk [monitor://C:\temp*] but nothis being ingested

i tried running this SPL search on my SH to check if something related to JSON extraction is but nothing returned

test_styp | rex "incoming=\"(?.+)\", transformed=" | spath = incoming

Could you please help ?

Re: UF is not forwarding the JSON data to indexers

richgalloway — Wed, 22 Apr 2020 20:10:06 GMT

Search for index=test1 sourcetype=test_styp to see if you find anything. Searches should always specify an index name.
Verify Splunk can read the files. Run splunk list monitor on the UF to see if the file is really being monitored.

Re: UF is not forwarding the JSON data to indexers

PavelP — Wed, 22 Apr 2020 20:10:32 GMT

Hello @newsplunker1

can you check that your monitor stanza includes disabled = 0? If you don't set it to 0 (zero), then it is disabled by default:

disabled = [0|1]
* Whether or not the event collector input is active.
* Set this setting to "1" to disable the input, and "0" to enable it.
* Default: 1 (disabled).

Re: UF is not forwarding the JSON data to indexers

manjunathmeti — Wed, 22 Apr 2020 20:12:42 GMT

1. If temp is directory then monitor should be:

 [monitor://C:\temp\*.json]

2. If not, then check user running splunk forwarder service has access to C:\temp*.json.

3. Search the data with index name.

index=test1 sourcetype=test_styp

Re: UF is not forwarding the JSON data to indexers

newsplunker1 — Thu, 23 Apr 2020 12:21:01 GMT

Thanks rich,
I tried that but nothing returned - i tried the splunk list command and showed no directory is being monitored which is weird because i have other directories working properly .

Re: UF is not forwarding the JSON data to indexers

newsplunker1 — Thu, 23 Apr 2020 12:21:33 GMT

Thanks Pave - Did that but no changes

Re: UF is not forwarding the JSON data to indexers

PavelP — Thu, 23 Apr 2020 12:23:17 GMT

Have you restarted splunk?

Re: UF is not forwarding the JSON data to indexers

newsplunker1 — Thu, 23 Apr 2020 12:37:43 GMT

Thanks Manjunath,

I actually have it that way with the temp*.json . And i tried the full syntax ( index,sourcetype) , nothing changed. I checked the user access and has a full access to that path.

Re: UF is not forwarding the JSON data to indexers

newsplunker1 — Thu, 23 Apr 2020 12:40:21 GMT

Yes i restarted after making the changes . I keep seeing this "TailingProcessor - Adding watch on path: C:\temp\ . so to me , its able to see the path but not able to read it ? if so , the splunk account has access to that path , so i dont know whats going on

Re: UF is not forwarding the JSON data to indexers

PavelP — Thu, 23 Apr 2020 12:58:20 GMT

run this query in CMD (adjust the splunk path as needed):

C:\programfiles\splunkforwarder\bin\splunk.exe    _internal call /services/admin/inputstatus/TailingProcessor:FileStatus

Re: UF is not forwarding the JSON data to indexers

newsplunker1 — Thu, 23 Apr 2020 19:26:04 GMT

So i got the file now ingested into indexers (There is something wrong with the file format) but im having problem extracting the JSON fields properly .

Re: UF is not forwarding the JSON data to indexers

newsplunker1 — Wed, 30 Sep 2020 05:07:58 GMT

So i got the file now ingested into indexers (There was something wrong with the file format) but im having problems extracting the JSON fields properly . im not getting all of the lines .

Here is my props file now

[test]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Structured
disabled = false
pulldown_type = true

Re: UF is not forwarding the JSON data to indexers

PavelP — Thu, 23 Apr 2020 19:52:01 GMT

Hello @newsplunker1

glad you worked it out!

Please create a new question, so more people can see it and help!

Re: UF is not forwarding the JSON data to indexers

richgalloway — Fri, 24 Apr 2020 14:09:03 GMT

Please post a new question showing the original data and what is indexed.

Re: UF is not forwarding the JSON data to indexers

newsplunker1 — Tue, 28 Apr 2020 15:29:17 GMT

The file format was the issue - I also uploaded the file into splunk instance and generates the props file then copied it to where the UF is installed