topic Re: Universal forwarder parsin in Getting Data In

Universal forwarder parsin

dogushan — Thu, 03 Aug 2017 11:49:21 GMT

Hello guys
i am new at splunk and i am using splunk cloud trial
I have a log file like this, and my event so.

2017-07-31_15:46:26.625 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2017-07-31_15:46:26.813 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                          xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                          xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                          xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                          xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                          xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                          xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                          xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                          xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                          xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2017-07-31_15:46:26.920 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2017-07-31_15:46:26.922 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

i want to break events at " 2017-07-31_15:46:26.625 " .

My props.conf file

[testLinux]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
LINE_BREAKER = .*_[0-9]*:[0-9]*:[0-9]*.[0-9]*
TRUNCATE = 10000
NO_BINARY_CHECK = 1

i want to see events like this

    event1 : 2017-07-31_15:46:26.625  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

    event2 : 2017-07-31_15:46:26.813 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx



 event3 : 2017-07-31_15:46:26.920 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

 event4 : 2017-07-31_15:46:26.922 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Anyone help me ? sorry for my bad english 🙂

Re: Universal forwarder parsin

gcusello — Tue, 29 Sep 2020 15:12:03 GMT

Hi
you used a wrong TIME_FORMAT , you have to use %Y-%m-%d_%H:%M:%S.%3N
in addition change MAX_TIMESTAMP_LOOKAHEAD = 23.

When you say "i want to break events at " 2017-07-31_15:46:26.625 " are you meaning that you don't want to index events but only timestamp or that every timestamp is the start of a new event?
if the first use TRUNCATE = 24.
if the second one, it's alredy OK.
Bye.
Giuseppe

Re: Universal forwarder parsin

dogushan — Thu, 03 Aug 2017 12:57:51 GMT

still same 😕

Re: Universal forwarder parsin

dogushan — Thu, 03 Aug 2017 13:35:06 GMT

there are no change in my logs 😕

Re: Universal forwarder parsin

gcusello — Thu, 03 Aug 2017 14:54:15 GMT

strange thing because using your TIME_FORMAT you should have a wrong timestamp!
probably I understood that you didn't reach to index logs.
what is the difference you're speaking? string "event1 : " before timestamp?
Splunk takes log as they are, you can modify them using regexes at index time, but it isn't a good idea.
Could you share more information abut your need?
Bye.
Giuseppe

Re: Universal forwarder parsin

alemarzu — Thu, 03 Aug 2017 15:08:08 GMT

Hi there, try removing the LINE_BREAKER and use this.

SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = (\d{4}-\d{2}-\d{2})

Re: Universal forwarder parsin

dogushan — Thu, 03 Aug 2017 19:00:27 GMT

i just want break events in miliseconds. for example , i have 6 logs in 1 seceonds and 5 logs in another seconds , but splunk putting them together and shows me 2 events. but i want to see 11 events.

Re: Universal forwarder parsin

dogushan — Thu, 03 Aug 2017 19:08:13 GMT

i did not create any index. is this a problem ? 😄

Re: Universal forwarder parsin

dogushan — Thu, 03 Aug 2017 19:17:11 GMT

on command line
./splunk add index test
The object "index" is not supported on this installation.

but i can create new index in web

Re: Universal forwarder parsin

dogushan — Thu, 03 Aug 2017 19:19:12 GMT

still same 😕
i try many changes in props.conf file , there is no changes in my logs

Re: Universal forwarder parsin

cpetterborg — Thu, 03 Aug 2017 23:20:39 GMT

Try:

[testLinux]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=^20\d\d-\d\d-\d\d
TIME_FORMAT=%Y-%m-%d_%H:%M:%S.%N
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=30

You can add the TRUNCATE line if you like.

Re: Universal forwarder parsin

dogushan — Fri, 04 Aug 2017 05:59:38 GMT

no changes 😕 i m traying many things but there is no changes in my logs.

Re: Universal forwarder parsin

dogushan — Fri, 04 Aug 2017 06:13:14 GMT

my inputs.conf file :

[monitor:///var/log/test.log]
sourcetype=testLinux


[monitor:///var/log/test3.log]
sourcetype=testLinux

Re: Universal forwarder parsin

gcusello — Fri, 04 Aug 2017 07:13:57 GMT

I always put index in my inputs.conf configurations, but this isn't your problem.

SHOULD_LINEMERGE should be at true and not to false.
I'd try to not use TIME_PREFIX = ^ and leave Splunk to understand wher an event starts.

The best way to proceed is to download an example of your logs and follow the web guided Add Data procedure [Settings -- Add data].
in this way you can immediately test you configuration.

Bye.
Giuseppe

Re: Universal forwarder parsin

dogushan — Fri, 04 Aug 2017 07:46:09 GMT

there was a forwarder yesterday 😕 but now
""You currently don't have any forwarders installed. If you've recently installed a new forwarder, click the refresh button below to reload page.""

i dont have outputs.conf at splunk_home/etc/system/local/ directory. is this a problem ? 😄

./splunk list forward-server
Active forwards:
input-prd-p-xxxxxxxxxxxxxxxxxxxx

Re: Universal forwarder parsin

gcusello — Fri, 04 Aug 2017 08:05:46 GMT

Hi dogushan,
they are two different problems.

About the original problem try the last procedure (web Add data).

About the second one: you must have an outputs.conf in your forwarder!
it could be in an app or in $SPLUNK_HOME/etc/system/local.
You can find it using /opt/splunkforwarder/bin/splunk cmd btool outputs list --debug;

with this command you have all the outputs.conf configurations and positions.

Bye.
Giuseppe

Re: Universal forwarder parsin

dogushan — Fri, 04 Aug 2017 08:34:40 GMT

web Add data -->> uploads -->> or forward ?

Re: Universal forwarder parsin

gcusello — Fri, 04 Aug 2017 09:41:05 GMT

Upload
Bye.
Giuseppe

Re: Universal forwarder parsin

alemarzu — Fri, 04 Aug 2017 12:50:56 GMT

Remember that parsin changes will only be apply to new events.