About djreschke

djreschke · ‎06-27-2022

@richgalloway limits.conf in the local folder limits.conf [thruput] maxKBps = 00

djreschke · ‎06-27-2022

@richgalloway I agree, it is not a lot of files. Current ulimit settings # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514862 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 102400 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 514862 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

djreschke · ‎06-23-2022

@richgalloway The time it stops is unknown at this time, I get alerted when logs stop being collected on a daily basis. Its a syslog server so a lot of files, 20 - 40 files depending on activity. ulimit is set at 0.

djreschke · ‎06-23-2022

Has anyone run into an issue where a Splunk HF, is not monioring files being written to it. This HF is also a syslog server so files are being written to it and the monirotied inputs are on the server. The file ingestion happens after a restart. Any pointers?

djreschke · ‎11-24-2021

@PickleRick Thank you for doing some additional testing and validation. I'll need to kick this up the flag pole to get the fix implemented. Kind of sucks that there are more then one way that Splunk reads time. I would think you would want to go off index time for everything and then change the time zone based off the users location. i know there are other variables like time zone of the log source. Thanks for all of the back and forth.

djreschke · ‎11-23-2021

@PickleRick Correct no delay in indexing. The raw log is in UTC, but because there is not timezone follower, the search head is interpreting it as EST. Indextime is the correct time of the event, so wouldn't that indicatie that the event is being indexed at the right time, and the search head is miss interrupting the based on the props.conf. So there is no why to change the historical events from what you are telling. Would a better fix be change the format in the logs from the original source. How would I accomplish this. The indexers do not have any custom props deployed from the sourcetype, unless I am missing something? Data for this log is ingested from a UF that monitors a file directory. "The proper solution (but applying only to newly ingested events) is to set a proper timezone for this source/sourcetype/host on the ingesting component (first HF or Indexer that's receiving and parsing data)."

djreschke · ‎11-23-2021

@PickleRick Here is the expanded log and the event for reference. I included indextime so you can see the difference. c_ip = 2.2.2.2 cs_method = GET cs_uri_stem = /test/test.aspx date_hour = 7 date_wday = tuesday date_zone = local host = machine index = test sc_status = 404 sc_substatus = 0 IndexTime = 11/23/2021 02:06:29 X_FORWARDED_FOR = 99.99.99.99 blnStartHere = True cs_Cookie_ = - cs_Referer_ = - cs_User_Agent_ = Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/96.0.4664.45+Safari/537.36 cs_bytes = 0 cs_username = - cs_version = HTTP/1.1 date = 2021-11-23 s_ip = 1.0.0.1 s_port = 999 s_sitename = W3SVC2 sc_bytes = 0 sc_win32_status = 0 time = 07:06:28 time_taken = 98 _time = 2021-11-23T07:06:28.000-05:00 Event 2021-11-23 07:06:28 W3SVC2 host 2.2.2.2 GET /test/test.aspx 999 - 1.0.0.1. HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/96.0.4664.45+Safari/537.36 - - test.com 404 0 0 0 0 99.99.99.99

djreschke · ‎11-22-2021

@PickleRick So is the change that needs to made on the indexing side or would a change in the log format be the best route. The Event is being indexed at the right time but since the is no time zone specified in the logs ie. 2021-11-22 08:35:55 versus 2021-11-22:08:35:55 -0500, event time is showing as 08 AM EST when it is indexed at 08 AM UTC. _time = 2021-11-22 08:38:18 IndexTime = 11/22/2021 03:38:49 index = index diff = 0 Search used: index=index | rename _indextime as IndexTime | eval diff=IndexTime-_time | convert ctime(IndexTime) as IndexTime | eval diff=if(diff < 0, "0", diff) | table _time IndexTime diff index

djreschke · ‎11-22-2021

@PickleRick What version of Splunk are you running? I am running 8.1.1.

djreschke · ‎11-19-2021

I am trying to correlate 2 different logs one is in EST and the is in UTC. The UTC logs, I have tried to specific the time zone as UTC in props and then let my user timezone preference do the conversion. Props was only updated on the search head. [soucretype] TZ = UTC This does not seem to work and i have read different post saying that updating the props file should work but not for historical events. Is there a way to do this indexed events?

djreschke · ‎11-19-2021

@PickleRick Agreed, the UI is correct, I can't post screenshots. I have open a support ticket with Splunk. Thank you for validating that there is a difference. I'll keep this post updated as I here back from Support.

djreschke · ‎11-19-2021

@PickleRick All of the time information is being parsed correctly, it is simply not so simply different on the _time field from the Web UI to the exported csv. Web UI _time is showing: 2021-11-19T11:34:02.000-05:00 CSV Export _time is showing: 2021-11-19T11:34:02.000+0000 I never noticed the difference before today but for users that don't log into Splunk, they are assuming that the CSV _time is in UTC, so they are converting this to EST which puts this event at 630 in the morning, which is not correct.

djreschke · ‎11-19-2021

So when I export the log the it what it shows as _time, but i am seeing the correct _time in the field on the Search head. Is this a know thing that it drops the timezone when you export a log.

djreschke · ‎11-19-2021

Can you please clarify some more? Its right that is showing 2021-11-19T11:34:02.000+0000 Or is this wrong from based of my timezone.

djreschke · ‎11-19-2021

My timezone is in EST, so I would imagine the it would show something like -500?

djreschke · ‎11-19-2021

_time: 2021-11-19T11:34:02.000+0000 date_hour: 11 date_mday: 19 date_wday: friday date_year: 2021 date_zone: -300 raw log snippet [19/Nov/2021:11:34:02 -0500] 2021-11-19T11:34:02.000+0000 indicates UTC. Does this indicate timezone?

djreschke · ‎10-25-2021

Needed to create a transforms to fix the issue.

djreschke · ‎10-25-2021

I have a props conf file that is not parsing data as i expected. I can see in the raw log that the IIS log has the header information in it. [sourcetype] DATETIME_CONFIG = INDEXED_EXTRACTIONS = w3c LINE_BREAKER = ([\r\n]+) MAX_TIMESTAMP_LOOKAHEAD = 32 NO_BINARY_CHECK = true SHOULD_LINEMERGE = false category = Web description = W3C Extended log format produced by the Microsoft Internet Information Services (IIS) web server detect_trailing_nulls = auto disabled = false pulldown_type = true When i manual upload the log file and assigning it to it will parse out the log with the same settings. [sourcetypeA] DATETIME_CONFIG = INDEXED_EXTRACTIONS = w3c LINE_BREAKER = ([\r\n]+) MAX_TIMESTAMP_LOOKAHEAD = 32 NO_BINARY_CHECK = true SHOULD_LINEMERGE = false category = Web description = W3C Extended log format produced by the Microsoft Internet Information Services (IIS) web server detect_trailing_nulls = auto disabled = false pulldown_type = true This props is on the searchhead were i am searching the data. The IIS logs is being capture via a UF. Has anyone run into this before?

djreschke · ‎09-15-2021

When I test the regex in both regex101 and using the rex command in the search bar and they parsed out the fields correctly. Now that i have added them to the props conf on the search head, they are capturing extra information. The Result field is the one that is mainly caputuring the SessionID which the the capture is Verified or Failed. Thank you all for your help with this. props.conf [exp_test] DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Custom pulldown_type = true CHECK_FOR_HEADER = false CHARSET = AUTO EXTRACT-SessionID = (?<=SessionID:)(?P<SessionID>.+) EXTRACT-Result = \VerificationResult:(?P<Result>.+) EXTRACT-UserName = (?<=User:)(?P<UserName>.+) EXTRACT-Response_1 = (?<=Response_1:)(?P<Response_1>.+) EXTRACT-Response_2 = (?<=Response_2:)(?P<Response_1>.+) Sample Log Time: 13-09-2021 10:08:19 VerificationResult: Failed SessionID: K3K2N2G3JPSOZNOWJFOMFPBP.pidd1v-210913090809460797217 User: LAST, FIRST 13-09-2021 10:10:18 Response_1: 1st reqest Sent! for User: LAST, FIRST 13-09-2021 10:10:19 Response_1: 1st response received! for User: LAST, FIRST Time: 13-09-2021 10:10:19 SessionID and User Mapping: SessionID: 3EV6PLCHK795Z8FQBKKYS3Z3.pidd2v-210913091018537820706 User: LAST, FIRST 13-09-2021 10:15:13 Response_1: 1st reqest Sent! for User: LAST, FIRST 13-09-2021 10:15:14 Response_1: 1st response received! for User: LAST, FIRST Time: 13-09-2021 10:15:14 SessionID and User Mapping: SessionID: GAWJ1C7ZWNAWCVTEEIWGE3LL.pidd2v-210913091513558630064 User: LAST, FIRST 13-09-2021 10:15:33 Response_1: 1st reqest Sent! for User: LAST, FIRST 13-09-2021 10:15:33 Response_1: 1st response received! for User: LAST, FIRST 13-09-2021 10:15:38 Response_1: 1st reqest Sent! for User: LAST, FIRST 13-09-2021 10:15:39 Response_1: 1st response received! for User: LAST, FIRST Time: 13-09-2021 10:15:39 SessionID and User Mapping: SessionID: 2SYZV3QHCZKYM2YTYIJLVL3E.pidd2v-210913091538460803649 User: LAST, FIRST 13-09-2021 10:15:47 Response_1: 2nd request sent! for the user verification SessionID: 2SYZV3QHCZKYM2YTYIJLVL3E.pidd2v-210913091538460803649 13-09-2021 10:15:48 Response_1: 2nd response received! for user verification SessionID: 2SYZV3QHCZKYM2YTYIJLVL3E.pidd2v-210913091538460803649 Time: 13-09-2021 10:15:48 VerificationResult: Verified SessionID: 2SYZV3QHCZKYM2YTYIJLVL3E.pidd2v-210913091538460803649 User: LAST, FIRST 13-09-2021 10:16:47 Response_1: 1st reqest Sent! for User: LAST, FIRST 13-09-2021 10:16:48 Response_1: 1st response received! for User: LAST, FIRST Time: 13-09-2021 10:16:48 SessionID and User Mapping: SessionID: D5JVVUR3AAKFURITHCI993H9.pidd2v-210913091647448944771 User: LAST, FIRST

djreschke · ‎09-13-2021

@richgalloway We are starting to see data ingested in real from the log. No changes made in either the application nor the conf file. Before posting this question we did restart the forwarder but we were not seeing the logs ingested. Next I need to work on the Regex to help parse the fields but I can open a separate question for that. Thank you for your help.

djreschke · ‎09-10-2021

@richgalloway We haven't got to prod yet and that was estimate. I imagine it will be much less. Thanks for the replies and the help thus far. More testing is scheduled for Monday at this point.

djreschke · ‎09-10-2021

@richgalloway Not that big is from i understand as under 100GB. We have restarted the UF with no success. I'll have to review the logs today to see if i can find the issue, and i will share my findings. Thank you for help thus far.

djreschke · ‎09-09-2021

The file is being monitored by a UF, The problem is after the initial log is ingested, if anything additional is written to the log it does not ingest that. The log source itself does not roll after a certain size, and i am told that file does not get that big, as the service is restart monthly for maintenance. I'm not seeing any errors in the logs, but i also might be looking for the wrong thing. - Any tips/advice here?

djreschke · ‎09-09-2021

Good morning everyone, I am trying to ingest a log that does not roll over after a new, only when the service that writes the log is restarted. We have done some testing using cRcSalt and so far that has not helped to continually monitor the file as it is written. Any advice would be appreciated. inputs.conf [monitor://E:\Tomcat 9.0\logs\tomcat9-stdout.*.log] sourcetype = test index = test blacklist = \.(gz|bz2|z|zip)$ disabled = false CRCSALT = <SOURCE> Props.conf [test] DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Custom pulldown_type = true CHECK_FOR_HEADER = false CHARSET = AUTO EXTRACT-SessionID = (?<=SessionID:)(?P<SessionID>.+) EXTRACT-Result = (?<=VerificationResult:)(?P<Result>.+) EXTRACT-UserName = (?<=User:)(?P<UserName>.+) EXTRACT-Response = (?<=Account Response:)(?P<Response>.+) EXTRACT-Second_Response = (?<=Verification_test:)(?P<Second_Response>.+)

djreschke · ‎12-18-2020

index=xyz sourcetype=hx_cef_syslog host=1.1.1.1 | convert ctime(_indextime) AS indextime | eval delay=_indextime-_time | table _time delay indextime date_zone host source sourcetype _raw

Posts	46
Solutions	2
Karma Given	12
Karma Received	1
Member Since	‎01-09-2019

Online Status	Offline
Date Last Visited	‎08-16-2022 04:20 PM

Why have monitored inputs stopped working?

Is there a way in Splunk to change the timezone or...

Why is _time indicating UTC when the event happene...

IIS Logs not Parsing as expected

I need help with my props conf to extract fields c...

Log Ingestion - Custom Log Source

Eventtime(_time) is showing 5hours prior to indext...

Splunk Admin searches being queued, but not runnin...

Splunk Returning different Stats for same time ran...

Event Time is 4 hours ahead of the actual event.

Re: Monitored Inputs stopped working.

Re: Monitored Inputs stopped working.

Re: Monitored Inputs stopped working.

Why have monitored inputs stopped working?

Re: Is there a way in Splunk to change the timezon...

Re: Is there a way in Splunk to change the timezon...

Re: Is there a way in Splunk to change the timezon...

Re: Is there a way in Splunk to change the timezon...

Re: Why is _time indicating UTC when the event hap...

Is there a way in Splunk to change the timezone or...

Re: Why is _time indicating UTC when the event hap...

Re: Why is _time indicating UTC when the event hap...

Re: Why is _time indicating UTC when the event hap...

Re: Why is _time indicating UTC when the event hap...

Re: Why is _time indicating UTC when the event hap...

Why is _time indicating UTC when the event happene...

Re: IIS Logs not Parsing as expected

IIS Logs not Parsing as expected

I need help with my props conf to extract fields c...

Re: Log Ingestion - Custom Log Source

Re: Log Ingestion - Custom Log Source

Re: Log Ingestion - Custom Log Source

Re: Log Ingestion - Custom Log Source

Log Ingestion - Custom Log Source

Re: Eventtime(_time) is showing 5hours prior to in...