About craigkleen

craigkleen · ‎02-01-2021

That was the ticket. Under ${SPLUNK_HOME}/etc/system, the limits.conf was the same. But, on the UF, under ${SPLUNK_HOME}/etc/apps/SplunkUniversalForwarder/ the default was overridden to 256K. So, I made a local directory and updated there. So, thanks for the pointer!

craigkleen · ‎01-29-2021

From a machine standpoint, the HF and UF are the same. Both are virtual servers that are clones of each other. The only difference is the Linux version (going from RHEL6 to RHEL8). If I splunk: host=HF index=_internal eps=* group=per_source_thruput source=panfwlog The max EPS I get is right around 1,200. A similar search with host=UF during the time the firewall is sending to this new server, is showing me EPS under 4? Super weird. The data is getting written to disk, and when I switch the firewall back to the old server, the UF eventually does catch up, but it's not reading like the HF does.

craigkleen · ‎01-29-2021

On both, that's the usual process. Native "rsyslog" daemon writing to a file, and UF then reading that.

craigkleen · ‎01-29-2021

Currently, my firewall logs (PaloAlto) are sent via syslog to a virtual Linux machine. On that machine, I run a full version of Splunk (Heavy Forwarder 8.x) that sends into separate indexers. I was planning to migrate the syslog data to new Linux servers and use Universal Forwarder instead, but running into what looks like some serious performance issues. The UF will send a big chunk of data to start, but then the index stops receiving from the UF. I tried the post at https://community.splunk.com/t5/Getting-Data-In/Universal-Forwarder-ParsingQueue-KB-Size/td-p/50410 to increase the size of the parsingqueue, but that didn't help. I'm not quite sure what to look at next. Maybe the stream is too much for UF to handle? I haven't found anything definitive on that subject.

craigkleen · ‎10-01-2020

I ended up just resolving this "well enough" with some field extractions. I made two field extractions to get the "timestamp_1900" and "timestamp_1900fract", then made a calculated field for "timestamp" with this sourcetype. The calculation looks like: strftime(timestamp_1900 - 2208988800 + round(timestamp_1900fract / 4294967295, 3), "%m-%d-%Y %H:%M:%S.%3N")

craigkleen · ‎09-22-2020

Hi, I'm trying to get data in from a file where data is in the following format (anonymized): {" seq " :55619 ," ntp_time " : [ 3809782725 , 1802580594 ]," reporting_id " : {" tugid " : "server "," ep_type " : " sip "," side " : " SS "," mac " : "aa:bb:cc:dd:ee:ff "," user " : "username "," dn " : " 43128 "}," stream_id " : {" sip_callid " : " hexstring "," local_uri " : " sips:emailstring:5061 "," remote_uri " : " sips:emailstring:5061 ; transport=tls "," ep_stream_id " :5053 }," event " : " rtcp_tx "," rtcp_block " : {" addr_local " : "ipaddr :24794 "," addr_remote " : "ipaddr 5036 "," cname " : "emailstring "," snd_ssrc " :680275594 ," recv_ssrc " :3888553685 ," snd_pktcnt " :206158433963 ," snd_bcnt " :4121132523374324448 ," rx_loss_total " :139753940844544 ," rx_loss_fract " :0 ," rx_jtr " :-139758235811834 ," rtt " :139753940844544 }," rtp_stats " : {" observed_pt " :0 ," observed_codec " : " RTP_CODEC_G711_U "}} So, a nice JSON. But, that pair of integers in ntp_time{} are seconds since 1/1/1900 and a fractional second, not 1/1/1970. I'm really, really hoping I don't have to write a second script that writes out the correct timestamp. On my indexers, for the sourcetype I've defined for this, I've the following: [baddate] REGEX = ntp_time\":\[(?<baddate>\d+) INGEST_EVAL = gooddate = baddate - 2208988800 I also have props.conf calling the transform, and fields.conf setting "INDEXED=True" for baddate. But I don't get the field in search yet. Would this even work though? Does anyone have any other strategies I can try? I don't really care about the fractional second, but would work it in if I can get something to work.

craigkleen · ‎05-29-2020

Yeah, answering my own post. The issue was that we used the wrong "endpoint". We chose "USGovGCCHigh" instead of "Worldwide", because Worldwide kept erroring out. Lesson-learned that the Worldwide looked different to our Palo Alto Firewall as far as the "app ID" went, so the PAN blocked it. Once the firewall was open, and we switched the Endpoint to Worldwide, it worked perfectly.

craigkleen · ‎05-28-2020

Hello, We've followed the instructions for version 2.0.2 of splunk_ta_o365, running on a Splunk 8.0.2 HF. But in the logs we're getting: 2020-05-28 07:45:25,328 level=ERROR pid=2827626 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=utils.py:wrapper:67 | datainput=b'O365_Management_Activity' start_time=1590677124 | message="Data input was interrupted by an unhandled exception." Traceback (most recent call last): File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/utils.py", line 65, in wrapper return func(*args, **kwargs) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 102, in run executor.run(adapter) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunksdc/batch.py", line 47, in run for jobs in delegate.discover(): File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/management_activity.py", line 127, in discover subscription.start(session) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 160, in start response = self.perform(session, 'POST', '/subscriptions/start', params) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 169, in _perform return self._request(session, method, url, kwargs) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 181, in _request raise O365PortalError(response) File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 26, in __init_ message = str(response.status_code) + ':' + payload TypeError: can only concatenate str (not "bytes") to str We've checked and rechecked our Azure permissions, which look like: Hoping someone has a lead on what's going wrong. At least one poster with a similar error said that "Grant Permissions" was missed on the Azure side, but as far as we can tell it looks OK there. Thanks

craigkleen · ‎05-14-2019

Yeah, that gets me to the same place. But gets a little unwieldy in my use case. To expand my original data with an example, it's like: eventNum,Data,sequence_number eventOne,origData,6675670249450679850 eventOne,origData,6675670249450679847 eventOne,origData,6675670249450679801 eventTwo,extradata,6675670249450679800 eventOne,origData,6675670249450679653 eventTwo,extradata,6675670249450679652 eventOne,origData,6675670249450679645 eventOne,origData,6675670249450679643 eventTwo,extradata,6675670249450679642 eventOne,origData,6675670249450679523 eventTwo,extradata,6675670249450679522 Where, I'm trying to add the fields in "extraData" to the fields in "origData", when the only thing I have coupling this data is this sequence_number that's off by one. So, it seems shorter use a single statement like: | eval commonSeq=if(eventNum="eventOne", tonumber(substr(tostring(sequence_number), -6)) - 1, tonumber(substr(tostring(sequence_number), -6))) | transaction commonSeq Was really hoping that there was more of a function based approach I'm missing, rather than a rex-based one.

craigkleen · ‎05-14-2019

I have a log file with a very large number in it, it's a sequence number, and doesn't seem to have anything to do with time, they're all unique. They look like: sequence_number 6675670249450679850 6675670249450679847 6675670249450679801 6675670249450679800 6675670249450679653 6675670249450679652 6675670249450679645 6675670249450679643 6675670249450679642 6675670249450679523 6675670249450679522 There's a relationship between logs when the numbers differ by 1, but the logs contain different information. I'm trying to do a transaction to group these lines, but to get "sequence_number - 1", Splunk seems to round horribly. I only really need to compare the least significant digits, so I have a workaround to create a field based on data in the higher number with: | eval subSeq=tonumber(substr(tostring(sequence_number), -6)), firstSeq=subSeq - 1 And something similar to the other log type. But, is there a better way?

craigkleen · ‎01-18-2019

Sorry for the delay. Splunk's response wasn't great. I basically had to write a "restart" into my log rotation script. It might be fixed with the 7.x series now, but I haven't had the time to go back and check.

craigkleen · ‎06-27-2017

I have, and submitted a diag. Waiting to hear back.

craigkleen · ‎06-22-2017

Will do. Thx

craigkleen · ‎06-22-2017

Recently updating from 6.5.3 to 6.6.1, I started running into a situation where at least one of my Heavy Forwarders would intermittently stop sending data. The Heavy Forwarder is on a RedHat 6.x system, virtualized, with the latest patches. After reading some answers, I found the command: ./bin/splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus Which for at least one of the files output the following: <s:key name="/var/log/aruba/controller1.log"> <s:dict> <s:key name="file position">22440152</s:key> <s:key name="file size">20726882</s:key> <s:key name="parent">/var/log/aruba/*.log</s:key> <s:key name="percent">108.27</s:key> <s:key name="type">open file</s:key> </s:dict> </s:key> I figure it being over 100% read is odd. When it got into this state, I checked the metrics in _internal for this monitored file and they had stopped recording. With RedHat, I use 'logrotate', running on an hourly schedule. The change from the file being read to stopping seems to have occurred on the hour. It would stop working for 1-3 hours, then suddenly start reading again all by itself. It's worked before with 6.5.3, but not with 6.6.1. I downgraded last night to 6.5.3 on this one Heavy Forwarder, and it's back to working again, so I've eliminated logrotate and other OS patches. Can anyone advise on some other things to try with 6.6.1 to see if there's some new setting I should be using?

craigkleen · ‎10-18-2016

Unfortunately, that middle one doesn't work well. Because the regex matches colons on either side, if the mac address is entirely single digits, it will match the first :x: but the next one skips a field because it would look like "x:" to SED. What I've had to do so far is: SEDCMD-macfix1 = s/(.*Option 82.*)$([a-f0-9]:)/\1(0\2/g SEDCMD-macfix2 = s/(.*Option 82.*\([a-f0-9]{2}):([a-f0-9]:)/\1:0\2/g SEDCMD-macfix3 = s/(.*Option 82.*\([a-f0-9]{2}:[a-f0-9]{2}):([a-f0-9]:)/\1:0\2/g SEDCMD-macfix4 = s/(.*Option 82.*\([a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}):([a-f0-9]:)/\1:0\2/g SEDCMD-macfix5 = s/(.*Option 82.*\([a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}):([a-f0-9]:)/\1:0\2/g SEDCMD-macfix6 = s/(.*Option 82.*\([a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}):([a-f0-9]$)/\1:0\2/g I was just hoping someone out there would know some more tricks to sed to shorten that.

craigkleen · ‎10-17-2016

So, some companies in their infinite wisdom strip leading zeroes from the bytes WITHIN MAC addresses, so we end up with logs that make it a little hard to search consistently. I suppose a little mental arithmetic is easy enough, but I'd really love to find a way to fix this problem at index time, even though I know it will marginally increase my license. Here's an example of one log: Oct 17 10:43:28 10.0.0.1 dhcpd[13431]: Option 82: received a REQUEST DHCP packet from relay-agent 10.0.1.1 with a circuit-id of "31:31:31:2d:31:31:31:2d:41:31" and remote-id of "6c:6d:6e:6f:70:71:72:73:74:75:76:77" for 10.0.1.100 (0:5:a6:b:ea:80) lease time is undefined seconds. I'd like to be able to replace ALL single hex digits that shows up in the bolded string to be a compliant xx:xx:xx:xx:xx:xx style MAC address, inserting leading zeroes along the way, without having to have SEDCMD run multiple times in a row.

craigkleen · ‎07-08-2016

Good grief. So, I went one step further and installed the Splunk_TA_paloalto app directly onto the forwarder. After, of course, I remembered that I'm not using UniversalForwarder on my syslog server (when it's almost a year between certain kinds of updates, you forget things). It's a HeavyForwarder. So, once that was added, I'm now getting logs tagged right. Silly. Thanks for your help~

craigkleen · ‎07-08-2016

Okay, going directly to the indexer seems to have worked. The format looks a little different: Jul 8 14:34:21 10.0.0.1 1 2016-07-08T14:34:21-07:00 pan-1 - - - - 1,2016/07/08 14:34:21,010108001182,CONFIG,0,0,2016/07/08 14:34:21,10.0.0.2,,delete,panadmin,Web,Succeeded, deviceconfig system ntp-servers secondary-ntp-server,1013,0x0,0,0,0,0,,pan-1 There are 4 different timestamps in that message! Seriously, what a waste of splunk license space. Thanks. It's not really my ideal solution, but I guess it'll have to do.

craigkleen · ‎07-08-2016

Maybe the log is different? Our PAN-OS is 7.0.6.

craigkleen · ‎07-08-2016

On the Universal forwarder, the ~/etc/outputs.conf file looks like: [tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] disabled = 0 server = splunkindexer:9997 [tcpout-server://splunkindexer:9997]

Posts	43
Solutions	3
Karma Given	2
Karma Received	9
Member Since	‎10-14-2014

Online Status	Offline
Date Last Visited	‎04-11-2022 07:13 PM

UF versus HF processing

Timestamp in 1/1/1900 format

Office 365 Add-On connection errors - unhandled ex...

Very large number math

Splunk 6.6.1 stops monitoring files

SEDCMD for MAC address that are missing leading ze...

Palo Alto Networks App for Splunk: Reading Palo Al...

Sideview button action to refresh page

Why are we getting a deployment checksum failure o...

How to call an external python script to populate ...

Re: UF versus HF processing

Re: UF versus HF processing

Re: UF versus HF processing

UF versus HF processing

Re: Timestamp in 1/1/1900 format

Timestamp in 1/1/1900 format

Re: Office 365 Add-On connection errors - unhandle...

Office 365 Add-On connection errors - unhandled ex...

Re: Very large number math

Very large number math

Re: Splunk 6.6.1 stops monitoring files

Re: Splunk 6.6.1 stops monitoring files

Re: Splunk 6.6.1 stops monitoring files

Splunk 6.6.1 stops monitoring files

Re: SEDCMD for MAC address that are missing leadin...

SEDCMD for MAC address that are missing leading ze...

Re: Palo Alto Networks App for Splunk: Reading Pal...

Re: Palo Alto Networks App for Splunk: Reading Pal...

Re: Palo Alto Networks App for Splunk: Reading Pal...

Re: Palo Alto Networks App for Splunk: Reading Pal...