About alfredoh14

alfredoh14 · ‎06-11-2024

I am working on a linux server, does anyone have a few spl suggestions? which logs to look for? what text to look for in thelogs? that would be great help. thanks for the responses but i need somethin more specific, like what logs/text to look for. i have the linux app installed and the os index is basically indexing most of the important log files in /ar/log, but do not know what to look for or which log to specifically target. help would be apreciated.

alfredoh14 · ‎06-07-2024

Hello, I need to create a simple alert that would satisfy the below DOD STIG: SPLK-CL-000320 - Splunk Enterprise must be configured to notify the System Administrator (SA) and Information System Security Officer (ISSO), at a minimum, when an attack is detected on multiple devices and hosts within its scope of coverage. We do not have a budget to buy the paid splunk security app. But we haveSPLUNK enterprise 9 installed. Moreover, we are inside an intranet so attacks, if any, would be minimal. therefore, I would like to get any ideas of what would be considered an attack? for example I have the below ideas myself: 1. user logs in but is denied access for whatever reason. 2. user attempts to open a file he/she does not have rights to. I am experienced on splunk more than linux security so any help would be apreciated.

alfredoh14 · ‎06-07-2024

hello, I have 2 files that contains the path of the root Certificate Authority that issued my server certificate. Not sure if the above is logically correct but i do have 2 files that have the paths to the certificate authority, and I must use both of them somehow/someway in web.conf. my web.conf settings are similar to the below: ### START SPLUNK WEB USING HTTPS:8443 ### enableSplunkWebSSL = 1 httpport = 8443 ### SSL CERTIFICATE FILES ### privKeyPath = $SPLUNK_HOME\etc\auth\DOD.web.certificates\privkey.pem serverCert = $SPLUNK_HOME\etc\auth\DOD.web.certificates\cert.pem sslRootCAPath = However, for "sslRootCAPath =", i need to add both of the files i have. However, the splunk documentation does not specify how I would be able to add these files, or if I can add them with ",", or if there is another property that would allow additional files with certificate authority paths. the files and sample of their content are below. Basically they are made-up of "noise" except for the header and footer section of the file. So not sure how I would combine them into a single file if that is what splunk would like. ca-200.pem -----BEGIN CERTIFICATE----- MIIEjzCCA3egAwIBAgICAwMwDQYJKoZIhvcNAQELBQAwWzELMAkGA1UEBhMCVVMxGDAWBgNVBAoT -----END CERTIFICATE----- MyCompanyRootCA3.pem -----BEGIN CERTIFICATE----- MIIDczCCAlugAwIBAgIBATANBgkqhkiG9w0BAQsFADBbMQswCQYDVQQGEwJVUzEY -----END CERTIFICATE---- any help is apreciated.

alfredoh14 · ‎06-04-2024

Hello, I am getting the below error when i attempt to execute the process of creating a secret storage in /opt/splunk/bin [root@ba-log bin]# splunk rdrand-gen -v -n 1M -m reseed_delay -o /opt/splunk/rdrand.bin Passphrase that unlocks secret storage: /usr/bin/env: ‘python’: No such file or directory /opt/splunk/etc/system/bin/gnome_keyring.py: process returned non-zero status: status=done, exit=127, stime_sec=0.005357, max_rss_kb=13716, vm_minor=216, sched_vol=1, sched_invol=7 Error accessing secret storage. permission of file: -rwxr-xr-x. 1 root root 42392 Jan 6 2023 /usr/bin/env I verified with splunk envvers and the path to /usr/bin is in there. I run the splunk command as root user, so why isn't splunk able to read the /usr/bin/env file?

alfredoh14 · ‎05-15-2024

Thank you that is something that i will use if I cannot find anything that would actually do what i need. the issue is that the lookup file column is one word while the sql field would be many characters. the example I gave provided a structure wherein you could use rex however, in my real life data, there is no structure, for example, : scbt_owner could be found as "scbt" or "scbt_owner" or " as scbt" or "where scbt" if it were those 4 examples i gave, then yes i would be able to use rex but they might be different text in the SQL column. basically, the issue is that i would like to use the lookup file lk_wlc_app_short , to do a "in" the sql field. so i would use the lookup file as a base, and if any of the text in the lookup file match the sql field, i would flag them as a match and I would be able to get the final table output I want. I am not sure if splunk can do this, I know it can do a match if both the sql field and the lk_wlc_app_short field are the same (as you gave me in your example), but can Splunk be able to determine which rows of the lookup file match the SQL field without having to parse with rex since i know the sql field text would be random?

alfredoh14 · ‎05-09-2024

hello I need to determine the app name based on a lookup table for the SPL search below. the SPL search results has a field, called SQL, which has the sql syntax which contains one of the keywords in a field of the lookup table. I am not sure if join, union, inputlookup, lookup and/or combination of where command will solve this puzzle. Any help is apreciated. the lookup file name is: lookup_weblogic_app.csv the lookup file sample values are: lk_wlc_app_short lk_wlc_app_name ART Attendance Roster Tool Building_Mailer Building Mailer SCBT Service Center Billing Tool SPL search results: SQL ''' as "FIELD",''Missing Value'' AS "ERROR" from scbt_owner.SCBT_LOAD_CLOB_DATA_WORK ''' as "something ",''Missing Value'' AS "ERROR" from ART_owner.ART_LOAD_CLOB_DATA_WORK from Building_Mailer_owner.Building_Mailer_ SPL final outcome desire: lk_wlc_app_short SQL scbt ''' as "FIELD",''Missing Value'' AS "ERROR" from scbt_owner.SCBT_LOAD_CLOB_DATA_WORK ATR ''' as "something ",''Missing Value'' AS "ERROR" from ART_owner.ART_LOAD_CLOB_DATA_WORK Building_Mailer from Building_Mailer_owner.Building_Mailer_

alfredoh14 · ‎04-13-2024

hello, We upgraded our red hat 7 to 9 this past monday. and splunk stopped sending emails. We were inexperience and unprepare for this so we upgraded our splunk enterprise from 9.1 to 9.13 to see if this would fix it. It did not. then we upgraded to 9.2, that did not fix it. I started adding debug mode to everything and found that splunk would send the emails to postfix and the postfix logs would state the emails were send. however, after looking at it closer, I notice the from field of the splunk sendemail generated emails had the from field like: splunk@prod not splunk@prod.mydomain.com (as they used to before we upgraded to redhat 9 When we use mailx, the fron field from field is constructed correctly such as: splunk@prod.domain.com extra python debugging does not show the from field but only the user and the domain: from': 'splunk', 'hostname': 'prod.mydomain.com', My stanza in /opt/splunk/etc/system/local/alert_action.conf: [email] hostname = prod.mydomain.com Does anyone know how to fix this? Is there a setting in splunk that would make sure the email from field is constructed correctly. It is funny that if you add an incorrect "to" address splunk whines but if splunk create a incorrect to field address in sendemail, it is fine and, just send it to postfix and let it handle it, lol dandy 🙂

alfredoh14 · ‎04-13-2024

that was awsome help, it really just solidify the assumption that I had which was that splunk was sending the emails to the postfix server and they were getting dropped by it. My next step was to add debugging on postfix, and i found out that the "to" field was splunk@prod not splunk@prod.mydomain.com the server name (hostname) on linux is prod but when i access splunk web UI, it is https://prod.mydomain.com:8080/en-US When I added the python debugging it only showed that the user who was sending it was splunk and the domain was (based on debug logs of python sendemail): from': 'splunk', 'hostname': 'prod.mydomain.com', this did not show me that the sendemail.py command would construct the from field to only splunk@prod (I had to look at the postfix logs for that info). My stanza in /opt/splunk/etc/system/local/alert_action.conf: [email] hostname = prod.mydomain.com again, we can send emails from cli using mailx but splunk cannot using sendemail.py because it is not cnstructing the from field correctly so although postfix sents it, the smtp server which receives it drops it. so, do you have any idea of where I have to set a setting so that the from field is constructed correctly by the sendemail.py script?

alfredoh14 · ‎04-13-2024

Hello, I am trying to troubleshoot sendemail.py since after an upgrate to red hat 9 our splunk stopped sending emails. I understand the command to use the splunk python interpreter in the cli is: splunk cmd python /opt/splunk/etc/apps/search/bin/sendemail.py however, how do i combine the above with the below _internal search results so i can see what the interpreter would provide as feedback (such as errors). _raw results of a sendemail: subject="old: : $: server-prod - AlertLog_Check - 4 Log(s) ", encoded_subject="old: : $: server-prod - AlertLog_Check - 4 Log(s) ", results_link="https://MyWebsite:8080/app/search/@go?sid=scheduler__nobody__search__RMD50fd7c7e5334fc616_at_1712993040_1213", recipients="['sysadmin@MyWebsite.com']", server="localhost" any examples would e greatly apreciated, thanks, A totally blind Splunker with a mission 🙂

alfredoh14 · ‎09-23-2022

Hello, I have a odd issue which seems to have been resolved but I would like to know the root cause of this issue. I inherited a splunk configuration with one of the stanza entries in inputs.conf being: [monitor:///var/log/messages*] sourcetype=syslog index = os disabled = 0 When I perform a ls -l on /var/log/messages* I get the below: -rw-------. 1 root root 7520499 Sep 23 07:15 messages -rw-------. 1 root root 4795535 Aug 28 01:45 messages-20220828 -rw-------. 1 root root 6636499 Sep 4 01:42 messages-20220904 ... When I do a spl search on any of the possible sources, since the stanza uses "*", I get no results except for the source=messages. I do not get results for the source=messages-20220828 (even if I extend the earliest=-365d). When the rsyslog executed and rotated the messages log file this past week, at about 2 am on saturday, splunk stopped indexing the messages log file. the messages log file kept being populated by linux so that side seems to be working as expected. the last log entry splunk recorded was: _time = 2022-09-18 01:46:40 _raw = Sep 18 01:46:40 ba-dev-web rsyslogd: [origin software="rsyslogd" swVersion="8.24.0-57.el7_9.3" x-pid="1899" x-info="http://www.rsyslog.com"] rsyslogd was HUPed I restarted the splunkforwarder on the server with the issue and this fix the issue and splunk started indexing the messages log entries again. To attempt to create a permanent solution to this issue because restarting the forwarder manually is not a adequate solution for this issue I created the below stanza: [monitor:///var/log/messages] index = test disabled = 0 I do not believe I need the "*" because 1) messages* sources are not being indexed by splunk, so why use "*". (only source=messages). 2) we do not need to index messages backup log files. When I came to work today, 18 hours after the "fix" (restart of splunk forwarder), my stanza is still working and indexing log entries as expected but the previous one: [monitor:///var/log/messages*] does not index log entries any more. I used the working one and determine that the last entries before splunk stopped indexing were: first column is _time and next column is _raw 2022-09-22 14:03:38 Sep 22 14:03:38 ba-prod-web audisp-remote: queue is full - dropping event 2022-09-22 14:03:38 Sep 22 14:03:38 ba-prod-web systemd: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'. 2022-09-22 14:03:38 Sep 22 14:03:38 ba-prod-web systemd: Stopping Systemd service file for Splunk, generated by 'splunk enable boot-start'... 2022-09-22 14:03:38 Sep 22 14:03:38 ba-prod-web splunk: Dying on signal #15 (si_code=0), sent by PID 1 (UID 0) 2022-09-22 14:03:38 Sep 22 14:03:38 ba-qa-web audisp-remote: queue is full - dropping event 2022-09-22 14:03:37 Sep 22 14:03:37 ba-qa-web audisp-remote: queue is full - dropping event 2022-09-22 14:03:36 Sep 22 14:03:36 ba-qa-web audisp-remote: queue is full - dropping event the last entry for the stanza that stopped working was: 2022-09-22 14:03:37 Sep 22 14:03:37 ba-qa-web audisp-remote: queue is full - dropping event all the other monitor an dscripted inputs are working on that server except for the one above. the version of the forwarder is 7.2.3. I am running other forwarders with this version that are indexing messages log entries and they are working as expected. the stanza I used was a copy and paste from the Splunk_TA_nix add-on (except I removed the other log files and just used messages), so IMO this would be the bbest practices". I have a few questions: 1. why might be the reason why the stanza with "*' not work anymore while the one without it works? 2. Am I correct to believe that we do not need the stanza with "*", what are the consequences that I might not be aware of not using a stanza with "*"? 3. why would uid 1 (root) kill splunk (believe this is the reason why splunk stopped indexing messages log files again the 2nd time)? 4. any insights to understand this issue would be greatly apreciated. As far as i know right now, using my stanza should be good practice if we do not need the backup messages log files but I am concern I am missing something.

alfredoh14 · ‎08-05-2022

How would I be able to: use those results with an append or join to filter out the alert timeframes. tis is mainly what I want to be able to iterate through the start and end _time/time (similar to what you suggested). but I do not know how I would be able to filter by these results once I have them. for example, if I do not know how many start of maintenance and end of maintenance I have in the search results, then how will i be able to use append or join? I will not know because the query will be in a dashboard and the user might select a time frame that contains several starts/ends of maintenance or it might have none. any help is apreciated as I am breaking my head over this.

alfredoh14 · ‎08-05-2022

Hello, I have a log file that admins can write when they start or stop their server maintenance. This is then jued to silence email alerts so admins do not get email alerts when they are doing server maintenance. When the admin will start server maintenance they will write "start of maintenance...." into a specific log file (the source). When the admin will stop server maintenance they will then write "sen of maintenance...", to that same file. However, since the email alerts reset themselves after a period (4 hours ) after splunk read the "start of maintenance..." some admins will "forget" to write the "stop of maintenance..." to this file. task: I need to have an "start of maintenance..." and corresponding "end of maintenance..." entry. if I only have a "start of maintenance..." then I must use SPL to insert an event that has "end of maintenance..." and that the _time (or another field that is time-related) has the time of the "start of maintenance..." + 4 hours. So for example, if "start of maintenance..." _time is 2022/08/05 16:00:00 then I must create a event that has _time (or a time field)) of 2022/08/05 20:00:00. If there is a corresponding "end of maintenance...." within 4 hours of having a "start of maintenance..." then I should do nothing. My ultimate goal is to create a dashboard with results filtered by "start of maintenance.." _time and "end of maintenance..." _time, but in order to do this I first have to make sure I have both "start of maintenance..." and "end of maintenance..." _time/Time values.

alfredoh14 · ‎08-05-2022

Hello, this is the first time i post here but I have learn alot from this website by just using google search. Situation: At work server admins ask if I could "silence" splunk email alerts when they were doing maintenance so that they do not get emails of errors during server maintenance. I was able to do this because I created a maintenance.log in the /var/log/ folder that splunk keeps track of. if the admins write: "start of maintenance..." then any alert that monitors this logs will stop sending emails. when the admins write: "end of maintenance..." then splunk knows it can start sending emails since maintenance period is completed. this was useful to silence apache access log alerts that occurred during maintenance, meaning the admins did not get alerts that the apache access log wrote while admins were during maintenance as denoted by the _time of "stat of maintenance..." and _time of "end of maintenance...." Task: I have to show search results that do not contain any results that were reported during a maintenance period in a dashboard. this means that any search results between the _time of "start of maintenance...." and _time of "end of maintenance..." should not be included in the results. Moreover, there might be times when maintenance happened several times, for example, if maintnenace was done twice in one day or if they are searching for a time period of say, 1 month, and it shows there were 3 "starts of maintneance" and 3 corresponding "end of maintenance..." entries. Action: I have writen SPL that will get all the results: earliest=-1d (host="Server-web" source="/var/log/httpd24/error_log") OR (host="Server-Web" index=bizapps source=/var/log/bizapps_maintenance.log) I am not sure if splunk SPL can pull this off but am confident someone can help me out. If you need more info, let me know.

alfredoh14 · ‎01-12-2021

when i type in the command line (cmd not powershell): splunk search "*" -maxout 0 | find /c /v "" I get the return of about 195k records. however when i filter by one of the sourcetype, from one of the splunk free tutorials, by typeing: splunk search "*" 'sourcetype=e' -maxout 0 | find /c /v "" I still get the same result of 195k records. the "| find /c /v "" " should return only the line count. so if i filter by the sourcetype it should return around 165k records which are the number of records associated with that source type currently in my splunk db. can someone help me correct my syntax, i tried google searches but none were able to give me an example, which I think is due to the fact I do not know how the syntax should work in the first place. I am using the windows cmd, and I plan to use the windows cmd, not the website interface, thanks for understanding.

Posts	15
Solutions	0
Karma Given	1
Karma Received	1
Member Since	‎01-12-2021

Online Status	Offline
Date Last Visited	‎06-12-2024 10:20 AM

How create an alert that will trigger when server ...

How add multiple certificate authority path files ...

splunk not able to access python file /usr/bin/env...

Combining static single-word from lookup table wit...

sendemail.py on splunk 9.21 and redhat9 not creati...

How use splunk python interpreter to troubleshoot

Why is inputs.conf not indexing /var/log/messages?

How to artificially insert event based on an exist...

How filter results by custom start and end string ...

help fixing cli search with sourcetype specified

Re: How create an alert that will trigger when ser...