All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

would you be able to share your configuration that passed the token to that initial view? I can't get that to work either.
Hi Splunk Gurus,  I’m working on a script to programmatically check if logs from a specific host are available in Splunk. For this, I’m using token-based authentication. I’ve created a role and a us... See more...
Hi Splunk Gurus,  I’m working on a script to programmatically check if logs from a specific host are available in Splunk. For this, I’m using token-based authentication. I’ve created a role and a user with the necessary permissions, and generated a token for that user. However, when I try to run the following curl command against my Splunk Cloud instance: curl -k -H "Authorization: Bearer <your_token>" \ https://<your_splunk_instance>.splunkcloud.com:443/services/server/info   I receive a 303 status code, and I’m not sure what I might be doing wrong. I’ve checked multiple forums but haven’t been able to find a clear solution. Could you please help me understand what might be causing this and how I can resolve it? Thank you in advance!
This resulted in the same source IP in multiple rows, I need it by source IP.
I had already tried that annoyingly it looked like this:   index=*proxy* SOCKS | eval destination=coalesce(dest, dest_port) | rex field=url mode=sed "s/^SOCKS:\/\/|:\d+$//g" | eval network=case(mat... See more...
I had already tried that annoyingly it looked like this:   index=*proxy* SOCKS | eval destination=coalesce(dest, dest_port) | rex field=url mode=sed "s/^SOCKS:\/\/|:\d+$//g" | eval network=case(match(src_ip,"<REDACTED>"),"user",1=1,"server") | eval Proxy_day = strftime(_time, "%d-%m-%y") | join type=left max=0 src_ip [ search index=windows_events EventID=4624 NOT src_ip="-" NOT user="*$" | stats count by IpAddress, user | rename IpAddress as src_ip | rename user as win_userid | fields - count ] | eval userid=coalesce(userid, win_userid) | join type=left max=0 userid [ search index="active_directory" | stats count by username, fullname, title, division, mail | rename username as userid ] | rename src_ip as "Source IP" | sort -Proxy_day | stats values(mail) as "Email Address" values(username) as "User ID" values(destination) as Destination values(network) as Network values(Proxy_day) as Day values(url) as URL by "Source IP" | eval "Email Address"=mvindex('Email Address',0,4), Day=mvindex(Day,0,4), URL=mvindex(URL,0,4)   It reduced the source IP's to below what was visible on a unfiltered search which meant it was removing results that I needed.
We have been having some strange performance issues with some of our dashboards and we would like some advice on how to troubleshoot these issues and fix them. Despite the underlying searches being ... See more...
We have been having some strange performance issues with some of our dashboards and we would like some advice on how to troubleshoot these issues and fix them. Despite the underlying searches being extremely fast, sometimes results will take upwards of 30 seconds to be displayed in the corresponding dashboard panels. Infrastructure and dashboard details We are running a distributed on-prem Splunk environment with one search head and a cluster of three indexers. All instances are on version 9.2.2, although we have been able to replicate these issues with a 9.4.2 search head as well. We have six core dashboards, ranging from simple and static to considerably dynamic and complex. About 95% of the searches in this app’s dashboards are metric-based and use mstats. Each individual search is quite fast, with most searches running in under 0.5s, even in the presence of joins/appends. Most of these searches have a 10s refresh time by default. Problem We have been facing a recurring issue where certain panels will sometimes not load for several seconds (10-30 seconds usually). This tends to happen in some of the more complex dashboards, particularly after drilldowns/input interactions – doing so often leads to "Waiting for data" messages displayed inside the panels. One of two things tends to happen: The underlying search jobs run successfully but the panels do not display data until the next refresh, which causes the search to re-run; panels behave as normal afterwards: The pending searches start executing but do not fetch any results for several seconds, which can lead to the same search taking variable amounts of time to execute. Here is an example of the same search taking significantly different amounts of time to run (ran just 27s apart): Whenever a search takes long to run, the component of the search that takes the longest to run, is, by far, the dispatch.stream.remote.<one_of_the_indexers> component which, to the best of our knowledge, represents the amount of time spent by the search head waiting for data streamed back from an indexer during a distributed search. We have run load tests consisting of opening our dashboards several times in different tabs simultaneously for prolonged periods of time and monitoring system metrics such as CPU, network, and memory. We were not able to detect any hardware bottlenecks, only a modest increase in the CPU usage and load average for the search head and indexers, which is expected. We have also upgraded the hardware the search head is running on (96 cores, 512 GB RAM) and despite the noticeable performance increase, the problem still occurs occasionally. We would greatly appreciate the community's assistance in helping us troubleshoot these issues.
Great question—you’re absolutely right that doing literal matches of lat/lon values from your car traces to the lookup file won’t work well, because GPS points are rarely going to land exactly on the... See more...
Great question—you’re absolutely right that doing literal matches of lat/lon values from your car traces to the lookup file won’t work well, because GPS points are rarely going to land exactly on the same coordinates, especially with only ~100 reference points describing the whole track. Instead, what you really want is for each car GPS point to find the nearest point in your lookup file and use its segment name.
Hi @NorthropGrumman  There are a number of ways you could tackle this, but generally using join is low down the list of options as it can be resource intensive. Instead you could look to bring back ... See more...
Hi @NorthropGrumman  There are a number of ways you could tackle this, but generally using join is low down the list of options as it can be resource intensive. Instead you could look to bring back all the data you need and then use stats on it. You could do this with something like index=network (etc) OR index=active_directory (etc) OR index=etc.etc.. or..you could use append, which would add the additional data to the results before then using stats to bring it all together, try the following as a starting point, it might need a little refinement as I dont have your data to test against. index=*proxy* SOCKS earliest=-90d@d latest=now | eval destination=coalesce(dest, dest_port), userid=coalesce(user, username), type="proxy" | rex field=url mode=sed "s/^SOCKS:\/\/|:\d+$//g" | eval network=case(match(src_ip,"<Redacted>"),"user",1=1,"server"), Proxy_day = strftime(_time, "%d-%m-%y") | append [ search index=windows_events EventID=4624 NOT src_ip="-" NOT user="*$" earliest=-90d@d latest=now | stats count by IpAddress, user | rename IpAddress as src_ip, user as win_userid | eval type="windows", userid=win_userid ] | append [ search index="active_directory" earliest=-90d@d latest=now | stats count by username, fullname, title, division, mail | rename username as userid | eval type="ad" ] | stats values(mail) as mail, values(fullname) as fullname, values(title) as title, values(division) as division, values(userid) as userid, values(win_userid) as win_userid, values(destination) as destination, values(network) as network, values(Proxy_day) as Proxy_day, values(url) as url, dc(url) as url_count by src_ip, type | where type="proxy" | eval userid=coalesce(userid, win_userid) | stats values(mail) as "Email Address", values(userid) as "User ID", values(destination) as Destination, values(network) as Network, dc(Proxy_day) as Day_Count, dc(url) as URL_Count by src_ip, url, Proxy_day | stats values("Email Address") as "Email Address", values("User ID") as "User ID", values(Destination) as Destination, values(Network) as Network, values(Day_Count) as Day_Count, dc(Proxy_day) as Total_Days, values(URL_Count) as URL_Count by src_ip, url    Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
@NorthropGrumman  If you just want to limit with 5 with your existing search, you can use mvindex. Check and let me know if this is what you are trying to achieve. Eg: | stats values(mail) as "Em... See more...
@NorthropGrumman  If you just want to limit with 5 with your existing search, you can use mvindex. Check and let me know if this is what you are trying to achieve. Eg: | stats values(mail) as "Email Address" values(username) as "User ID" values(destination) as Destination values(network) as Network values(Proxy_day) as Day values(url) as URL by "Source IP" | eval "Email Address" = mvindex('Email Address', 0, 5) | eval "User ID" = mvindex('User ID', 0, 5) | eval Destination = mvindex(Destination, 0, 5) | eval Day = mvindex(Day, 0, 5) | eval URL = mvindex(URL, 0, 5) Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!
Hi everyone and thanks in advance. I'm trying to collate all our SOCKS traffic on our network over the last 90 days. Our IP's rotate and as a result I can't run this search for all time, I have to ... See more...
Hi everyone and thanks in advance. I'm trying to collate all our SOCKS traffic on our network over the last 90 days. Our IP's rotate and as a result I can't run this search for all time, I have to run it for 90 days individually, Which is where I got to here: index=*proxy* SOCKS earliest=-1d latest=-0d | eval destination=coalesce(dest, dest_port), userid=coalesce(user, username) | rex field=url mode=sed "s/^SOCKS:\/\/|:\d+$//g" | eval network=case(match(src_ip,"<REDACTED>"),"user",1=1,"server") | stats values(domain) as Domain values(userid) as Users values(destination) as Destinations by url, src_ip, network | convert ctime(First_Seen) ctime(Last_Seen) | sort -Event_Count | join type=left max=0 src_ip [ search index=triangulate earliest=-1d latest=-0d |stats count by ip,username |rename username AS userid |rename ip as src_ip ] | join type=left max=0 src_ip [ search index=windows_events EventID=4624 NOT src_ip="-" NOT user="*$" earliest=-1d latest=-0d | stats count by IpAddress, user | rename IpAddress as src_ip | rename user as win_userid | fields - count ] |eval userid=coalesce(userid, win_userid) | join type=left max=0 userid [ search index="active_directory" earliest=-1d latest=-0d | stats count by username,fullname,title,division,mail | rename username as userid ] Then a colleague suggested I do it slightly differently and run it over the 90 days but link it together which is where we got to here: index=*proxy* SOCKS | eval destination=coalesce(dest, dest_port) | rex field=url mode=sed "s/^SOCKS:\/\/|:\d+$//g" | eval network=case(match(src_ip,"<Redacted>"),"user",1=1,"server") | eval Proxy_day = strftime(_time, "%d-%m-%y") | join type=left max=0 src_ip [ search index=windows_events EventID=4624 NOT src_ip="-" NOT user="*$" | stats count by IpAddress, user | rename IpAddress as src_ip | rename user as win_userid | fields - count ] | eval userid=coalesce(userid, win_userid) | join type=left max=0 userid [ search index="active_directory" | stats count by username, fullname, title, division, mail | rename username as userid ] | rename src_ip as "Source IP" | stats values(mail) as "Email Address" values(username) as "User ID" values(destination) as Destination values(network) as Network values(Proxy_day) as Day values(url) as URL by "Source IP" However the problem I'm running into now is in the data produced there could be 100's of URL's / Emails / Day associated with the source IP which makes the data unactionable and actually starts to break a .csv when exported. Would anyone be able to help? Ideally I'd just like the top for example 5 results, but I've had no luck with that or a few other methods I've tried. Even SplunkGPT is failing me - is it even possible?
Thank you @PickleRick, so much to unpack.... I'm not reading that I'm wrong regarding the field extraction, so where do you submit an issue to correct this? Even if it is a single field it should at... See more...
Thank you @PickleRick, so much to unpack.... I'm not reading that I'm wrong regarding the field extraction, so where do you submit an issue to correct this? Even if it is a single field it should at least work as intended. I also noticed that both the auditd and linux_audit sourcetypes of the TA_nix app are also in a section following this: # Stanzas in this section are legacy configuration stanzas. So I'm guessing that there is no "current" way to collect audit log. Neither scripted through ausearch or by reading the logfile. The solution therefore seems straightforward, I need another app to deal with audit logs. I cannot use any scripted solution relying on ausearch but must read a local audit log file being dumped. To make the suggested app https://splunkbase.splunk.com/app/4232 (which does look good , thank you) "backwards compatible" I'd need to perform "several minor (unsupported) changes". Or I can just "live with it" as it is not working correctly at the moment anyway and switch over to another sourcetype. This manual entry https://docs.splunk.com/Documentation/AddOns/released/Linux/Configure4 points to the same sourcetype as the documentation for the AuditD TA, "linux:audit". But also indicates the use of yet another app https://splunkbase.splunk.com/app/3412 which I will be unable to use as it relies on changes to auditd on the host and making use of HEC traffic rather than a filewatch. While I am leaning towards the "minor but unsupported changes", what would be the recommended path forward from someone with a deeper understanding of the issue? a) Performing the "minor but unsupported changes" to the app, i.e. including the deprecated/legacy linux_audit sourcetype and installing the recommended TA in the SH cluster (as well as HF and IX as per the documentation)? b) Switch the defined sourcetype (linux_audit) under the audit filewatch stanza in the inputs.conf of the Splunk_TA_nix app deployed to all universal forwarders over to linux:audit and installing the recommended TA in the SH cluster (as well as HF and IX as per the documentation)? c) Building a new or heavily modifying an existing or merging several TAs to deal with audit logs in a manner closer to what is expected? I apologize if any cynicism or irritation is not fully tucked away and that this has become a bit of a snowball. I do appreciate all the help, feedback and suggestions.
hi @Venality  The code that is kicking out this error is: session_key = sys.stdin.read() if not session_key: print("No session key received. Exiting") sys.exit(1) T... See more...
hi @Venality  The code that is kicking out this error is: session_key = sys.stdin.read() if not session_key: print("No session key received. Exiting") sys.exit(1) This suggests that the passAuth param is not set, or is set to a user that does not exist in your inputs.conf. The default for this is: ## inputs.conf ## [script://$SPLUNK_HOME/etc/apps/phantom/bin/scripts/phantom_retry.py] passAuth = admin When you setup Splunk, did you seed it with/create a different username for your admin user? Or have you changed this value? In short, it looks like you need to update the passAuth to be a valid admin user (or the inbuilt "splunk-system-user") on your system. You need to make this change in the local directory, not default, as it could get overwritten.  ## $SPLUNK_HOME/etc/apps/phantom/local/inputs.conf ## [script://$SPLUNK_HOME/etc/apps/phantom/bin/scripts/phantom_retry.py] passAuth = AdminUsernameHere    Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi @unluakin    use the below btool command and check   $SPLUNK_HOME/bin/splunk btool web list --debug
@PickleRick  Thanks for adding that extra context! Just to clarify, I wasn’t suggesting anything should run on an indexer(I believe you referenced this in earlier comment) — I was outlining the avai... See more...
@PickleRick  Thanks for adding that extra context! Just to clarify, I wasn’t suggesting anything should run on an indexer(I believe you referenced this in earlier comment) — I was outlining the available ingestion methods (HEC and SDK). And yep, I completely agree: HEC is usually best deployed on a Heavy Forwarder, especially in production environments(Again it depends on the requirements/situation). Also, I’ve actually built both standalone scripts and modular inputs using the Python SDK(If you have dev background, yep its straight forward) — so I meant that quite literally! It’s a solid way to integrate external sources without needing a forwarder #https://dev.splunk.com/enterprise/docs/devtools/python/sdk-python/ Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!
Thanks for the feedback.   How can I check this?   3.web.conf should n't override in the ES App.    
Transform classes are called in alphabetical order. And please don't call out specific people for help. That's rude.
That's not entirely true. Typically you'd set up a HEC input on a HF layer. True, you can use HEC input directly on indexers but it's not the best solution typically.  Also, what does "install Pytho... See more...
That's not entirely true. Typically you'd set up a HEC input on a HF layer. True, you can use HEC input directly on indexers but it's not the best solution typically.  Also, what does "install Python SDK and write a script" mean? Have you ever done that? With SDK you can write a modular input which... tadaaaam! runs on a HF. Technically - again - you could run it on idx but that's an even worse idea.
Hi @schose  I believe that WLM configuration existed (but not supported/fully implemented) in UF <9.x (You can confirm exact versions by checking for existence of workload_policy.conf, workload_pool... See more...
Hi @schose  I believe that WLM configuration existed (but not supported/fully implemented) in UF <9.x (You can confirm exact versions by checking for existence of workload_policy.conf, workload_pools.conf & workload_rules.conf files).  This led to SPL-224264 which caused some failures starting UF 9.x when upgrading from 8.x where the startup script contained the ExecStartPost commands you referenced.  I believe this should be fixed in a later 9.0.x and 9.1.x version but cannot find the exact version at the moment.  As a workaround for this users should disable and re-enable boot-start.  I have also submitted feedback regarding the Reference unit file template in the docs which obviously hasnt been updated the reflect the change when they removed the WLM configurations from UF.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi @Cheng2Ready  You can use a * for any value in the current directory/segment or ... to recursively wildcard. Therefore you can do the following: [monitor://\\njros1bva0597\d$\LogFiles\warcraft-9... See more...
Hi @Cheng2Ready  You can use a * for any value in the current directory/segment or ... to recursively wildcard. Therefore you can do the following: [monitor://\\njros1bva0597\d$\LogFiles\warcraft-9.*\logs\*] disabled = false host = NJROS1BVA0621 alwaysOpenFile = 1 sourcetype = Image Importer Logs Check out https://docs.splunk.com/Documentation/Splunk/latest/Data/Specifyinputpathswithwildcards for more info on this and https://community.splunk.com/t5/Getting-Data-In/What-is-the-proper-use-of-wildcard-in-a-file-monitor-path/m-p/347282 for other good examples. Is there any particular reason you're using alwaysOpenFile=1? This is only useful for files that do not update modification time or size and adds resource overhead, so wanted to check.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Here is my props and transforms props.conf [default] TRANSFORMS-save_original_sourcetype = save_original_sourcetype TRANSFORMS-clone_for_thirdparty = clone_for_thirdparty [data_to_thirdparty] S... See more...
Here is my props and transforms props.conf [default] TRANSFORMS-save_original_sourcetype = save_original_sourcetype TRANSFORMS-clone_for_thirdparty = clone_for_thirdparty [data_to_thirdparty] SHOULD_LINEMERGE = true TRANSFORMS-updateFields = sourcetype_raw_updated TRANSFORMS-route_thirdparty = route_thirdparty transforms.conf # 1. Save original sourcetype as a field (for use in the clone) [save_original_sourcetype] SOURCE_KEY = MetaData:Sourcetype REGEX = (.+) FORMAT = orig_sourcetype::$1 WRITE_META = true # 2. Clone only events from vme_ops_prod to sourcetype=data_to_thirdparty [clone_for_thirdparty] SOURCE_KEY = _MetaData:Index REGEX = ^test_np$ DEST_KEY = MetaData:Sourcetype CLONE_SOURCETYPE = data_to_thirdparty WRITE_META = true # 3. Meta-data transforms for the clone [sourcetype_raw_updated] SOURCE_KEY=MetaData:orig_sourcetype REGEX=^orig_sourcetype::(.*)$ FORMAT = $1##$0 DEST_KEY=_raw # 4. Route the cloned event ONLY to thirdparty [route_thirdparty] SOURCE_KEY = _MetaData:Index REGEX = (^test_np.*) DEST_KEY = _TCP_ROUTING FORMAT = dev_thirdparty   I'm sending logs (no cooked data) to thirdpartyserver over TCP without disturbing existing flow. so I just cloned the event and adding original sourcetype to the cloned event and sending to thirdparty output group. now the issue here is I can't find the original source type.  
@MatheoCaneva1  You can send data directly to a Splunk index using a Python script by leveraging the HTTP Event Collector (HEC) or the Splunk SDK for Python. Both methods bypass the need for a f... See more...
@MatheoCaneva1  You can send data directly to a Splunk index using a Python script by leveraging the HTTP Event Collector (HEC) or the Splunk SDK for Python. Both methods bypass the need for a forwarder Option 1 - Send Data via HTTP Event Collector -Enable HEC in Splunk -Create script and send data Option 2 - Use Splunk SDK for Python -Install splunk SDK -Create script using Splunk SDK and send data Option 1 is lightweight, fast and easy. Option 2 is having more functionalities, since you are interacting with full Splunk API. Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!