All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

We are using Microsoft Graph Security API Add-On to ingest all security alerts for our org using the Microsoft Graph Security API. Recently access token became invalid and the data paused. We renewed... See more...
We are using Microsoft Graph Security API Add-On to ingest all security alerts for our org using the Microsoft Graph Security API. Recently access token became invalid and the data paused. We renewed the secret key and updated with new one. Till then data ingestion is paused and no backfill happened after that. Is there any way I can backfill that paused one week data. Any help or point of contacts or link to read would be beneficial? Is that something to do in source end to re-ingest the data via Add-On again?
Hi there, I have this query: index=_internal source="*license_usage.log" | eval bytes=b | eval GB = round(bytes/1024/1024/1024,3) | timechart span=1d sum(GB) by h This query shows results like th... See more...
Hi there, I have this query: index=_internal source="*license_usage.log" | eval bytes=b | eval GB = round(bytes/1024/1024/1024,3) | timechart span=1d sum(GB) by h This query shows results like this: _time host1 .... 2023-11-10     2023-11-11     ...       And I want results like this: Host 2023-11-10 .... host1     host2     ...       How I can do this?
Scatter plots are two-dimensional so they take 3 arguments: value, x-axis, and y-axis.  A fourth argument would call for a three-dimensional chart and that calls for an add-on like https://splunkbase... See more...
Scatter plots are two-dimensional so they take 3 arguments: value, x-axis, and y-axis.  A fourth argument would call for a three-dimensional chart and that calls for an add-on like https://splunkbase.splunk.com/app/3138.  See https://docs.splunk.com/Documentation/Splunk/9.1.2/Viz/ScatterChart
Hello @Manami , We are experiencing the same thing with Splunk Enterprise, Memory utilization on average went up ~30% and CPU load over 50% across the indexing tier when we moved to this version. I ... See more...
Hello @Manami , We are experiencing the same thing with Splunk Enterprise, Memory utilization on average went up ~30% and CPU load over 50% across the indexing tier when we moved to this version. I will let you know if anything is found with the recent case which was opened. Were you able to find the problem with the universal forwarder?  
Hello, I have a problem, how do I get a fourth value from a table and scatterplot to use the value in a token? We have a table with 4 columns and a scatterplot chart for display. article, value, ... See more...
Hello, I have a problem, how do I get a fourth value from a table and scatterplot to use the value in a token? We have a table with 4 columns and a scatterplot chart for display. article, value, calculate category and PartnerId Unfortunately, the PartnerId is not displayed in the scatterplot chart. Can I somehow read out the fourth value to display further details on a PartnerID in a dashboard?
Thanks a lot! This was very helpful and exactly what I needed. I appreciate you sharing the documentation links as well, been reading through it. 
As far as I know, no volunteer here possesses mind-reading superpower.  If you want concrete help, illustrate (in text) relevant data input (anonymize as needed but preserver key characteristics), il... See more...
As far as I know, no volunteer here possesses mind-reading superpower.  If you want concrete help, illustrate (in text) relevant data input (anonymize as needed but preserver key characteristics), illustrate (in text) desired output - you already did, then explain the logic to arrive at result from input. If you have a field called "month" with values "August", "Sept", "Oct", and have a field named country with values "AUS", "JER", "IND", "ASI", this search will give you a semblance of what you illustrated. | chart count over month by country  
The command you are looking for is eventstats. index=zzzzzz | stats count as Total, count(eval(txnStatus="FAILED")) as "Failed_Count", count(eval(txnStatus="SUCCEEDED")) as "Passed_Count" by country... See more...
The command you are looking for is eventstats. index=zzzzzz | stats count as Total, count(eval(txnStatus="FAILED")) as "Failed_Count", count(eval(txnStatus="SUCCEEDED")) as "Passed_Count" by country, type, ProductCode, errorinfo | eventstats sum(Total) as Total by country | fields country, ProductCode, type, Failed_Count, Passed_Count, errorinfo, Total  
Sometimes, running the same search generates different orders when trellis visualization is used.  For example,   ((sourcetype=A field1=*) OR (sourcetype=B user=* field2=*)) clientip=* earliest="0... See more...
Sometimes, running the same search generates different orders when trellis visualization is used.  For example,   ((sourcetype=A field1=*) OR (sourcetype=B user=* field2=*)) clientip=* earliest="01/24/2023:02:00:00" latest="01/24/2023:08:00:00" | fields clientip user field1 field2 | eval user = mvindex(split(user, ":"), 1) | eventstats values(user) as user by clientip | eval clientip = clientip . if(isnull(user), "/", "/" . mvjoin(user, ",")) | timechart span=5m limit=19 count(field1) as s1 count(field2) as s2 by clientip   Here, field1 only exist in sourcetype A, user and field2 only exist in sourcetype B; search period is fixed in the past.  This means that search result cannot change.  But the following are two screenshots of two consecutive executions.   They show the same number of trellis with exact same clientip titles; each clientip's graph is also the same across the two runs.  But obviously the order is rearranged. (In Statistics view, columns are arranged in lexicographic order of "s1:clientip" and "s2:clientip".) Is there some way to be certain of the order?
Hi This is doable. You could e.g. add several time pickers on your dashboards and add those to your queries as tokens. I cannot recall now if this needs you to add additional tokens to set those limi... See more...
Hi This is doable. You could e.g. add several time pickers on your dashboards and add those to your queries as tokens. I cannot recall now if this needs you to add additional tokens to set those limits correctly in your search? r. Ismo
Please illustrate/mock the data.  Without knowing the actual data structure, it is impossible to know the relationship and your true intention.
The log line is about 9999 characters long with spaces, and not all the log line is ingested - I think i need to create a limits.conf file?  Absolutely.  Good data is the only guarantee that any... See more...
The log line is about 9999 characters long with spaces, and not all the log line is ingested - I think i need to create a limits.conf file?  Absolutely.  Good data is the only guarantee that any work on it will be valid. This said, Splunk's KV extraction does not look beyond the first occurrence of key. (And that's a good thing.  It is a risky proposition for any language to assume the intention of multiple occurrences of a left-hand side value.) The main problem is caused by the developers, who take pains to invent a structured data that is not standard.  It seems that they use foo[] to indicate an array (events), then use bar() to indicate an element; inside element, they use = to separate key and value.  Then, on top of this, they use geez() to signal a top level structure ("VUpdate") with key-value pairs that includes the events[] array.  If you have any influence over developers, you should urge them, beg them, implore them to use a standard structured representation such as JSON. If not, you can use Splunk to try to parse out the structure.  But this is going to be messy and will never be robust.  Unless your developers swear on their descendants' descendants (and their ancestors' ancestors) not to change format, you future can be ruined at their whim. Before I delve into SPL, I also want to clarify this: Splunk already give you the following fields: channelCode, contentType, duration, eventNumber, eventTitle, events, onAir, system, type, and utcStartDateTime.  Is this correct?  While you can ignore any second level fields such as eventTitle and eventNumber, I also want to confirm that events includes the whole thing from [ all the way to ].  Is this correct? I'll suggest two approaches, both rely on the structure I reverse engineered above.  The first one is straight string manipulation, and uses Splunk's split function to isolate individual events.   | fields system channelCode channelCode type events | eval events = split(events, "),") | mvexpand events | rename events AS _raw | rex mode=sed "s/^[\[\s]*Event\(// s/[\)\]]//g" | kv kvdelim="=" pairdelim=","   The second one tries to "translate" your developers's log structure into JSON using string manipulation.   | rex field=events mode=sed "s/\(/\": {/g s/ *\)/}}/g s/=\s+/=/g s/\s+,/,/g s/(\w+)=([^,}]+)/\"\1\": \"\2\"/g s/\"(true|false)\"/\1/g s/Event/{\"Event/g" | spath input=events path={} | fields - events | mvexpand {} | spath input={} | fields - {} | rename Event.* As *   The second approach is not more robust; if anything, it is less.  But it better illustrates the perceived structure.  Either way, your sample data should give you something like channelCode contentType duration eventNumber eventTitle onAir system type utcStartDateTime UH Prog 00:00:05.000 725538339 BooRadley true GRP1-VIPE NextEvents 2023-11-17T15:42:10.160Z UH Bumper 00:00:02.000 725538313 REGGAE-2 false GRP1-VIPE NextEvents 2023-11-17T15:42:15.160Z UH Commercial 00:01:30.000 725538320 CHRISITAN MISSION false GRP1-VIPE NextEvents 2023-11-17T15:42:17.160Z This is an emulation you can play with and compare with real data   | makeresults | eval _raw = "20231117154211 [18080-exec-9] INFO EventConversionService () - SArts: VUpdate(system=GRP1-VIPE, channelCode=UH, type=NextEvents, events=[Event(onAir=true, eventNumber=725538339, utcStartDateTime=2023-11-17T15:42:10.160Z, duration=00:00:05.000, eventTitle=BooRadley, contentType=Prog ), Event(onAir=false, eventNumber=725538313, utcStartDateTime=2023-11-17T15:42:15.160Z, duration=00:00:02.000, eventTitle= REGGAE-2, contentType=Bumper), Event(onAir=false, eventNumber=725538320, utcStartDateTime=2023-11-17T15:42:17.160Z, duration=00:01:30.000, eventTitle=CHRISITAN MISSION , contentType=Commercial)])" | extract ``` data emulation above ```   Hope this helps.
Greetings, I have Splunk 9.1.1 trying to import an aruba 7210 into splunk using the aruba app with udp 514. Sourcetype: aruba:syslog. I have other devices (cisco) going into the same splunk instance ... See more...
Greetings, I have Splunk 9.1.1 trying to import an aruba 7210 into splunk using the aruba app with udp 514. Sourcetype: aruba:syslog. I have other devices (cisco) going into the same splunk instance and they are reporting ok.  the splunk server can ping the aruba and vice versa.  should i try any other source types?  anything else I should look for to search under the hood to see why communication is not occurring? Thank you,
I have a use case that requires logging to be captured and have following this document here: How do I set up the ForgeRock Identity Cloud app for Splunk? Which references --> https://splunkbas... See more...
I have a use case that requires logging to be captured and have following this document here: How do I set up the ForgeRock Identity Cloud app for Splunk? Which references --> https://splunkbase.splunk.com/app/6272 ForgeRock Identity Cloud App for Splunk captures audit and debug logs from ForgeRock Identity Cloud tenants. A sample dashboard is included to graphically illustrate various captured metrics, for example, authentication events, identity registrations, and top-active users. Sample searches are also included to extend or modify the sample dashboard. Problem is the app should not be calling the following endpoint: /monitoring/logs/tail  It should be calling the following endpoint as noted in the ForgeRock Product Documentation-> /monitoring/logs To reduce unwanted stresses on the system, Identity Cloud limits the number of requests you can make to the /monitoring/logs endpoint in a certain timeframe: The page-size limit is 1000 logs per request. The request limit is 60 requests per minute. The theoretical upper rate limit is therefore 60,000 logs per minute. The reason this needs to be changed is when  using the Logs tail endpoint The /monitoring/logs/tail endpoint has the same limits and response headers as the /monitoring/logs endpoint described above. However, the endpoint also has a limit of 20,000 lines per request, which supersedes the page-size limit of 1000 logs per request. Because calls to the /monitoring/logs/tail endpoint do not always fetch all logs, use this endpoint for debugging only. Use the /monitoring/logs endpoint when you need to fetch all logs. I did find: grep -i -R "/tail" forgerock/   Which pointed me to :   forgerock//bin/input_module_forgerock.py:        response = helper.send_http_request(forgerock_id_cloud_tenant_url + "/monitoring/logs/tail", 'GET', parameters=parameters, payload=None, headers=headers, cookies=None, verify=True, cert=None, timeout=60, use_proxy=False)   Lines 51-52 of input_module_forgerock.py shows:   # The following examples send rest requests to some endpoint. response = helper.send_http_request(forgerock_id_cloud_tenant_url + "/monitoring/logs/tail", 'GET', parameters=parameters, payload=None, headers=headers, cookies=None, verify=True, cert=None, timeout=60, use_proxy=False) I suspect updating this to the following /monitoring/logs may resolve this and restarting the app:   # The following examples send rest requests to some endpoint. response = helper.send_http_request(forgerock_id_cloud_tenant_url + "/monitoring/logs", 'GET', parameters=parameters, payload=None, headers=headers, cookies=None, verify=True, cert=None, timeout=60, use_proxy=False) But when trying to grab logs its failing: 2023-11-16 15:33:34,178 DEBUG pid=261576 tid=MainThread file=connectionpool.py:_make_request:461 | https://openam-testxyz.id.forgerock.io:443 "GET /monitoring/logs?source=am-authentication%2Cam-access%2Cam-config%2Cidm-activity&_pagedResultsCookie=eyJfc29ydEzbnRpY25Il19fQ HTTP/1.1" 500 74 2023-11-16 15:33:34,179 INFO pid=261576 tid=MainThread file=base_modinput.py:log_info:295 | Unexpected response from ForgeRock: 500 2023-11-16 15:33:34,179 ERROR pid=261576 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events. Login Traceback (most recent call last): File "/opt/splunk/etc/apps/forgerock/bin/forgerock/aob_py3/modinput_wrapper/base_modinput.py", line 128, in stream_events self.collect_events(ew) File "/opt/splunk/etc/apps/forgerock/bin/forgerock.py", line 76, in collect_events input_module.collect_events(self, ew) File "/opt/splunk/etc/apps/forgerock/bin/input_module_forgerock.py", line 60, in collect_events response.raise_for_status() File "/opt/splunk/etc/apps/forgerock/bin/forgerock/aob_py3/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) Hoping someone has an idea @jknight 
In general case endpoint define what actions you can do for your event. But when I read again your issue, this is not your problem. I’m quite sure that @PickleRick ‘s note is valid in your case. You ... See more...
In general case endpoint define what actions you can do for your event. But when I read again your issue, this is not your problem. I’m quite sure that @PickleRick ‘s note is valid in your case. You need to try CLONE_SOURCETYPE and then manage those transforms etc. for each individual sourcetype separately.
Just wanted to toss in another possible resolution.  If your system was ungracefully shutdown, resource exhaustion, virtualization host failure, etc. it's possible the splunkd.pid file wasn't cleaned... See more...
Just wanted to toss in another possible resolution.  If your system was ungracefully shutdown, resource exhaustion, virtualization host failure, etc. it's possible the splunkd.pid file wasn't cleaned up.  From what I have found, when splunkd attempts to start, it looks at this file and attempts various operations on the existing PIDs that are listed within.  They are obviously not there and the service will fail to start.  This will be made apparent when attempting to start splunk via the binary:  SPLUNK_HOME/bin/splunk start This file is located in SPLUNK_HOME/var/run/splunk/ and can be safely removed to correct.  A new file will be created upon splunkd service starting successfully.   
1) Please show the SPL you've tried and tell us how it failed you.  It would help to see an actual (sanitized) event, too. The options to the extract command are swapped.  kvdelim is the character t... See more...
1) Please show the SPL you've tried and tell us how it failed you.  It would help to see an actual (sanitized) event, too. The options to the extract command are swapped.  kvdelim is the character that separates key from value, usually "="; pairdelim is the character that separates kv pairs, usually comma or space. 2) The props.conf file has a TRUNCATE setting that defaults to 10000.  Perhaps your system has a lower value.
OK. I did some more tries and the final "solution" would be like this (notice that as I need to rewrite the queue in the destination sourcetype based on the event's contents - need to match only some... See more...
OK. I did some more tries and the final "solution" would be like this (notice that as I need to rewrite the queue in the destination sourcetype based on the event's contents - need to match only some of the events - I have to make sure the queue-manipulating transforms are invoked before the event-cutting ones). props.conf: [test_sourcetype_to_recast] #Order of transform classes i crucial! You can do the same casting multiple transforms from #one class TRANSFORMS-0_extract_host = picklerick_extract_host TRANSFORMS-1_extract_source = picklerick_extract_source TRANSFORMS-2_recast_sourcetype = picklerick_recast_sourcetype TRANSFORMS-3_drop_dead = picklerick_drop_dead [destination_sourcetype] TRANSFORMS-0_drop_all_except_proper_ones = drop_dead_all,keep_matching_events TRANSFORMS-conditional_host_overwrite = conditional_host_overwrite TRANSFORMS-cut_most_of_the_event = cut_most_of_the_event transforms.conf: [picklerick_drop_dead] REGEX = sourcetype:destination_sourcetype DEST_KEY = queue FORMAT = nullQueue [picklerick_recast_sourcetype] REGEX = (.) CLONE_SOURCETYPE = destination_sourcetype [picklerick_extract_source] REGEX = source:(\w*) FORMAT = source::$1 DEST_KEY = MetaData:Source WRITE_META = true [picklerick_extract_host] REGEX = host:(\w*) FORMAT = host::$1 DEST_KEY = MetaData:Host WRITE_META = true [conditional_host_overwrite] REGEX = desthost=(\w*) FORMAT = host::$1 DEST_KEY = MetaData:Host WRITE_META = true [cut_most_of_the_event] REGEX = .*:([^:]*)$ FORMAT = $1 DEST_KEY = _raw WRITE_META = true [drop_dead_all] REGEX = . DEST_KEY = queue FORMAT = nullQueue [keep_matching_events] REGEX = sourcetype:destination_sourcetype DEST_KEY = queue FORMAT = indexQueue  This way if I include a string "sourcetype:destination_sourcetype" in my event contents, the event will be recast to the destination_sourcetype and processed accordingly. The downsides to this are two: 1) You can't specify the sourcetype you'll be cloning your event to dynamically so if you have many of them in one event stream... that's going to get complicated. 2) You have to account for all non-matching events in the destination sourcetype which can be tricky in case of an already existing sourcetype (like your linux_audit). I'm wondering if you could do something with it by firstly cloning that to an intermediate sourcetype, filtering your data there (the queue field should be retained over cloning) and cloning them again to the destination sourcetype. But that's getting ridiculously complicated.
I don't know off the top of my head if you can get to the PDF generation functionality from within Python - I've never tried in python and I haven't been able to find any documentation within Splunk'... See more...
I don't know off the top of my head if you can get to the PDF generation functionality from within Python - I've never tried in python and I haven't been able to find any documentation within Splunk's documentation if they officially expose those calls in their Python SDK. The PDF stuff in Splunk is based off of ReportLab, and lives in %SPLUNK_HOME%\Python-3.7\Lib\site-packages\reportlab, so you can poke aroundthere a bit to see if that helps. I wasn't finding anything obvious of Splunk-written python calling the ReportLab stuff to generate a PDF, your luck might be better. You can get a PDF generated from a REST endpoint - but do keep in mind that the overall PDF generation in Splunk is dashboard-centric, not search-centric.  So you will need to create a Dashboard that renders the results for your SID, and then have that be generated as a PDF.  Here are a few posts of people generating the PDF via a REST call which you can do from the CLI, within Python, etc.  Once you have those bytes you can send those on to where you need: Solved: Re: Generate PDF from View in REST API - Splunk Community Trigger a PDF via the command line - Splunk Community Historically the built-in PDF generation of Splunk has left something to be desired.  It's just been OK...There are some apps on Splunkbase that help with various PDF things you could look into as well: PDF Apps: https://splunkbase.splunk.com/apps?keyword=pdf     
Sorry I was not being clear in my question. The general picture is I am trying to monitor disk usage with events provided through collectd. The events being generated is in JSON format. I am looking... See more...
Sorry I was not being clear in my question. The general picture is I am trying to monitor disk usage with events provided through collectd. The events being generated is in JSON format. I am looking to split Values into values0 and values1, dsnames into dsnames0 and dsnames1 because those are just the names of the index I came up with of the values inside those arrays. Values has the indexes of two numbers and dsnames has the indexes of read and write. I am trying to get the values of both read and write and be able to create aggregate functions with it. Currently I am only able to show one dsname row, it can be read or write, however, I want to be able to show both.