About awmorris

awmorris · ‎01-20-2022

I've been trying to resolve this since October and not getting traction. Turning to the community for help: I have seemingly contradictory information within the same log line makes me question- do we have an issue or not? On the one hand, i think i do because the history command shows the search is cancelled... and I trust this information. However, there are artifacts in the logs that make me question if the search is fully running (which appears to be true since "fully_completed_search=TRUE"... so I am now confused if we have a problem or not.) Why do searches show fully_completed_search=TRUE and has_error_warn=FALSE when the info field (and history command) show "cancelled" and have a tag of "error" BOTTOM LINE QUESTION: Are my searches are running correctly and returning all results or not? Sample _audit log search activity that I found - not sure if this gives any usable insight Audit:[timestamp=10-01-2021 16:31:40.338, user=redacted_user, action=search, info=canceled, search_id='1633105804.108286', has_error_warn=false, fully_completed_search=true, total_run_time=18.13, event_count=0, result_count=0, available_count=0, scan_count=133645, drop_count=0, exec_time=1633105804, api_et=1633104900.000000000, api_lt=1633105800.000000000, api_index_et=N/A, api_index_lt=N/A, search_et=1633104900.000000000, search_lt=1633105800.000000000, is_realtime=0, savedsearch_name="", search_startup_time="1270", is_prjob=false, acceleration_id="98DCBC55-D36C-4671-93CD-1A950D796EC4_search_redacted_user_311d202b50b71a64", app="search", provenance="N/A", mode="historical_batch", workload_pool=standard_perf, is_proxied=false, searched_buckets=53, eliminated_buckets=0, considered_events=133645, total_slices=331408, decompressed_slices=11305, duration.command.search.index=120, invocations.command.search.index.bucketcache.hit=53, duration.command.search.index.bucketcache.hit=0, invocations.command.search.index.bucketcache.miss=0, duration.command.search.index.bucketcache.miss=0, invocations.command.search.index.bucketcache.error=0, duration.command.search.rawdata=2533, invocations.command.search.rawdata.bucketcache.hit=0, duration.command.search.rawdata.bucketcache.hit=0, invocations.command.search.rawdata.bucketcache.miss=0, duration.command.search.rawdata.bucketcache.miss=0, invocations.command.search.rawdata.bucketcache.error=0, roles='redacted', search='search index=oswinsec (EventID=7036 OR EventID=50 OR EventID=56 OR EventID=1000 OR EventID=1001) | eval my_ts2 = _time*1000 | eval indextime=_indextime |table my_ts2,EventID | rename EventID as EventCode']

awmorris · ‎08-01-2020

to4kawa · ‎12-13-2019

|makeresults |eval _raw="Process Create: UtcTime: 2017-04-28 22:08:22.025 ProcessGuid: {a23eae89-bd56-5903-0000-0010e9d95e00} ProcessId: 6228 Image: C:\Program Files (x86)\Google\Chrome\Application\chrome.exe CommandLine: \"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe\" --type=utility --lang=en-US --no-sandbox --service-request-channel-token=F47498BBA884E523FA93E623C4569B94 --mojo-platform-channel-handle=3432 /prefetch:8 CurrentDirectory: C:\Program Files (x86)\Google\Chrome\Application\58.0.3029.81\ User: LAB\rsmith LogonGuid: {a23eae89-b357-5903-0000-002005eb0700} LogonId: 0x7EB05 TerminalSessionId: 1 IntegrityLevel: Medium Hashes: SHA256=6055A20CF7EC81843310AD37700FF67B2CF8CDE3DCE68D54BA42934177C10B57 ParentProcessGuid: {a23eae89-bd28-5903-0000-00102f345d00} ParentProcessId: 13220 ParentImage: C:\Program Files (x86)\Google\Chrome\Application\chrome.exe ParentCommandLine: \"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe\" " | rex mode=sed "s/(\\\)/\1\1/g" | eval _raw=replace(_raw,"\"","\\\"") | rex mode=sed "s/(?m)\/$/\/ /g" | rex mode=sed "s/(?m) ?(.+?): (.+)$/\"\1\": \"\2\"/g" | rex mode=sed "s/(?m)$/,/g" | rex mode=sed "s/^(.+?):.+/{\"\1\":{/1" | rex mode=sed "s/,$/}}/1" | spath I do not recommend.

niketn · ‎05-26-2020

@woodcock how about this answer: https://answers.splunk.com/answers/529004/is-there-a-way-to-display-more-than-20-charts-at-a.html#answer-816673

woodcock · ‎06-19-2019

This also assumes that your events are true JSON (no whitespace) like my events are. If you really have all of that formatting whitespace, then have whomoever is sending it into Splunk trim out that fat and save you BIG BUCK$.

my2ndhead · ‎03-25-2019

Just like this e.g: | from datamodel:Authentication | extract vs. index=* source="XmlWineventlog:Security" tag=authentication NOT (user=*$ action=success ) The number of fields will not be the same, as extract does not add field aliases. Compared this with fieldsummary .

gjanders · ‎12-17-2018

I have a couple of dashboards in Alerts for Splunk Admins for this purpose, git links here and here Do they help? As you can see they are based on a conf presentation that is worth watching!

woodcock · ‎11-25-2019

There is a search on the Monitoring Console that gives you most of this but it is missing a piece: how to see what your actual effective retention is based on the buckets that are freezing. You can see that part with this search: index="_internal" AND sourcetype="splunkd" AND bucketmover AND freeze | rex "[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db[\/\\\]db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+" | rex "db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+.*?[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db" | sort 0 indexname - oldestTime | dedup indexname | eval retention = _time - oldestTime | fieldformat retention = tostring(retention, "duration") | table _time indexname retention

woodcock · ‎11-01-2018

This would make a GREAT project for an intern to write a custom splunk command. Python can trivially compute MD5 of a file. It would probably only take him a week to get it done (the python < 20 lines) and would enable your team to then have this powerful tool at your disposal for future supra-splunk solutions. That is what I would do (and I might even put one of my guys on this task as a learning exercise).

awmorris · ‎08-29-2018

Nevermind, I have one i'm happy with, for the benefit of others, here is where i landed: index=internal sourcetype=splunkd_access ( host="sh_svr*" ) ( method=POST OR method=DELETE ) ( user!=splunk-system-user user!=- user!=admin) ( uri_path=/servicesNS/* uri_path!="/user-prefs/" uri_path!="/notify" uri_path!="/batch_save" uri_path!="/servicesNS////jobs//control" uri_path!=/servicesNS//mobile_access ) | replace "/ui/views" with "/ui_views", "/props" with "*", "/distributed/peers*" with "/distributed_peers", "/server/serverclasses" with "/server_class" in uri_path | where mvcount( split( uri_path , "/" ) ) > 6 | eval activity = case( method=="POST" AND like( uri_path , "%/acl" ) , "Changed Permissions", method=="POST" AND NOT like( uri_path , "%/acl" ) , "Edited" , method="DELETE" , "Deleted" ) | rex field=uri_path "/servicesNS/(?[^\/]+)/(?[^\/]+)/(?[^\/]+)/(?[^\/]+)/(?[^\/]+)" | regex object_name!="\d+.\d+\w+-\w+-\w+-\w+-\w+" | eval object_name = urldecode( object_name ) | eval possession=if(like(uri,"%".user."%"),"Self-owned","N/A") | search possession != "Self-owned" | table _time, clientip, user, activity, app_dir, object_target, object_type, object_name, object_owner, possession | sort - _time

mportnoy_splunk · ‎08-26-2019

There is a typo above in the last SPL line - 'sourcetype' is spelled incorrectly...

DalJeanis · ‎03-28-2017

Easy - then you accept the one that you feel was most helpful or informative, and upvote the others.

awmorris · ‎12-02-2016

So, support answered my question and i did a subsequent face palm. I followed the documentation located here: http://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Specifyaproxyserver only to learn that i need to actually READ the entirety of the documentation.... specifically the part that says: "Note: The App Manager is not supported for use with a proxy server, if you use a proxy server with Splunk Web, you must download and update apps manually." Fault is mine for not reading the documentation ENTIRELY.

woodcock · ‎10-11-2016

Because index=main is a default dumpingground for when you don't specify an index, if your admin is doing his job correctly, it should have VERY few events coming into it. This means that you are going to have to dig into FAR more buckets and files to get 2000000 events. It looks like the events/bucket ratio is 20X less dense in main, than in the other indices. This makes sense to me. If you have to open 20 files to get 2000000 events (and each of those files requires uncompression) for index=udp_syslog but you have to open 2000000 files to get 2000000 for index=main , obviously it will take way longer. This does not even consider the fact that the older events in index=main are probably on slower storage, too.

dflodstrom · ‎05-01-2020

bump. We applied this fix ages ago but still seems to be an issue today.

Posts	39
Solutions	4
Karma Given	6
Karma Received	15
Member Since	‎12-18-2014

Online Status	Offline
Date Last Visited	‎01-20-2022 04:21 PM

Did my search work or not?

Moving beyond noob queries (comparing results)

Parsing of makeresults

advanced json handling

Schema Accelerated Event Search performance

Percent complete for data models

Can you help me craft a search that returns all in...

How do you hash an entire lookup file (detecting c...

How to audit user modifications to search head clu...

Search SPL to show messages menu

Did my search work or not?

Re: Moving beyond noob queries (comparing results)

Re: Parsing of makeresults

Re: What is the best way to get 100ish Greeen/Yell...

Re: advanced json handling

Re: Schema Accelerated Event Search performance

Re: Percent complete for data models

Re: Can you help me craft a search that returns al...

Re: How do you hash an entire lookup file (detecti...

Re: How to audit user modifications to search head...

Re: Search SPL to show messages menu

Re: Stymied by subsearches

Re: Splunk 6.5 is ignoring https_proxy in splunk-l...

Re: Significant performance differences running sa...

Re: A user can outputlookup and overwrite existing...