Knowledge Management

Why inputlookup doesn't return all the values?

CarmineCalo
Path Finder

Hello splunkers!

New problem to be solved...

This simple lookup

| inputlookup DOM_ServiceCatalogue

is not returning all the values (csv file is ~ 4MB, far away from the max size limit of 10MB set in the limit.conf, having ~ 7200 rows, 3 columns).
It seems to stop piping data from inputlook around row 2.500-3.000.
Lookup table is fine (i checked the content through the lookup editor app add-on).

These are the limit.conf settings

# maximum size of static lookup file to use a in-memory index for
max_memtable_bytes = 10000000
# maximum matches for a lookup
max_matches = 1000
# maximum reverse lookup matches (for search expansion)
max_reverse_matches = 50
# default setting for if non-memory file lookups (for large files) should batch queries
# can be override via a lookup table's stanza in transforms.conf
batch_index_query = true
# when doing batch request, what's the most matches to retrieve
# if more than this limit of matches would otherwise be retrieve, we will fall back to non-batch mode matching
batch_response_limit = 5000000
# maximum number of lookup error messages that should be logged
max_lookup_messages = 20

Do I have to change something to pipe all the data from inputlookup?

Tks!
Carmine

0 Karma
1 Solution

horsefez
Motivator

Hi CarmineCalo,

this might look like a strange question, but are there at some point in your csv-file values with (") double quotes?

If so, remove those double quotes and then the lookup will work as intended.

View solution in original post

0 Karma

493669
Super Champion

Hi @CarmineCalo,
Check if the index file is created alongside the CSV file in the lookups directory..
Because when a lookup CSV file is larger than limit(10MB), Splunk will create an index for the lookup file on disk. You will see the index file alongside the CSV file in the lookups directory. Every time that Splunk needs to access the lookup table, it examines the timestamp of the CSV file and the index file, and rebuilds the index file if needed
if any index file is present then try to delete that...

0 Karma

CarmineCalo
Path Finder

Unfortunately it continues to not working...

0 Karma

493669
Super Champion

so is their any index file was present?

0 Karma

CarmineCalo
Path Finder

In lookups folder (...\Splunk\etc\apps\search\lookups) there are only the lookup files (i tried to delete "DOM_ServiceCatalogue" file, but than inputlookup stopped to work).

THere is also a subfolder: lookup_file_backups, but nothing within the sub/sub/sub folder related to "DOM_ServiceCatalogue".

0 Karma

nickhills
Ultra Champion

Take a look at the search.log from the job inspector - this will give a clue as to what may be happening.

Is your |inputlookup doing this if its the only search you run, or does it just do this as part of a bigger spl query?

If my comment helps, please give it a thumbs up!
0 Karma

CarmineCalo
Path Finder

This is the main content of the search log but it doesn't help me...

01-22-2018 23:01:40.847 INFO  SearchParser - PARSING: | inputlookup DOM_ServiceCatalogue\n| rename ApplicationID as CI\n| lookup AMAP_ReqAvailability Cluster_Availability as PrimaryWindows OUTPUTNEW \n                                  ReqWeeklyAvailability as ReqWeekAva, \n                                  Sun as SunAvailability, \n                                  Mon as MonAvailability, \n                                  Tue as TueAvailability, \n                                  Wed as WedAvailability, \n                                  Thu as ThuAvailability, \n                                  Fri as FriAvailability, \n                                  Sat as SatAvailability,\n                                  Cluster_Ava_Code as Cluster_Ava_Code\n| stats max(ReqWeekAva) as ReqWeekAva, \n        max(MonAvailability) as ReqMonAva, \n        max(TueAvailability) as ReqTueAva, \n        max(WedAvailability) as ReqWedAva, \n        max(ThuAvailability) as ReqThuAva, \n        max(FriAvailability) as ReqFriAva, \n        max(SatAvailability) as ReqSatAva by CI,  Cluster_Ava_Code\n| fillnull value=0\n        \n| search CI="FRM"
01-22-2018 23:01:40.861 INFO  ISplunkDispatch - Not running in splunkd. Bundle replication not triggered.
01-22-2018 23:01:40.871 INFO  UserManager - Setting user context: admin
01-22-2018 23:01:40.871 INFO  UserManager - Free version does not have user services
01-22-2018 23:01:40.871 INFO  UserManager - Done setting user context: NULL -> NULL
01-22-2018 23:01:40.910 INFO  SortOperator - maxmem = 209715200
01-22-2018 23:01:40.915 INFO  UnifiedSearch - Processed search targeting arguments
01-22-2018 23:01:40.915 INFO  DispatchThread - BatchMode: allowBatchMode: 1, conf(1): 1, timeline/Status buckets(0):0, realtime(0):0, report pipe empty(0):0, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0
01-22-2018 23:01:40.917 INFO  DispatchThread - required fields list to add to remote search = *
01-22-2018 23:01:40.917 INFO  DispatchCommandProcessor - summaryHash=NS3d9d854163f8f07a summaryId=C7342F8D-CFAC-43F1-A7E8-3EF975823866_search_admin_NS3d9d854163f8f07a remoteSearch=

Thi inputlookup is both part of a big query (that it's not properly working due to the bug that I'm trying to fix), but in I'm debugging it stand-alone to fix the problem.

Carmine

0 Karma

horsefez
Motivator

Hi CarmineCalo,

this might look like a strange question, but are there at some point in your csv-file values with (") double quotes?

If so, remove those double quotes and then the lookup will work as intended.

0 Karma

CarmineCalo
Path Finder

Find the Issue!
THere where a strange char (different than ") in some fields...
Removed the strange char, now i pipe all the data from inputlookup!

Tks!
Carmine

0 Karma

horsefez
Motivator

Awesome to hear that 🙂

0 Karma

CarmineCalo
Path Finder

Yes, they were.
Replaced double quotes (") with quotes ('), but inputlookup continue to doesn't return all values...

Any other suggestion?

Carmine

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...