I am working on a complex model that needs to lookup values in a series of distinct tables. If I embed all of the information in the model itself, then the file quickly becomes unwieldy. I am hoping to find a solution where I can have a series of CSV files that contain all of the lookup tables, and then have my VBA code just quickly read each CSV file as necessary and return the appropriate value. My initial thought is to read each CSV file in working memory as needed, lookup the necessary values, then discard the information once the lookups are complete. Is that most efficient way to do it?
-
Could you load all your lookups into memory for the whole session, or would they be too large? What kind of lookups are you doing - just two-column or can the lookup value be from one of a number of columns?– Tim WilliamsJun 21, 2012 at 20:40
-
The lookups can be from multiple columns, but primarily just two-column stuff (or, more specifically, looking up the different 2-column combos in the same table multiple times - matching columns A and C, then B and D, then A and E, etc.)...I am not sure what memory limits are, but the tables are anywhere from a few hundred rows to 30,000 or 40,000 rows...– brentfJun 21, 2012 at 20:46
-
During a previous mission, I had excellent efficiency and reusability by creating a class module for each of the lookup file "types" I had to use. The advantage of this is that if the layout ever changes, you only have to update your class module. Additionally, if you work in a team, you can easily export and share those class modules.– iDevlopJun 21, 2012 at 21:48
2 Answers
Here's an idea which might work for you: load your csv file into a variant array the first time it's required, then subsequent calls will use the cached data. You can lookup values in any column and return the corresponding value from any other column.
EDIT: updated to show how to populate lookup arrays from CSV files
Sub Tester()
Dim arr1, arr2
arr1 = CsvToArray("D:\Analysis\tmp\Data1.csv")
arr2 = CsvToArray("D:\Analysis\tmp\Data2.csv")
Debug.Print TestLookup(arr1, "lookup1", 2, 1)
Debug.Print TestLookup(arr2, "lookup2", 3, 1)
'bunch more lookups...
End Sub
Function TestLookup(arr, val, lookincol As Integer, returnfromcol As Integer)
Dim r
r = Application.Match(val, Application.Index(arr, 0, lookincol), 0)
If Not IsError(r) Then
TestLookup = arr(r, returnfromcol)
Else
TestLookup = "Not found" 'or some other "error" value
End If
End Function
Function CsvToArray(filepath As String) As Variant
Dim wb As Workbook
Application.ScreenUpdating = False
Set wb = Workbooks.Open(filepath)
CsvToArray = wb.Sheets(1).Range("A1").CurrentRegion.Value
wb.Close False
End Function
-
Thanks for the response - I like this method, but I am not sure I understand it completely...I see how this is function is doing the look-up with a dynamic 'lookin' and 'returnfrom' col, but I am not clear on how I read the data from a file into an array (forgive my ignorance). I was thinking I would read each CSV into a user-defined type variable (I know file structures in advance) with a distinct variable array for each column. I would then pass my type variable to the lookup function to return the appropriate match. Is that compatible with this solution? Thanks again!– brentfJul 3, 2012 at 1:42
-
See edits: I altered the lookup function to add the data array as a parameter. Jul 3, 2012 at 3:54
-
Thanks again for the reply - I got this code working properly with my model. My last question would be related to efficiency/performance - is loading the CSV file into an actual workbook going to be a fast way to do it? My initial inclination was to read the file directly into an array through file scripting code - could you comment briefly on how efficient this method is versus reading the whole file into a giant string, for example, and then parsing the string based on the commas and new lines?– brentfJul 3, 2012 at 12:38
-
That's easy to test: if you're concerned about performance then you can easily run the relevant code in a loop and compare timings. I suspect opening the cvs as a workbook might be slightly slower, but maybe not if you factor in the time it takes to create the other approach. ;-) Jul 3, 2012 at 15:05
if you really have to do it in excel, then this is a method:
Function GetData(This As String, ResultCol As Integer)
Dim LastRow As Long
Application.ScreenUpdating = False 'Turn off screen refreshing
Workbooks.Open Filename:="E:\my_files\tables.csv" 'Open the CSV file
LastRow = Cells.Find("*", SearchOrder:=xlByRows, SearchDirection:=xlPrevious).Row 'used for range in vlookup
GetData = Application.WorksheetFunction.VLookup(This, Range(Cells(1, 1), Cells(LastRow, ResultCol)), ResultCol, False)
Workbooks("Tables.csv").Close 'Close the CSV file
Application.ScreenUpdating = True 'Turn on screen refreshing
End Function