Index the file when the program loads.
Create a Dictionary(Of ULong, Long)
, and when the program loads read through the file. For each line, add an entry to the dictionary showing the 13 digit value at the front of each line as the ULong key and the position in the file stream as the Long value.
Then, when a user puts in a key, you can check the dictionary, which will be almost instant, to find the exact location on disk you need and seek there directly.
Building the file index at program start may take a few moments, but you'll only ever have to do it once. Right now, you either need to search through the entire thing every time a user wants to do a search, or keep several hundred megabytes of text file data in memory. Once you have the index, looking up a value in the dictionary and then seeking directly to it should appear to happen almost instantly.
I just saw this comment:
there could be more than 1 occurrences of a 13 digit number so must search the whole file.
Based on that, the index should be a Dictionary(Of ULong, List(Of Long))
, where adding a value to entry first creates a list instance if one doesn't already exist, then adds the new value to the list.
Here's a basic attempt typed directly into the reply window without the aid of testing data or Visual Studio that likely therefore still contains several bugs:
Public Class MyFileIndexer
Private initialCapacity As Integer = 1
Private Property FilePath As String
Private Index As Dictionary(Of ULong, List(Of Long))
Public Sub New(filePath As String)
Me.FilePath = filePath
RebuildIndex()
End Sub
Public Sub RebuildIndex()
Index = New Dictionary(Of ULong, List(Of Long))()
Using sr As New StreamReader(FilePath)
Dim Line As String = sr.ReadLine()
Dim position As Long = 0
While Line IsNot Nothing
'Process this line
If Line.Length > 13 Then
Dim key As ULong = ULong.Parse(Line.SubString(0, 13))
Dim item As List(Of Long)
If Not Index.TryGetValue(key, item) Then
item = New List(Of Long)(initialCapacity)
Index.Add(key, item)
End If
item.Add(position)
End If
'Prep for next line
position = sr.BaseStream.Position
Line = sr.ReadLine()
End While
End Using
End Sub
'Expect key to be a 13-character numeric string
Public Function Search(key As String) As List(Of String)
'Will throw an exception if parsing fails. Be prepared for that.
Dim realKey As ULong = ULong.Parse(key)
Return Search(realKey)
End Function
Public Function Search(key As ULong) As List(Of String)
Dim lines As List(Of Long)
If Not Index.TryGetValue(key, lines) Then Return Nothing
Dim result As New List(Of String)()
Using sr As New StreamReader(FilePath)
For Each position As Long In lines
sr.BaseStream.Seek(position, SeekOrigin.Begin)
result.Add(sr.ReadLine())
Next position
End Using
Return Result
End Function
End Class
'Somewhere public, when your application starts up:
Public Index As New MyFileIndexer("G:\USER\FOLDER\tester.txt")
Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
Dim lines As List(Of String) = Nothing
Try
lines = Index.Search(TextBox1.Text)
Catch
'Do something here
End Try
If lines IsNot Nothing Then
Using sw As New StreamWriter($"G:\USER\{TextBox1.Text}.txt")
For Each line As String in lines
sw.WriteLine(line)
Next
End Using
End If
End Sub
And for fun, here's a generic version of the class that lets you supply your own key selector function to index any file that stores a key with each line, which I could see being generally useful for, say, larger csv data sets.
Public Class MyFileIndexer(Of TKey)
Private initialCapacity As Integer = 1
Private Property FilePath As String
Private Index As Dictionary(Of TKey, List(Of Long))
Private GetKey As Func(Of String, TKey)
Public Sub New(filePath As String, Func(Of String, TKey) keySelector)
Me.FilePath = filePath
Me.GetKey = keySelector
RebuildIndex()
End Sub
Public Sub RebuildIndex()
Index = New Dictionary(Of TKey, List(Of Long))()
Using sr As New StreamReader(FilePath)
Dim Line As String = sr.ReadLine()
Dim position As Long = 0
While Line IsNot Nothing
Dim key As TKey = GetKey(Line)
Dim item As List(Of Long)
If Not Index.TryGetValue(key, item) Then
item = New List(Of Long)(initialCapacity)
Index.Add(key, item)
End If
item.Add(position)
'Prep for next line
position = sr.BaseStream.Position
Line = sr.ReadLine()
End While
End Using
End Sub
Public Function Search(key As TKey) As List(Of String)
Dim lines As List(Of Long)
If Not Index.TryGetValue(key, lines) Then Return Nothing
Dim result As New List(Of String)()
Using sr As New StreamReader(FilePath)
For Each position As Long In lines
sr.BaseStream.Seek(position, SeekOrigin.Begin)
result.Add(sr.ReadLine())
Next position
End Using
Return Result
End Function
End Class
listy
when you only need a single line? Get the input from the user, open the file, and start reading one line at at time and trying the match to the input. If it doesn't match, throw it away and move on to the next line. If it matches, you've already got the line, and you just stop reading the file.List(Of String)
to use as the enumerable source, use the System.IO.File.ReadLines Method that will create string iterator. i.e.Dim result = From n In System.IO.File.ReadLines("G:\USER\FOLDER\tester.txt")
.