18

In the interest of not rewriting an open source library, I want to treat a string of text as a file in python 3.

Suppose I have the file contents as a string:

not_a_file = 'there is a lot of blah blah in this so-called file'

I want to treat this variable, i.e. the contents of a file, as a path-like object that way I can use it in python's open() function.

Here is a simple example which shows my dilemma:

not_a_file = 'there is a lot of blah blah in this so-called file'
file_ptr = open(not_a_file, 'r')

Clearly the example doesn't work because not_a_file is not a path-like object. I don't want to write a file nor create any temporary directories for portability purposes.

With that said, I need is to solve this mystery:

not_a_file = 'there is a lot of blah blah in this so-called file'
... Something goes here ... 
file_ptr = open(also_not_a_file, 'r') 

What I've Tried So Far

  1. I've looked into StringIO and tried using that as a path-like object and no dice: import StringIO output = StringIO.StringIO() output.write('First line.\n') file_ptr = open(output,'r') Well this doesn't work because StringIO isn't a path-like object.

  2. I've tried tempfile in a similar fashion with no success. import tempfile tp = tempfile.TemporaryFile() tp.write(b'there is a lot of blah blah in this so-called file') open(tp,'r')

  3. Finally I tried mmap to see if I can write the string into memory and then open the memory pointer with open with no success.

Any help is appreciated! :-)

Edit 1: What I'm thinking of to possibly solve problem

So pathlib.PurePath can work with open() if PurePath is initialized to a file. Perhaps I can create an instance of a class that inherits PurePath and when read by open(), it reads my string. Let me give an example:

from pathlib import PurePath

not_a_file = 'there is a lot of blah blah in this so-called file'
class Magic(PurePath):
    def __init__(self, string_input):
        self.file_content = string_input
        PurePath.__init__(self, 'Something magical goes here')
    #some more magic happens in this class

also_not_a_file = Magic(not_a_file)
fp = open(also_not_a_file,'r')
print(fp.readlines()) # 'there is a lot of blah blah in this so-called file'
4
  • Why do you want to use the open command? output.write("line 1\n"); output.seek(0); file_ptr=output Jun 6, 2017 at 4:07
  • @clockwatcher so in a different library that I'm going to use, there is this open command expecting the usual file input. For my purposes and portability I need to somehow pass the string into the open somehow.
    – torrho
    Jun 6, 2017 at 4:27
  • Is there a way to use pathlib.PurePath with a string to create an object that will read the string? open(pathlib.PurePath('path/to/filename.ext'),'r') is valid python. Can I incorporate PurePath and my string into a class which when passed into open() reads my string?
    – torrho
    Jun 6, 2017 at 4:54
  • Does this answer your question? How do I wrap a string in a file in Python?
    – Kermit
    Mar 15, 2022 at 15:48

5 Answers 5

25

StringIO returns an StringIO object, it's almost equivalent to the file object returned by the open statement. So basically, you can use the StringIO in place of the open statement.

# from StringIO import StringIO  # for Python 2, old
from io import StringIO
with StringIO('there is a lot of blah blah in this so-called file') as f:
    print(f.read())

Output:

there is a lot of blah blah in this so-called file
4
  • 3
    This won't suffice. I would need a path like object also_not_a_string to be a valid input into open(). This object, when read via open() should have the 'there is a lot of blah blah in this so-called file' string.
    – torrho
    Jun 6, 2017 at 4:24
  • So you're basically wanting to create a real existing file onto storage? Since open() must take an openable file name/path as an argument. You can use tempfile, but that's just like creating a file, use it, and delete it later. Is that what you want instead?
    – Taku
    Jun 6, 2017 at 4:26
  • See my revised edit of #2. That is an example of tempfile which does not work. Again, I need a path-like object when read via open(), yields the string 'there is a lot of blah blah in this so-called file'
    – torrho
    Jun 6, 2017 at 4:36
  • Sometimes (rarely) this simply is not possible. TensorFlow is one such rare problem. You must specify a filename and cannot send in a file object/pointer. Aug 3, 2021 at 6:11
10

You can create a temporary file and pass its name to open:

On Unix:

tp = tempfile.NamedTemporaryFile()
tp.write(b'there is a lot of blah blah blah in this so-called file')
tp.flush()
open(tp.name, 'r')

On Windows, you need to close the temporary file before it can be opened:

tp = tempfile.NamedTemporaryFile(delete=False)
tp.write(b'there is a lot of blah blah blah in this so-called file')
tp.close()
open(tp.name, 'r')

You then become responsible for deleting the file once you're done using it.

3
  • Did you pass tp.name and not tp to open? Jun 6, 2017 at 5:13
  • The unix example isn't working for me. Is NamedTemporaryFile() a path-like object? I get: Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: expected str, bytes or os.PathLike object, not _TemporaryFileWrapper
    – torrho
    Jun 6, 2017 at 5:13
  • I'm still not able to read the contents of the object properly. open(tp.name, 'r').readlines() just gives me ''
    – torrho
    Jun 6, 2017 at 5:18
3

With what I can tell from your comments and recent edits, you want a file that can be opened using the open statement. (I'll leave my other answer be since it's the more correct approach to this type of question)

You can use tempfile to solve your problem, it basically is doing this: create your file, do stuff to your file, then delete your file upon closure.

import os
from tempfile import NamedTemporaryFile

f = NamedTemporaryFile(mode='w+', delete=False)
f.write("there is a lot of blah blah in this so-called file")
f.close()
with open(f.name, "r") as new_f:
    print(new_f.read())

os.unlink(f.name) # delete the file after
0

the other answers didn't work for me, but I managed to figure it out.

When using Python 3, you'll want to use io package.

    import io
    with io.StringIO("some initial text data") as f:
        # now you can do things with f as if it was an opened file.
        function_that_requires_a_Fileobject_as_argument(f)
0

You can overwrite what open means for a particular function - similarly to what unittest.mock.patch does. However to do so, you have to keep track of the module where your target function is located. To make sure that later on open is restored, you can take advantage of decorators or context managers:

class FileToStringIOPatch:
    def __init__(self, module, new_reader=io.StringIO, **kwargs):
        self.new_reader = new_reader
        self.globals_dict = vars(module)
        self.old_open = self.globals_dict.get('open', open)
        self.kwargs = kwargs

    def __enter__(self):  # initialize context manager - overwrite `open`
        def new_open(file, mode='r', buffering=None, encoding=None, errors=None, newline=None, closefd=True):
            return self.new_reader(file, **self.kwargs)
        self.globals_dict['open'] = new_open

    def __exit__(self, exc_type, exc_val, exc_tb): # close ctx manager - restore `open`
        self.globals_dict['open'] = self.old_open

Example usage when "mocked" function declaration is in the same module as where it's called:

def read(path):
    with open(path, 'r') as f:
        print(f.read())

# ...
import sys
module_reference = sys.modules[__name__]

string = "siala_baba_mak"
with FileToStringIOPatch(module_reference)
    read(string)  # prints siala_baba_mak
read('path/to/file.txt')  # outside of context manager it still works as usual

Example usage when "mocked" function declaration is in another module

def read(path):
    with open(path, 'r') as f:
        print(f.read())

# ...

from xyz import read
import xyz
module_reference = xyz

string = "siala_baba_mak"
with FileToStringIOPatch(module_reference)
    read(string)  # prints siala_baba_mak
read('path/to/file.txt')  # outside of context manager it still works as usual

You can even provide custom readers to add additional functionalities e.g. reading from AWS S3 bucket.

from s3path import S3Path
path = S3Path("s3://path/to/file.txt")
with FileToStringIOPatch(module_reference, new_reader=path.open):
    read_(path)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.