Locked learning resources

Join us and get access to thousands of tutorials and a community of expert Pythonistas.

Unlock This Lesson

Locked learning resources

This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.

Unlock This Lesson

How Context Managers Work, Step by Step

After writing your first own context manager, you’ll walk through the process, which is executed when using a context manager.

00:00 All right, so I want to talk a little bit more about the steps that Python takes behind the scenes for this example to actually work. You can see here, we defined this ManagedFile class, and we’ve got this constructor here that just assigns the name, it remembers the name from the file that we want to create.

00:16 It doesn’t actually open the file until the .__enter__() method is called. And that’s kind of how you want to structure your context managers, right?

00:24 So that, really, the resource gets acquired when the .__enter__() method is called and it gets released when the .__exit__() method is called. And then the .__exit__() method also takes additional parameters that will tell you about some exception that might’ve happened, in case you want to inspect that, log that, or do something with that.

00:40 I’m not doing that here—I’m just checking like, “Hey, did we actually open anything?” If so, then we’re going to close the file. And now, obviously, with this example you need to remember that this is sort of a useless wrapper around the open() function here, because the open() function already pretty much does that when it functions as a context manager, right? So this is merely an example to illustrate that, but you could imagine this would be some other kind of resource, like a database connection. So now, what’s going on here in this example?

01:10 Because when we create an instance of ManagedFile, we’re not actually immediately calling the .__enter__() method. So I could also do something like this—I’m going to call this mf, which is not an ideal name, but for this example

01:26 it’ll work. So, I’m creating

01:30 this ManagedFile object. And up until now,

01:35 we didn’t actually call the .__enter__() method, right? You can see that here with this exception, because the object doesn’t have a .file attribute yet—we didn’t assign that yet, because we didn’t call .__enter__() yet—or, Python didn’t call .__enter__() yet. Now, when I go ahead and do with mf, and then I can say something like with mf as the_file,

02:00 then I’m going to enter this context where the resource is acquired, and now I would have to go the_file.write('hello.txt'). And we can talk about what’s going on here, right?

02:13 So, what you can see here with this statement, with mf as the_file—behind the scenes this is going to call .__enter__(), and then it’s going to assign the return value—so, this is going to return the actual file—

02:26 it’s going to assign that to this name here, the_file. And then within the context of this ManagedFile I can use the_file to actually write to the file, right?

02:37 Because this was the actual file object that I needed to write to the file. Then, when we leave the context—like, right after I’m making this call here—we’re going back one indentation level, we’re leaving this context. Then immediately Python is going to call the .__exit__() method, or the dunder exit method.

02:55 All right. So, I hope that gives you a better idea of how these context managers actually work behind the scenes.

Avatar image for blu

blu on June 13, 2019

What happens when nesting with statements for the same object? Let’s say I have an instance foo of class Bar, which has a context manager defined

class Bar:

    def __init__(self):
        my_resource = None

    def __enter__(self):
        my_resource = allocate_resource()

    def __exit__(self, ...):
        free_resource(my_resource)

what happens if I nest WITH statements like this:

foo = Bar()

with foo:
    with foo:
        with foo:
            do_something

Does Python handle that for me? I.e. will it only call enter upon the first WITH statement and exit when exiting from the same level?

If not, how should I handle this myself? If I have enter check whetehr the resource is already allocated and only allocate it, if not, it will be released upon exiting the innermost level, so that’s not an option, I believe. Alternatively, I could use a nesting counter:

def __init__(self):
    self.my_resource = None
    self.counter = 0

def __enter__(self):
    if self.counter == 0:
        self.my_resource = allocate_resource()
    self.counter += 1

def __exit__(self, ...):
    if self.counter == 1: 
        free_resource(self.my_resource)
    self.counter -= 1

How will that work if an exception occurs? Will exit be called the necessary number of times, so that the resource is eventually freed up?

Avatar image for Dan Bader

Dan Bader RP Team on Dec. 21, 2020

Does Python handle that for me? I.e. will it only call enter upon the first WITH statement and exit when exiting from the same level?

Yep, nested context managers work as expected and Python takes care of calling __enter__ and __exit__ at the appropriate time.

Here’s an example from my Python Tricks book:

class Indenter:
    def __init__(self):
        self.level = 0

    def __enter__(self):
        self.level += 1
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.level -= 1

    def print(self, text):
        print(' ' * self.level + text)
with Indenter() as indent:
    indent.print('hi!')
    with indent:
        indent.print('hello')
        with indent:
            indent.print('bonjour')
    indent.print('hey')

Output:

hi!
    hello
        bonjour
hey
Avatar image for Dan Bader

Dan Bader RP Team on Dec. 21, 2020

Quick note since I just got an email about this being confusing – at the end of the video I’m showing the following code example:

mf = ManagedFile("hello.txt")
with mf as the_file:
    the_file.write("hello.txt")

This will create a file named hello.txt and then write the text hello.txt to it.

The example would be clearer if I had used Hello, World! as the sample file contents, like in the previous examples:

mf = ManagedFile("hello.txt")
with mf as the_file:
    the_file.write("Hello, World!")

I hope that clears up what determines the filename and the file contents (what gets written to the file) in the example :)

Avatar image for akolal

akolal on Dec. 22, 2020

Yes it does. Thank you.

Avatar image for Mohamed Awnallah

Mohamed Awnallah on June 28, 2022

Got the idea behind context managers and how they mostly work but I’ve a specific concern about scoping in “with block”. You could access all the variables declared inside with block later. How does that really happen ?

Become a Member to join the conversation.