Join us and get access to thousands of tutorials and a community of expert Pythonistas.
This lesson is for members only. Join us and get access to thousands of tutorials and a community of expert Pythonistas.
The Producer-Consumer Pipeline: Part 1
In this course, you’re going to write a producer-consumer pipeline. To understand what a producer-consumer pipeline is, image a conveyor belt. On the left you have a producer, and on the right you have a consumer. The producer is going to produce some piece of data and is going to put it on the conveyor belt. When the consumer receives this data, it will do something with it.
00:00 All right. So as you enter the end of this course, you’re going to build a producer-consumer pipeline. And before you start actually writing code, I just want to make sure you understand what a producer-consumer pipeline is.
00:16 Imagine you have like a conveyor belt, or a pipeline, and on the left you have a producer, and on the right a consumer. The producer is going to produce some type of object, some piece of data, and it’s going to put this data into the pipeline.
00:34 It’s going to continue putting things into the pipeline.
00:40 When it reaches the consumer, the consumer will then read this object, or this data, from the pipeline and do something with it. Maybe it’s going to send this data to another server, or it’s going to write this data to a database or a hard disk—it’s going to do something with it.
00:59 It consumes the data from the pipeline, and then it does whatever it does with it. In the meantime, the producer is still producing and putting more objects, more data, into the pipeline. So, this is what you’re going to create—some mechanism like this, which is used all over all kinds of software.
01:20 You might see it as a message queue or a queueing system. Data pipelines doing ETL work with various kinds of data use pipelines to do stuff like this. You are going to learn the skills to create a multi-threaded and thread-safe consumer-producer pipeline.
Become a Member to join the conversation.
Alvaro Formoso on April 2, 2020
Thank you for this useful course, Lee! One question I had after seeing all this videos, is if I am able to use ThreadPools outside if name == ‘main’ module.
For example, in a method of a class to make (for example) 3 or 4 read_sql querys faster.
Thanks!