[FRPythoneers] Sockets and threads in HTTP app

Matt Gushee mgushee at havenrock.com
Mon Apr 14 19:11:35 MDT 2003

Hello all--

I am developing a tool for HTTP diagnostics. Briefly, it has a Tkinter
GUI that displays HTTP request and response messages; from the client's
point of view it is a proxy server. So of course the application has a
server component, and I am currently using a subclass of BaseHTTPServer
from the standard library.

Now, when a request comes in I need to do two things with it: display it
in the GUI and forward it to the destination server. Let's see, this
gets a bit convoluted, but the server component (and thus the request
handler) is running in a worker thread, and when it gets the request it
is supposed to process it into an easily-manageable data structure which
it then writes to a queue (a Queue.Queue instance, that is). For the
GUI, I wrote a customized mainloop that polls the Queue and displays
whatever data comes through.

Well, the problem with this is that, when the request handler writes to
the queue, I want the body of the request to be in the form of a list of
strings. That means, one way or another, reading the entire contents of
self.rfile. But self.rfile is an open socket, and apparently it doesn't
get closed as long as a request is in progress. But I can't simply
forward the request immediately, because I intend for the user to be
able to edit headers and/or the body in the GUI before passing it on.

So I came up with the following kludge, which obtains all available text
in self.rfile:

    def clone_endless_file(self, f, wait=0.5):
        """Given a non-terminating file object, e.g. an open socket,
returns a StringIO containing all available contents."""
        q = Queue.Queue(0)
        thr = threading.Thread(target=lambda ff=f, qq=q, re=self._read_endless:
                                                   re(ff, qq))
        # The pause is to (more-or-less) ensure that the queue gets loaded
        # up before we try to read it.
        new_file = StringIO()
        while 1:
                line = q.get(0)
            except Queue.Empty:
        return new_file

    def _read_endless(self, f, q):
        line = f.readline()
        while line:
            line = f.readline()

This works, but I have a feeling I'm asking for trouble down the road. I
don't know a whole lot about threads, but I imagine this will lead to an
accumulation of blocked threads. Apparently, too, it prevents the socket
from *ever* getting closed--at least, when I test it with a browser, the
browser continues trying to pull something even after the whole response
has been delivered. Given that this is a developer's tool, I'm not sure
that that matters--the main thing is to be able to view the request and
response, and it's working in that sense. I just have a bad feeling
about this technique.

Can anyone suggest a better way to process the request data (preferably
without getting into the bowels of the HTTP library, but I'll do that if
it's the only good solution)?

Matt Gushee                 When a nation follows the Way,
Englewood, Colorado, USA    Horses bear manure through
mgushee at havenrock.com           its fields;
http://www.havenrock.com/   When a nation ignores the Way,
                            Horses bear soldiers through
                                its streets.
                            --Lao Tzu (Peter Merel, trans.)

More information about the FRPythoneers mailing list