[FRPythoneers] Sockets and threads in HTTP app
mgushee at havenrock.com
Mon Apr 14 19:11:35 MDT 2003
I am developing a tool for HTTP diagnostics. Briefly, it has a Tkinter
GUI that displays HTTP request and response messages; from the client's
point of view it is a proxy server. So of course the application has a
server component, and I am currently using a subclass of BaseHTTPServer
from the standard library.
Now, when a request comes in I need to do two things with it: display it
in the GUI and forward it to the destination server. Let's see, this
gets a bit convoluted, but the server component (and thus the request
handler) is running in a worker thread, and when it gets the request it
is supposed to process it into an easily-manageable data structure which
it then writes to a queue (a Queue.Queue instance, that is). For the
GUI, I wrote a customized mainloop that polls the Queue and displays
whatever data comes through.
Well, the problem with this is that, when the request handler writes to
the queue, I want the body of the request to be in the form of a list of
strings. That means, one way or another, reading the entire contents of
self.rfile. But self.rfile is an open socket, and apparently it doesn't
get closed as long as a request is in progress. But I can't simply
forward the request immediately, because I intend for the user to be
able to edit headers and/or the body in the GUI before passing it on.
So I came up with the following kludge, which obtains all available text
def clone_endless_file(self, f, wait=0.5):
"""Given a non-terminating file object, e.g. an open socket,
returns a StringIO containing all available contents."""
q = Queue.Queue(0)
thr = threading.Thread(target=lambda ff=f, qq=q, re=self._read_endless:
# The pause is to (more-or-less) ensure that the queue gets loaded
# up before we try to read it.
new_file = StringIO()
line = q.get(0)
def _read_endless(self, f, q):
line = f.readline()
line = f.readline()
This works, but I have a feeling I'm asking for trouble down the road. I
don't know a whole lot about threads, but I imagine this will lead to an
accumulation of blocked threads. Apparently, too, it prevents the socket
from *ever* getting closed--at least, when I test it with a browser, the
browser continues trying to pull something even after the whole response
has been delivered. Given that this is a developer's tool, I'm not sure
that that matters--the main thing is to be able to view the request and
response, and it's working in that sense. I just have a bad feeling
about this technique.
Can anyone suggest a better way to process the request data (preferably
without getting into the bowels of the HTTP library, but I'll do that if
it's the only good solution)?
Matt Gushee When a nation follows the Way,
Englewood, Colorado, USA Horses bear manure through
mgushee at havenrock.com its fields;
http://www.havenrock.com/ When a nation ignores the Way,
Horses bear soldiers through
--Lao Tzu (Peter Merel, trans.)
More information about the FRPythoneers