Little help with closing socket.

Hi, im sending bytes over a socket form pocket PC to desktop Pc. Sending is working ok.
Now I wish to close the socket on PocketPC side, so I use the following on pocket PC:

handler.Shutdown(SocketShutdown.Both);
handler.Close();
handler = null;

But on the desktop side the socket is not closing
Heres what I have on desktop side:

private void startListenThread()
{
writeLine("Starting Service...");

while (true)
{
allDone.Reset();
socket.BeginAccept(
new AsyncCallback(acceptCallback),
socket);

allDone.WaitOne();
}
}

public void acceptCallback(IAsyncResult ar)
{
allDone.Set();

// receive
Socket listener = (Socket)ar.AsyncState;
Socket handler = listener.EndAccept(ar);

handlers.Add(handler); // handlers is an arrayList
setToolStripStatusLabel1(handlers.Count + " Pocket PC's Connected");

byte[] mBytes = new byte[5];
while (handler != null && handler.Connected)
{
try
{
handler.Receive(mBytes);
// do whatever with bytes

}

catch
{
handler.Shutdown(SocketShutdown.Both);
handler.Close();
handlers.Remove(handler);
handler = null;
setToolStripStatusLabel1(handlers.Count + " Pocket PC's Connected");
}

}
if (handler != null)
{
handler.Shutdown(SocketShutdown.Both);
handler.Close();
handlers.Remove(handler);
handler = null;
setToolStripStatusLabel1(handlers.Count + " Pocket PC's Connected");

}
}

What should I be doing on desktop so the socket cleans up correctly and not just locks up

Thanks for your time.

Paul.