Coonecting socket-based applications in a peer-to-peer local network

[Enquiring Mind]

Hi,

I have just written a socket-based client-server pair of programs, called
ProjectServerA and ProjectClientA. They work fine on the same computer using
'localhost' for the RemoteHost of the client socket component, and '1002'
for the port number. I have now tried running the 2 programs on 2 computers
in my peer-to-peer home network, but cannot get the client program to
connect to the server program.

Both computers run under Windows XP Pro SP2. The computers are called
'Desktop01' and 'Laptop01', and use dynamic IP address allocation. I have
created an exception for port 1002 in the firewall of computer Desktop01,
and later did the same in the firewall of computer Laptop01 (though I am not
sure why this exception should be required in the client computer). I then
launch program ProjectServerA from the Shared folder, and start it
listening.

On computer Laptop01 if I type 'ping Desktop01' in the command prompt box, I
get a positive response, with an IP address for the Desktop01 computer, for
example '169.254.71.84'. If I launch program ProjectClientA on computer
Laptop01 and enter for the remote host name either 'Desktop01' or the IP
address given by ping, and try to connect, the connection fails. Similarly
if I type 'telnet Desktop01 1002' in the command prompt box, I get a
negative response.

Any ideas from network administration experts what further needs to be done
to get these 2 applications to talk to each other?

TIA

EM

[Enquiring Mind]

"Enquiring Mind" <***@nospam.btopenworld.com> wrote in message news:48846fb2$***@newsgroups.borland.com...

PS:

1. Since my previous post I have changed the server port number to 8080,
having learned that port numbers below 1024 are reserved.

2. The computer that I am using as a server has more than one network
interface card - one is an Ethernet interface, the other is a wireless
network interface. I have read in a recent post on the same subject that
there can be problems when trying to convert from a hostname to an IP
address when the there are multiple network interfaces in a computer. So I
have tried using the IP address of each when trying to connect - without
success, however.

So I am still unable to connect to the server from the client on the laptop
<sigh>!

Some considerable time ago I developed a demonstration DCOM client-server
application pair, and on that occasion used as the RemoteMachine parameter
passed to the CreateRemoteCOMObject function the same computer name I am now
using . On that occasion, after the difficult security settings and server
installation settings had been resolved, I seem to remember that the client
was able to connect to the server successfully.

It seems to me that the various onerous security settings that had to be
made for the DCOM server should not be necessary for a plain socket server -
presumably because the socket sever cannot be activated by a remote client
application, but only by an authorized user of the server computer. It
should therefore pose lower security risks. I am right in thinking this?

[Remy Lebeau (TeamB)]

Post by Enquiring Mind
Both computers run under Windows XP Pro SP2.

The firewall on the server PC is likely blocking the connection.

Post by Enquiring Mind
The computers are called 'Desktop01' and 'Laptop01', and use
dynamic IP address allocation.

Have you tried using static IPs yet?

Post by Enquiring Mind
I have created an exception for port 1002 in the firewall of computer
Desktop01

Have you tried disabling the firewall completely?

Post by Enquiring Mind
and later did the same in the firewall of computer Laptop01 (though
I am not sure why this exception should be required in the client
computer).

It is not.

Post by Enquiring Mind
I then launch program ProjectServerA from the Shared folder, and
start it listening.

Did you verify that it is actually listening properly? Use netstat.exe to
make sure port 1002 is actually in the LISTENING state.

Post by Enquiring Mind
On computer Laptop01 if I type 'ping Desktop01' in the command
prompt box, I get a positive response, with an IP address for the
Desktop01 computer, for example '169.254.71.84'. If I launch program
ProjectClientA on computer Laptop01 and enter for the remote
host name either 'Desktop01' or the IP address given by ping, and
try to connect, the connection fails.

What kind of error are you getting exactly? Pinging will only tell you
whether the server PC is physically reachable on the network. It will not
tell you whether a client can connect to any listening sockets on the
server.

Post by Enquiring Mind
Similarly if I type 'telnet Desktop01 1002' in the command prompt
box, I get a negative response.

All the more reason to suspect either the firewall is blocking access to
that port, or the server is not actually listening properly.


Gambit

[Enquiring Mind]

"Remy Lebeau (TeamB)" <***@no.spam.com> wrote in message news:4884be65$***@newsgroups.borland.com...
Thanks for your help. I have now found the cause the problem - the server
program was still assigning to the server socket LocalHost property the
value 'localhost' used for development rather than the IP address of the
server computer's network connection as exposed to the rest of the network.
So the problem was due to a programming rather than a configuration error.

Post by Remy Lebeau (TeamB)
The firewall on the server PC is likely blocking the connection.

After opening an exception in the firewall for the port assigned to the
server socket this possibility was eliminated.

Post by Remy Lebeau (TeamB)

Post by Enquiring Mind
The computers are called 'Desktop01' and 'Laptop01', and use
dynamic IP address allocation.

Have you tried using static IPs yet?

No, but it doesn't seem to be mandatory to use static IPs.

Post by Remy Lebeau (TeamB)

Post by Enquiring Mind
I have created an exception for port 1002 in the firewall of computer
Desktop01

Have you tried disabling the firewall completely?

No (I value the security of my computer too much !), but I have discovered
that I can get away without doing this if appropriate exceptions are created
in the firewall.

Post by Remy Lebeau (TeamB)

Post by Enquiring Mind
and later did the same in the firewall of computer Laptop01 (though
I am not sure why this exception should be required in the client
computer).

It is not.

That's useful to know - many thanks.

Post by Remy Lebeau (TeamB)

Post by Enquiring Mind
I then launch program ProjectServerA from the Shared folder, and
start it listening.

Did you verify that it is actually listening properly? Use netstat.exe to
make sure port 1002 is actually in the LISTENING state.

When I posted my query I didn't know how to do this. I have since discovered
that it can be done using the 'netstat' command, and with this tool I was
able to trace the cause of the problem.

Post by Remy Lebeau (TeamB)
What kind of error are you getting exactly? Pinging will only tell you
whether the server PC is physically reachable on the network. It will not
tell you whether a client can connect to any listening sockets on the
server.

What commandline command can be used to determine whether or not a port is
active on a remote computer?

A couple more questions about working with network connections:

1. A computer can have more than one network connection, each having a
different IP address. My computer has a wired Ethernet connection and a
wireless connection. The method TIPSocket.LookUpHostAddr(Histname) returns
only one IP address rather than a list. So how can a complete list of
available IP addresses be obtained?

2. The TIPSocket.Open method doesn't have any parameters or a return value.
I would have expected at least a Timeout parameter and a result parameter to
indicate whether the method terminated because a connection was successfully
made or because the Timeout limit was reached. How can a timeout be
specified for a client socket connection attempt?

3. Does the TIpSocket.Open method run in a separate thread, given that it
could take 'forever'?

3. I have been told that to find out whether a remote connection is active,
one can use the API functions InternetGetConnectedState and
InternetCheckConnection. In which Delphi unit are they declared? Also, they
don't appear in the SDK help index - so which Windows helpfile covers them?

Regards,

EM

[Remy Lebeau (TeamB)]

the server program was still assigning to the server socket LocalHost
property the value 'localhost' used for development rather than the IP
address of the server computer's network connection as exposed to
the rest of the network.

That would do it.

What commandline command can be used to determine whether or
not a port is active on a remote computer?

There is no build-in command for that. You need to use a port scanner.
There are plenty of third-party scanners available.

The method TIPSocket.LookUpHostAddr(Histname) returns only one
IP address rather than a list. So how can a complete list of available
IP addresses be obtained?

LookupHostAddr() retreives all of the IPs internally, but only returns the
first one in the list to you. The list itself is not in any particular
order, either. The OS determines the ordering on its own accord.

TIpSocket does not expose access to the entire list. You will have to call
the socket API gethostbyname() function directly in order to do that.

The TIPSocket.Open method doesn't have any parameters or a return value.

It doesn't need any. All of its input values are implemented as properties,
and it will set the Active and Connected properties to True/False depending
on the output.

How can a timeout be specified for a client socket connection attempt?

You need to open the socket in non-blocking mode and then call Select()
manually in order to handle such a timeout, ie:

var
bConnected: Boolean;
begin
TcpClient1.BlockMode := bmNonBlocking;
// set other properties as needed...
TcpClient1.Open;
if TcpClient1.Active and (not TcpClient1.Connected) then
begin
bConnected := False;
TcpClient.Select(nil, @bConnected, nil, 5000);
if not bConnected then
TcpClient1.Close;
end;
end;

Does the TIpSocket.Open method run in a separate thread, given
that it could take 'forever'?

No.

I have been told that to find out whether a remote connection is
active, one can use the API functions InternetGetConnectedState
and InternetCheckConnection.

Those are not reliable ways to test for an available Internet connection.
Even Microsoft says as much. There are too many ways a PC can be connected
to the Internet nowadays (LAN, proxy, Wireless, etc). You are best off
simply not trying to determine that status manually. Just attempt to
connect normally and check for any errors that may occur.

In which Delphi unit are they declared?

WinInet.

Also, they don't appear in the SDK help index

Then you are using an old SDK reference.

so which Windows helpfile covers them?

I suggest referring to MSDN online, or at least downloading the latest SDK
help from it.


Gambit

[Enquiring Mind]

Post by Remy Lebeau (TeamB)

How can a timeout be specified for a client socket connection attempt?

You need to open the socket in non-blocking mode and then call Select()
<snip>

Thanks for the code. Very useful, because I had only a vague awareness of
the existence of a Select method. I presume it is a wrapper to the API
function of the same name.

I don't fully understand why the second parameter of the Select method,
called WriteReady, indicates that a connection has been made.

I also tried the following alternative approaches, for a client socket in
blocking mode:

procedure TForm1.ButtonConnectClick(Sender: TObject);
var
WaitResult: Boolean;
begin
with TcpClient1 do
begin
{.}RemoteHost:= EditRemoteIPAddress.Text;
{.}RemotePort:= EditRemotePort.Text;
Screen.Cursor:= crHourglass;
{$ifdef Option5}
with Timer1 do
begin
{.}Interval:= FConnectTimeout;
{.}Enabled:= True;
end;
{$endif}
{.}Open; {Attempt to connect}
{$ifdef Option6}
WaitResult:= {.}WaitForData(FConnectTimeout);
if not WaitResult then
HandleFailureToConnect;
{$endif}
end;
end;

procedure TForm1.Timer1Timer(Sender: TObject);
begin
if not TcpClient1.Connected then
HandleFailureToConnect;
Timer1.Enabled:= False;
end;

procedure TForm1.HandleFailureToConnect;
begin
Beep;
MessageDlg('Unable to connect to remote socket', mtError, [mbOK], 0);
TcpClient1.Close;
Screen.Cursor:= crDefault;
end;

The Timer approach seems to work OK, while the WaitFor approach seems to
work in a slightly erratic manner. Have you any views on the relative merits
of WaitForData versus Select?

Regards,

EM

[Remy Lebeau (TeamB)]

Post by Enquiring Mind
Thanks for the code. Very useful, because I had only a vague
awareness of the existence of a Select method. I presume it is
a wrapper to the API function of the same name.

Yes.

Post by Enquiring Mind
I don't fully understand why the second parameter of the Select
method, called WriteReady, indicates that a connection has
been made.

Have you read the documentation for the API's connect() and select()
functions yet?

--- connect() ---

With a nonblocking socket, the connection attempt cannot be completed
immediately. In this case, connect() will return SOCKET_ERROR, and
WSAGetLastError() will return WSAEWOULDBLOCK. In this case, the application
can:

1. Use select() to determine the completion of the connection request by
checking if the socket is writeable


--- select() ---

The parameter "writefds" identifies those sockets which are to be
checked for writability. If a socket is connecting (nonblocking),
writability means that the connection establishment successfully completed.

Post by Enquiring Mind
I also tried the following alternative approaches, for a client

That won't work for a blocking socket. TTimer is a message-based timer. As
soon as you call Open(), the calling thread will become blocked, preventing
it from processing any Timer messages until Open() exits and execution
returns to the thread's message loop. The only way to implement a connect
timeout on a blocking socket is to disconnect the socket from a different
thread if the desired timeout elapses.


Gambit

[Enquiring Mind]

Post by Remy Lebeau (TeamB)
Have you read the documentation for the API's connect() and select()
functions yet?

I have glanced at the API function definitions, but have yet to read the
detailed explanations. Hope to do so soon! So WriteReady means that the
socket connection has been established and is ready to be written to.

With regard to my query about TBaseSocket.WaitForData versus
TBaseSocket.Select, I have now looked at the WaitForData code in the Sockets
unit, and see that it just calls Select(@ReadReady, nil, @ExceptFlag,
TimeOut). It therefore seems to do something similar to your code, but
doesn't change the client socket's blocking mode and it waits for the socket
to be ready for reading rather than for writing. I interpret ReadReady to
mean that the socket connection has been established, and contains incoming
data waiting to be read. However WaitForData seems to terminate as soon a
socket connection is established, even if there is no data waiting to be
read.

I tried to implement your code as follows:

procedure TForm1.ButtonConnectClick(Sender: TObject);
var
ConnectedStatus: Boolean;
begin
with TcpClient1 do
begin
{.}BlockMode := bmNonBlocking;
{.}RemoteHost:= EditRemoteIPAddress.Text;
{.}RemotePort:= EditRemotePort.Text;
Screen.Cursor:= crHourglass;
{.}Open;
if {.}Active and (not {.}Connected) then
begin
ConnectedStatus:= False;
{.}Select(nil, @ConnectedStatus, nil, FConnectTimeout);
if not ConnectedStatus then
HandleFailureToConnect
else
{Call OnConnect event handler:}
TcpClient1Connect(TcpClient1); {*}
end;
end;
end;

I have encountered the following problems with the above code:

1) Without the addition of the manual call of the OnConnect event handler
marked by {*}, the TcpClient1.OnConnect event handler is never dispatched.
The OnConnect event handler updates the User Interface to show that the
connection is made, and initiates the socket reading thread. Thus, even when
the connection succeeds, the client application hangs, because OnConnect is
never dispatched.

2) The client application is designed to read data received on the client
socket in blocking mode within a loop in a secondary thread. Now that the
socket is in non-blocking mode, the code to read data received on the
client socket no longer works. I tried to change the blocking mode back to
bmBlocking after the connection is successfully established, but this has
the effect of terminating the connection.

Post by Remy Lebeau (TeamB)
That won't work for a blocking socket. TTimer is a message-based timer.
As soon as you call Open(), the calling thread will become blocked,
preventing it from processing any Timer messages until Open() exits and
execution returns to the thread's message loop. The only way to implement
a connect timeout on a blocking socket is to disconnect the socket from a
different thread if the desired timeout elapses.

Although the client socket is set up as blocking, it does not seem to block
the main thread of the application. When the TTcpSocket.Open method is
executed in the debugger, the cursor moves immediately on to the next
statement, even when the remote socket to which the client is trying to
connect is unreachable. In such a case the application waits indefinitely
with an hourglass cursor, because the OnConnect event handler that resets
the cursor to the default cursor is never fired. However the user interface
continues to respond to repaint messages, in as much as the application's
main form continues to repaint itself when necessary. It looks as if the
WinSock.connect function performs the connection attempt in a separate
thread, but the thread calls the connection attempt in blocking mode.

Thus the main form event loop continues to be able to process a message
received from the TTimer component even when the socket connect operation is
blocked, for example while attempting to connect to an unreachable remote
socket. That may explain why the timer approach does seem to work.

EM

[Enquiring Mind]

However WaitForData seems to terminate as soon a socket connection is
established, even if there is no data waiting to be read.

Correction! When a client successfully connects to the server, the server
sends the client an acknowledgement message, so there is always data waiting
to be read at the client end upon a successful connection.

[Remy Lebeau (TeamB)]

WriteReady means that the socket connection has been established
and is ready to be written to.

Yes.

I interpret ReadReady to mean that the socket connection has
been established, and contains incoming data waiting to be read.

ReadReady cannot be used to check for a successful connect(). Only
WriteReady can be used for that. ReadReady is not valid until after the
connection has already been established beforehand.

However WaitForData seems to terminate as soon a socket
connection is established, even if there is no data waiting to be read.

As it should, because the connection is not valid yet.
You are not taking into account when Open() is able to successfully
establish a connection without delay. Try this:

procedure TForm1.ButtonConnectClick(Sender: TObject);
var
ConnectedStatus: Boolean;
begin
with TcpClient1 do
begin
{.}BlockMode := bmNonBlocking;
{.}RemoteHost:= EditRemoteIPAddress.Text;
{.}RemotePort:= EditRemotePort.Text;
Screen.Cursor:= crHourglass;
{.}Open;
if {.}Active then
begin
ConnectedStatus := {.}Connected;
if (not ConnectedStatus) and (OnError event was not fired)
then
begin
// asynchronous connect, wait for result
{.}Select(nil, @ConnectedStatus, nil, FConnectTimeout);

if ConnectedStatus then TcpClient1Connect(TcpClient1);
end;
if ConnectedStatus then Exit;
end;
HandleFailureToConnect;
end;
end;

1) Without the addition of the manual call of the OnConnect event
handler marked by {*}, the TcpClient1.OnConnect event handler
is never dispatched.

The OnConnect event only works in blocking mode. One of the shortcomings of
TTcpClient (and TTcpServer) is that is does NOT implement any kind of
asynchronous event handling at all (unlike the older TClientSocket [and
TServerSocket] component, which did). If you want to support asynchronous
events, you have to implement them manually, such as by calling
WSAAsyncSelect() or WSAEventSelect() in your own code. As such, TTcpClient
in non-blocking mode has no way of responding to a successful connect that
occurs after Open() has already exited. This is just one of many reasons
why I keep telling people NOT to use TTcpClient (and TTcpServer). It is
simply not implemented very well.

The OnConnect event handler updates the User Interface to show
that the onnection is made, and initiates the socket reading thread.

You don't need to use a separate reading thread when the socket is in
non-blocking mode. If you use WSAAsyncSelect() or WSAEventSelect(), your
code will be notified when data arrives. So all of your code can reside in
the same thread. That is the whole reason why Microsoft invented
non-blocking sockets under Windows in the first place (well, back in the
Win.1 days when preemptive multithreading didn't exist yet, anyway). Pretty
much all other platforms use blocking sockets instead.

2) The client application is designed to read data received on the
client socket in blocking mode within a loop in a secondary thread.
Now that the socket is in non-blocking mode, the code to read
data received on the client socket no longer works.

Correct. Blocking and non-blocking mode require different handling of the
socket activity. You will have to update your code accordingly.

I tried to change the blocking mode back to bmBlocking after
the connection is successfully established, but this has the effect
of terminating the connection.

That is because the BlockMode property setter calls Close() internally. To
change the mode without disconnecting, you will have to call the API
ioctlsocket() directly instead.

Although the client socket is set up as blocking, it does not seem
to block the main thread of the application.

It will if you call Open() in the context of the main thread.

When the TTcpSocket.Open method is executed in the debugger,
the cursor moves immediately on to the next statement

The only way that can happen is if either 1) you are using a non-blocking
socket, or 2) the connection is being fully established, or fails, without
any delay on a blocking socket. Open() has to wait for the API connect()
function to exit. On a blocking socket, connect() waits for the connection
to be fully established or fails. On a non-blocking socket, connect()
always exits immediately and then the socket becomes signaled later on when
the connection succeeds or fails.

However the user interface continues to respond to repaint messages,
in as much as the application's main form continues to repaint itself
when necessary.

Because your code has returned execution flow back to the message loop.

It looks as if the WinSock.connect function performs the connection
attempt in a separate thread

Not necessarily in a separate thread, but yes, it can perform the connection
in the background separately from the calling thread, but only for a
non-blocking socket, not for a blocking socket.

Thus the main form event loop continues to be able to process a message
received from the TTimer component even when the socket connect
operation is blocked

Only for a non-blocking socket, and only when you are not inside of
Select(), as it also blocks and does not process messages.


Gambit

[Enquiring Mind]

"Remy Lebeau (TeamB)" <***@no.spam.com> wrote in message news:***@newsgroups.borland.com...
Thanks for your help.

Post by Remy Lebeau (TeamB)
You are not taking into account when Open() is able to successfully
procedure TForm1.ButtonConnectClick(Sender: TObject);
var
ConnectedStatus: Boolean;
begin
with TcpClient1 do
begin
{.}BlockMode := bmNonBlocking;

I have found that if the above statement is omitted, i.e. the BlockMode is
left unchanged, then the timeout code and the connection works properly. It
looks as if the OS automatically changes the BlockMode to NonBlocking while
a connect operation is in progress, so there's no need to do so manually.

Post by Remy Lebeau (TeamB)

Post by Enquiring Mind
Although the client socket is set up as blocking, it does not seem
to block the main thread of the application.

It will if you call Open() in the context of the main thread.

If one looks at the thread count for the client application in the Task
Manager, one finds that there are 2 threads. This adds credence to the idea
that even if TcpClient1.Open is called from the main thread, the work of
connecting to the remote socket is internally performed in a secondary
thread.

Regards,

EM

[Enquiring Mind]

Post by Enquiring Mind
I have found that if the above statement is omitted, i.e. the BlockMode is
left unchanged, then the timeout code and the connection works properly.
It looks as if the OS automatically changes the BlockMode to NonBlocking
while a connect operation is in progress, so there's no need to do so
manually.

Further to the above, I have found that in my client socket application the
following code to connect to a remote socket works whichever of the 3
wait-for options is selected:

procedure TForm1.ButtonConnectClick(Sender: TObject);
type
TWaitOptions= 1..3;
var
WaitOption: TWaitOptions;

procedure WaitForConnectionToCompleteOption1(Timeout: integer);
{This option uses a timer event handler to check for completion of
connection
and handle successful connection or timeout}
begin
with Timer1 do
begin
{.}Interval:= Timeout;
{.}Enabled:= True;
end;
end;

procedure WaitForConnectionToCompleteOption2(TcpClient: TTcpClient;
Timeout: integer);
{This option uses the Select method to wait for the client socket to
become
ready for writing- based on Remy Lebeau code}
var
WriteReady: Boolean;
begin
WriteReady:= False;
TcpClient.Select(nil, @WriteReady, nil, Timeout);
if not WriteReady then
HandleFailureToConnect;
end;

procedure WaitForConnectionToCompleteOption3(TcpClient: TTcpClient;
Timeout: integer);
{This option uses the WaitForData method to wait for the client socket to
become
ready for reading - only works if server sends connection
acknowledgement}
var
ReadReady: Boolean;
begin
ReadReady:= TcpClient.WaitForData(Timeout);
if not ReadReady then
HandleFailureToConnect;
end;

begin
WaitOption:= 2; {Set 1, 2 or 3 according to option to be tested}
with TcpClient1 do
if not {.}Connected then
begin
{.}RemoteHost:= EditRemoteIPAddress.Text;
{.}RemotePort:= EditRemotePort.Text;
Screen.Cursor:= crHourglass;
{Attempt to connect to remote socket:}
{.}Open;
if not {.}Connected then
{Wait for connection attempt to complete, and handle result:}
case WaitOption of
1: WaitForConnectionToCompleteOption1(FConnectTimeout);
2: WaitForConnectionToCompleteOption2(TcpClient1,
FConnectTimeout);
3: WaitForConnectionToCompleteOption3(TcpClient1,
FConnectTimeout);
end;
end;
end;

Regards,

EM

[Remy Lebeau (TeamB)]

Post by Enquiring Mind
I have found that if the above statement is omitted, i.e. the BlockMode
is left unchanged, then the timeout code and the connection works
properly.

The BlockMode property is set to bmBlocking by default. When set to that,
Open() would block until the connection is fully established, not allowing
any timeout code to execute in the meantime.

Post by Enquiring Mind
It looks as if the OS automatically changes the BlockMode to
NonBlocking while a connect operation is in progress

It does not. Read the documentation for connect():

http://msdn.microsoft.com/en-us/library/ms737625(VS.85).aspx

Post by Enquiring Mind
If one looks at the thread count for the client application in the Task
Manager, one finds that there are 2 threads. This adds credence to
the idea that even if TcpClient1.Open is called from the main thread,
the work of connecting to the remote socket is internally performed
in a secondary thread.

Even if that were true, TTcpClient itself is NOT creating that thread. Look
at TTcpClient's source code for yourself. Such a secondary thread would
have to be internal to the OS. Open() merely calls the API connect()
function and then waits for it to exit. In blocking mode, connect() is
REQUIRED to block the calling thread until the connection is established, or
until an error occurs. There is nothing asynchronous about that.


Gambit

[Enquiring Mind]

Post by Remy Lebeau (TeamB)
The OnConnect event only works in blocking mode. One of the shortcomings
of TTcpClient (and TTcpServer) is that is does NOT implement any kind of
asynchronous event handling at all (unlike the older TClientSocket [and
TServerSocket] component, which did). If you want to support asynchronous
events, you have to implement them manually, such as by calling
WSAAsyncSelect() or WSAEventSelect() in your own code. As such,
TTcpClient in non-blocking mode has no way of responding to a successful
connect that occurs after Open() has already exited. This is just one of
many reasons why I keep telling people NOT to use TTcpClient (and
TTcpServer). It is simply not implemented very well.
You don't need to use a separate reading thread when the socket is in
non-blocking mode. If you use WSAAsyncSelect() or WSAEventSelect(), your
code will be notified when data arrives. So all of your code can reside
in the same thread. That is the whole reason why Microsoft invented
non-blocking sockets under Windows in the first place (well, back in the
Win.1 days when preemptive multithreading didn't exist yet, anyway).
Pretty much all other platforms use blocking sockets instead.

Thanks for this background information. I can now more or less see that the
TTcpClient and TTcpServer components are only really suitable for blocking
sockets, in spite of the fact that the socket BlockMode can be set to
bmNonBlocking.. Is this because they are intended to be cross-platform, and
Unix/Linux only supports blocking sockets?

Are you saying that the earlier components TClientSocket and TServerSocket
under the hood call API functions like WSAAsyncSelect() or WSAEventSelect(),
thus saving the developer from having to program non-blocking sockets
directly at the Winsock API level?

What are the advantages of non-blocking sockets over blocking sockets? Why
don't Indy components cover non-blocking sockets? What are the pros and cons
of using WSAAsyncSelect() versus WSAEventSelect()?

I seem to perceive that blocking sockets can only be used on a pre-emptive
multitasking operating system like Windows NT and its later versions.
Otherwise non-blocking sockets must be used.

On a first reading it looks as if WSAAsyncSelect can be used to associate a
message handler with a socket event, so as you say the socket programming
can be executed entirely in the main thread of the program. The message
handler completes an asynchronous socket I/O operation initiated by a call
to send or recv.

On the other hand it appears that WSAEventSelect() requires a call to
WSAWaitForMultipleEvents for the operation to be completed, which is a
blocking function, and would therefore have to be called within a secondary
thread. The only advantage of WSAEventSelect over synchronous socket I/O
would seem to be that one can include other events in the list of events
being waited for, and thus gracefully exit from a socket reading thread that
is blocked pending arrival of some data to be read.

Which approach do developers usually prefer? As far as Indy is concerned,
the answer seems to be blocking sockets!

EM

[Remy Lebeau (TeamB)]

Post by Enquiring Mind
Thanks for this background information. I can now more or less see that
the TTcpClient and TTcpServer components are only really suitable for
blocking sockets, in spite of the fact that the socket BlockMode can be
set to bmNonBlocking.. Is this because they are intended to be
cross-platform,
and Unix/Linux only supports blocking sockets?

Probably.

Post by Enquiring Mind
Are you saying that the earlier components TClientSocket and
TServerSocket under the hood call API functions like WSAAsyncSelect() or
WSAEventSelect()

They used WSAAsyncSelect() with a private HWND.

Post by Enquiring Mind
What are the advantages of non-blocking sockets over blocking sockets?

Your code can do other things while a socket is busy, without having to use
multiple threads.

Post by Enquiring Mind
Why don't Indy components cover non-blocking sockets?

Because:

1) Indy is cross-platform, so it can't perform non-blocking operations on
non-Windows platforms.

2) Indy is designed with linear, non event-driven, threadable operations in
mind. Non-blocking sockets don't fit that model.

Post by Enquiring Mind
What are the pros and cons of using WSAAsyncSelect() versus
WSAEventSelect()?

They pretty much do the same thing. The only difference is that
WSAAsyncSelect() posts an application-defined message to a specified HWND,
whereas WSAEventSelect() signals a specified event object instead.

Post by Enquiring Mind
I seem to perceive that blocking sockets can only be used on a pre-emptive
multitasking operating system like Windows NT and its later versions.

They work fine on all Windows systems going back to Win95. The only reason
Microsoft invented non-blocking sockets is because Win3.1 couldn't support
them as it was not a multi-threaded platform. A blocking socket operation
could freeze up the entire OS.

Post by Enquiring Mind
On the other hand it appears that WSAEventSelect() requires a call to
WSAWaitForMultipleEvents for the operation to be completed,
which is a blocking function, and would therefore have to be called
within a secondary thread.

No. It can be used in a single thread. It has a timeout parameter. A
thread could start a non-blocking socket operation, move on to other things,
and then periodically poll the event(s) so see if the operation is finished.


Gambit

[Enquiring Mind]

Post by Remy Lebeau (TeamB)

Post by Enquiring Mind
Thanks for this background information. I can now more or less see that
the TTcpClient and TTcpServer components are only really suitable for
blocking sockets, in spite of the fact that the socket BlockMode can be
set to bmNonBlocking.. Is this because they are intended to be
cross-platform,
and Unix/Linux only supports blocking sockets?

Probably.

Am I right in thinking that the first operating system to conceive and
implement sockets as the basis for inter-computer communication was Unix,
and that Microsoft subsequently created Winsock as an essentially compatible
port of the Unix model to the Windows platform, but couldn't resist adding
Windows-specific features to something that should be kept
non-platform-specific?

The fact that Unix/Linux still holds a significant share of the internet
server market would seem to suggest that in a socket server application it's
a good idea to only use features common to Unix and Windows. Since the Mac
OS is based on Unix are its sockets facilities more or less compatible with
those of the original Unix?

Post by Remy Lebeau (TeamB)
They used WSAAsyncSelect() with a private HWND.

I only have Delphi 7. Where can I find the unit containing the TClientSocket
and TServerSocket components?

Post by Remy Lebeau (TeamB)

Post by Enquiring Mind
What are the advantages of non-blocking sockets over blocking sockets?

Your code can do other things while a socket is busy, without having to
use multiple threads.

In other words, it looks like non-blocking sockets offer no major advantages
over blocking sockets for a developer who is comfortable programming with
multiple threads!

Post by Remy Lebeau (TeamB)

Post by Enquiring Mind
Why don't Indy components cover non-blocking sockets?

1) Indy is cross-platform, so it can't perform non-blocking operations on
non-Windows platforms.

That seems to confirm my previous comment that there are no major advantages
in using non-blocking sockets if the code is to be portable to Windows and
Unix.

Post by Remy Lebeau (TeamB)
2) Indy is designed with linear, non event-driven, threadable operations
in mind. Non-blocking sockets don't fit that model.

Such a model makes the logic of the code much easier to comprehend. The same
applies to blocking sockets in a secondary thread (where all the work is
done in a single TThread.Execute method).

Post by Remy Lebeau (TeamB)

Post by Enquiring Mind
What are the pros and cons of using WSAAsyncSelect() versus
WSAEventSelect()?

They pretty much do the same thing. The only difference is that
WSAAsyncSelect() posts an application-defined message to a specified HWND,
whereas WSAEventSelect() signals a specified event object instead.

The WSAEventSelect approach seems to be much more consistent with that of
I/O through the serial port using WaitForMultipleObjects - as such it is to
be preferred, IMO, unless a message based thread object is used. There is no
conceptual difference to my mind between a bi-directional data stream
through a hard I/O channel, and the bi-directional data stream through a
socket, which is just a soft I/O channel.

Post by Remy Lebeau (TeamB)

Post by Enquiring Mind
I seem to perceive that blocking sockets can only be used on a
pre-emptive multitasking operating system like Windows NT and its later
versions.

They work fine on all Windows systems going back to Win95. The only
reason Microsoft invented non-blocking sockets is because Win3.1 couldn't
support them as it was not a multi-threaded platform. A blocking socket
operation could freeze up the entire OS.

So could it be argued that the need for non-blocking sockets is obsolete?

Post by Remy Lebeau (TeamB)

Post by Enquiring Mind
On the other hand it appears that WSAEventSelect() requires a call to
WSAWaitForMultipleEvents for the operation to be completed,
which is a blocking function, and would therefore have to be called
within a secondary thread.

No. It can be used in a single thread. It has a timeout parameter. A
thread could start a non-blocking socket operation, move on to other
things, and then periodically poll the event(s) so see if the operation is
finished.

For this single-thread approach to work, one must use the
WSAWaitForMultipleEvents to frequently check whether or not an event has
occurred (i.e. poll the event status) rather than wait for it to occur in a
wait-efficient way. To poll the events presumably means passing the API
function a zero or very short timeout interval. I would have thought that
this would be considerably less efficient in terms of processor cycles
consumed than waiting for the event to occur in a secondary thread, and
putting the thread to sleep in an efficient wait-state. I think I read
somewhere in the Windows documentation that polling should be avoided
whenever possible.

EM

[Remy Lebeau (TeamB)]

Post by Enquiring Mind
Am I right in thinking that the first operating system to conceive
and implement sockets as the basis for inter-computer communication
was Unix

Yes - specifically, Berkeley Unix.

Post by Enquiring Mind
and that Microsoft subsequently created Winsock as an essentially
compatible port of the Unix model to the Windows platform

Actually, Microsoft did not implement the first WinSock. Back in Windows
3.0, third-party apps had to provide their own socket stacks. It wasn't
until Windows 3.x that Microsoft started building its own into the OS.

Post by Enquiring Mind
but couldn't resist adding Windows-specific features to something
that should be kept non-platform-specific?

Microsoft didn't have much of a choice in that matter. As I explained
earlier, blocking sockets would not have worked very well in the Windows 3.x
environment. Non-blocking features had to be added to the API in order for
sockets to even be feasible. When Windows 95 was released, blocking sockets
could then be used effectively. But Microsoft has continued to carry on,
and even to expand on, its non-blocking features as well.

Post by Enquiring Mind
Since the Mac OS is based on Unix are its sockets facilities
more or less compatible with those of the original Unix?

Just about everyone's socket capabilities are mostly compatible with the
original socket API - even Microsoft's.

Post by Enquiring Mind
I only have Delphi 7. Where can I find the unit containing the
TClientSocket and TServerSocket components?

They are in the "ScktComp" unit.

Post by Enquiring Mind
That seems to confirm my previous comment that there are no major
advantages in using non-blocking sockets if the code is to be portable
to Windows and Unix.

Portable code, no. But if you are not worried about portability, then there
is no reason now to use non-blocking features if they are available.

Post by Enquiring Mind
The WSAEventSelect approach seems to be much more consistent
with that of I/O through the serial port using WaitForMultipleObjects

WSAEventSelect() and WSAWaitForMultipleObjects() were introduced in Winsock
2.0 when Microsoft added support for overlapped I/O and completion ports to
WinSock.

Post by Enquiring Mind
So could it be argued that the need for non-blocking sockets is obsolete?

Not necessarily. WinSock's overlapped I/O support provides for better
server scalability. With blocking sockets, you would usually need to have 1
thread per client connection. That is the model Indy servers currently use.
If you have a lot of clients connected at one time, that is a lot of
individual threads running. Eventually, you will reach OS limitations in
how many concurrent clients you can handle effectively. With non-blocking
and overlapped I/O, on the other hand, you can mimimize the number of
threads needed, and better optimize the threads you do have for the number
of CPUs that are present.


Gambit

[Enquiring Mind]

"Remy Lebeau (TeamB)" <***@no.spam.com> wrote in message news:488dfff4$***@newsgroups.borland.com...
Thanks for useful background information on the historical development of
socket facilities.

A point you seem to be making is that if the operating system only supports
single-threaded applications, as I think you said was the case before
Windows 95, then you cannot use blocking socket I/O without blocking the
whole application, so you must use non-blocking socket I/O instead. Since at
the lowest level all I/O functions are intrinsically blocking (correct me if
I'm wrong), I presume that to enable non-blocking I/O the operating system
has to execute each blocking I/O function call in a separate thread that the
OS itself creates and owns, thus relieving the application from having to
handle this chore.

The question of the superior scalability of non-blocking sockets over
blocking sockets is an interesting one. It's true that with blocking sockets
you need a separate thread for every socket connection to accommodate the
blocking I/O functions, and that these have to be created and managed by the
application. However with non-blocking sockets I would have thought that you
still need a separate thread for every socket connection blocking I/O
function call, but that the threads are now created and managed by the
operating system rather than by the application. I can see, nevertheless,
that since the lifetime of the system-created threads is likely to be
shorter than that of application-created threads, the number of threads in
existence at any one time is likely to be lower.

If one uses a TTcpServer component to handle client socket connections, then
to achieve maximum internet-style scalability with blocking socket I/O it
should suffice to limit the life of the thread created for the connection to
just the duration of a single request . For example:

procedure TForm1.TcpServer1Accept(Sender: TObject;
ClientSocket: TCustomIpClient);
{Note: Blocking mode socket}
var
Request, Response: string;
begin
{Read request:}
Request:= '';
ClientSocket.ReceiveLn(Request);
Response:= '';
DetermineResponse(Request, Response);
{Write response:}
ClientSocket.SendLn(Response);
end;

With this code the thread created to handle the client connection is
released shortly after the OnAccept event handler is exited, so exists only
for the duration of the single request.

EM

[Remy Lebeau (TeamB)]

Post by Enquiring Mind
A point you seem to be making is that if the operating system only
supports single-threaded applications, as I think you said was the
case before Windows 95, then you cannot use blocking socket I/O
without blocking the whole application so you must use non-blocking
socket I/O instead.

I was pointing out that Windows 3.x itself did not have any notion of
threads. Running a blocking operation in an application could freeze the
entire OS. That was no longer the case when Windows 95 was introduced.
Nobody targets Windows 3.x anymore, so don't worry about it.

Post by Enquiring Mind
However with non-blocking sockets I would have thought that you still
need a separate thread for every socket connection blocking I/O function
call, but that the threads are now created and managed by the operating
system rather than by the application.

Not exactly. I suggest you read up on how overlapped I/O and completion
ports actually work:

Socket I/O
http://msdn.microsoft.com/en-us/library/ms740523(VS.85).aspx

Socket overlapped I/O versus blocking/nonblocking mode
http://support.microsoft.com/kb/181611

Overlapped I/O and Event Objects
http://msdn.microsoft.com/en-us/library/ms740087.aspx
http://msdn.microsoft.com/en-us/library/aa923630.aspx

Synchronous and Asynchronous I/O
http://msdn.microsoft.com/en-us/library/aa365683.aspx

I/O Completion Ports
http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx

Post by Enquiring Mind
If one uses a TTcpServer component to handle client socket connections,
then to achieve maximum internet-style scalability with blocking socket
I/O
it should suffice to limit the life of the thread created for the
connection to just the duration of a single request .

Not really. Repeatedly creating and tearing down threads has quite a bit of
overhead to it, especially the shorter the time each thread runs for. If
you are going to use such a model, then thread pooling would be a better
choice.
TTcpServer (and TServerSocket) uses thread pooling internally.

Post by Enquiring Mind
With this code the thread created to handle the client connection
is released shortly after the OnAccept event handler is exited

No, it is not. Look at the TTcpServer source code more carefully. The
thread calls the OnAccept event in a loop, one firing per client connection.
When the OnAccept handler exits, the thread loops back and processes the
next client connection, or goes to sleep if there is no pending client.


Gambit

[Enquiring Mind]

I suggest you read up on how overlapped I/O and completion ports actually
...

Thanks for the useful references.

To qualify the point I made in my previous post about the relative
computational efficiency of non-blocking versus blocking socket I/O, I can
see that non-blocking socket I/O would be more efficient than a
well-programmed blocking socket I/O if the processor that performs the
background socket I/O is not one of the main processors used directly by the
OS to process threads, but a dedicated I/O interface processor. The same
also applies to overlapped I/O.

To revise the point I made about the suitability of the TTcpClient and
TTcpServer components for non-blocking socket I/O, I can see that although
the components don't support the more advanced Winsock 2 non-blocking and
overlapped socket I/O facilities (i.e. the API functions prefixed by WSA), I
would have thought that they can still be used for non-blocking socket I/O
if the select method is used to wait for asynchronous I/O completion.

Not really. Repeatedly creating and tearing down threads has quite a bit
of overhead to it, especially the shorter the time each thread runs for.
If you are going to use such a model, then thread pooling would be a
better choice.

I know that a thread pool can be used to avoid the overhead of repeated
thread object creation and destruction. But the point I made about the
impact of the duration of each thread session on overall efficiency still
holds, I would have thought, because the shorter the duration, the lower the
number of concurrently active threads.

Post by Enquiring Mind
With this code the thread created to handle the client connection
is released shortly after the OnAccept event handler is exited

No, it is not. Look at the TTcpServer source code more carefully. The
thread calls the OnAccept event in a loop, one firing per client
connection. When the OnAccept handler exits, the thread loops back and
processes the next client connection, or goes to sleep if there is no
pending client.

I was aware that TTcpServer uses a thread pool for client socket threads. By
'released', I didn't mean destroyed, but released from active service and
returned to the pool of inactive threads.

I have read in accounts of socket facilities provided by other languages
about the importance of calling the 'flush' function between a send and recv
call to ensure that socket I/O operation is completed and all data in the
socket's outgoing data stream buffer is sent down the line. I haven't found
a function by this name in the WinSock API, or in the Delphi components.
Does this mean that the send and recv Windows API functions automatically
take care of flushing the buffer?

Regards,

EM

[Remy Lebeau (TeamB)]

I would have thought that they can still be used for non-blocking
socket I/O

They can be, otherwise the BlockMode property would not exist in the first
place.

I have read in accounts of socket facilities provided by other languages
about the importance of calling the 'flush' function between a send and
recv call to ensure that socket I/O operation is completed and all data
in the socket's outgoing data stream buffer is sent down the line.

There is no 'flush' function in the socket API. Once outbound data has been
passed to a socket, it is out of the application's hands. The socket sends
the data on its own time.

I haven't found a function by this name in the WinSock API, or in the
Delphi components.

Because there isn't one.


Gambit

[Enquiring Mind]

"Remy Lebeau (TeamB)" <***@no.spam.com> wrote in message news:48909e51$***@newsgroups.borland.com...

I hope I am not making this thread go on too long - but it's an interesting
subject!

Post by Remy Lebeau (TeamB)

I would have thought that they can still be used for non-blocking
socket I/O

They can be, otherwise the BlockMode property would not exist in the first
place.

That's true, but the implementation of TBaseSocket class in the Sockets unit
doesn't seem to take into account the implications of the BlockMode property
on the validity of the code. Consider for example the method

function TBaseSocket.Receiveln(const eol: string): string;

As I see it, the method contains 2 conceptual errors:

Error 1) Since the code contains a call of Recv inside a loop, it probably
won't work if the socket is non-blocking, because it would not wait for the
completion of one Recv operation before proceeding to the next Recv
operation. Therefore the code should either check the blocking mode and
raise an exception if it's non-blocking, or else it should correctly handle
the non-blocking socket case and include some code to wait for the
completion of the Recv operation.

Error 2) The function's code previews ("peeks") the incoming stream in
chunks of 511 bytes or less, and searches the chunk for the termination
string "eol". If the termination string is found, then the function receives
the characters up to and including the end of the termination string and
terminates the loop, and if it isn't found, the function receives the whole
chunk and continues the loop. Thus 2 possible cases are considered: the
chunk of data in the socket's buffer either contains the termination string,
or it doesn't. But there is a third case not considered: the termination
string straddles 2 successive chunks. Should this case arise, the code would
not identify the end of string condition and the loop would not terminate
correctly.

All this leads me to think: wouldn't it be much better, and easier, if
Delphi simply provided a library of functions in Pascal syntax for the
Berkeley socket functions in the Windows API, as listed in the MS SDK help
files under "socket functions", and let the developer worry about thread
creation etc.? The library would be easily portable to any operating system
that supports the Berkeley socket model - i.e all the major OS's. Granted, a
developer could use the WinSock API functions directly - but the code would
be more readable and more portable if entirely in Pascal style.

It would also be nice if Delphi were to provide a library of data stream
I/O routines in Pascal syntax that support the MS asynchronous I/O model
(i.e. overlapped IO, synchronization objects, WaitForXXX functions, etc.).
These should be applicable without change to serial port data streams and
socket data streams. The conceptual model should be an abstract
time-discontinuous data stream accessed though a queued buffer. The
conceptual model should include notification of asynchronous I/O completion
by 4 methods: a) calling of an event handler b) posting a message c)
signalling a synchronization object d) calling a Windows completion routine.
The important thing is that the Delphi encapsulation should be in terms of
the abstract model, so that it can be readily ported to any OS whose API
supports asynchronous I/O. The data stream type could simply be an extended
type of file, that adds to the characteristics of the original Pascal file
type the concepts of access through a queue, asynchronous operation and
notification of completion.

I have noticed that both Java and C# provide socket routines that are
relatively close to the Berkeley socket model, and in both the socket
exposes input and output data streams that can be manipulated just like any
other data stream. If only Delphi could keep things equally simple!

EM

[Remy Lebeau (TeamB)]

Post by Enquiring Mind
That's true, but the implementation of TBaseSocket class in the Sockets
unit doesn't seem to take into account the implications of the BlockMode
property on the validity of the code.

Pretty much everything in that unit do not take non-blocking into account
properly.

Post by Enquiring Mind
wouldn't it be much better, and easier, if Delphi simply provided a
library of
functions in Pascal syntax for the Berkeley socket functions in the
Windows
API, as listed in the MS SDK help files under "socket functions", and let
the
developer worry about thread creation etc.?

Look at the VCL's "WinSock" unit (which the "Sockets" and "Scktcomp" units
use internally).


Gambit

[Enquiring Mind]

Post by Remy Lebeau (TeamB)
Not exactly. I suggest you read up on how overlapped I/O and completion
Socket I/O
http://msdn.microsoft.com/en-us/library/ms740523(VS.85).aspx
Socket overlapped I/O versus blocking/nonblocking mode
http://support.microsoft.com/kb/181611
Overlapped I/O and Event Objects
http://msdn.microsoft.com/en-us/library/ms740087.aspx
http://msdn.microsoft.com/en-us/library/aa923630.aspx
Synchronous and Asynchronous I/O
http://msdn.microsoft.com/en-us/library/aa365683.aspx
I/O Completion Ports
http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx

Following the guidelines in the above links, I have started looking into
writing some non-blocking socket I/O routines using WinSock API functions
like WSAWaitForMultipleEvents. But the first problem I have encountered is
that many of the WSAxxx API functions needed are not included in my version
of the Winsock unit (shipped with Delphi 7). I have checked dates on the
internet and have found that Winsock 2 was published a considerable time
before Delphi 7, so I was wondering why the omission. To add further to my
puzzlement is the fact that the Windows SDK help files shipped with my
version of Delphi do include the WSAxxx routines.

Are the WSAxxx routines to be found in some other unit, then?

Regards,

EM

[Remy Lebeau (TeamB)]

Post by Enquiring Mind
Following the guidelines in the above links, I have started looking into
writing some non-blocking socket I/O routines using WinSock API
functions like WSAWaitForMultipleEvents. But the first problem I
have encountered is that many of the WSAxxx API functions needed
are not included in my version of the Winsock unit (shipped with Delphi
7). I have checked dates on the internet and have found that Winsock 2
was published a considerable time before Delphi 7, so I was wondering why
the omission.

You would have to ask CodeGear that. The VCL has not been updated to use
WinSock 2 in any version. Not even Delphi 2007 has the functions declared.

Post by Enquiring Mind
To add further to my puzzlement is the fact that the Windows SDK help
files shipped with my version of Delphi do include the WSAxxx routines.

That has always been the case.

Post by Enquiring Mind
Are the WSAxxx routines to be found in some other unit, then?

No. You will have to declare them yourself, or find a third-party unit that
does it for you.


Gambit