Access violation inside TIdTCPClientCustom

[Michael Stieler]

Hello,

we have massive problems debugging one of our products depending on
Indy 10.
The program is multi-threaded, and works fine for hours.. but then the
program exits with an access violation and I am not able to track it
because it never happened in the IDE yet.

Now at one time.. an access violation occured inside IDE and it pointed
me to inside TIdTCPClientCustom module, more specific, the Connect method.


At the top of the function there is

if IOHandler = nil then begin
IOHandler := MakeImplicitClientHandler;
IOHandler.OnStatus := OnStatus;
ManagedIOHandler := True;
end;

This call should make sure, that IOHandler is not nil inside the function.

After this there are a few calls to IOHandler which seemed to work.
For example:
IOHandler.Port := FPort; //BGO: just to make sure

The exception was thrown a few lines later at

IOHandler.Open;

Since I can't understand why this could happen in a linear execution,
I guess it happens because of a multi-threading issue.

I have the main VCL thread which can could the Connection (user clicks
Stop, but the error occurs without it) and have a "polling" thread which
checks if the connection is still open and re-opens if neccesary (every
30s) and in this thread context the communication is run, that is calls of
conn.IOHandler.WriteLn and conn.IOHandler.ReadLn;
(it's a telnet linke protocol)

Do you see any problems with this design which could cause this strange
exception inside Indy code?

Regards,
Michael

[Marco Caspers]

Post by Michael Stieler
Hello,

<snip>

Post by Michael Stieler
Since I can't understand why this could happen in a linear execution,
I guess it happens because of a multi-threading issue.

That is not neccesarely the case.
As you may (or may not) know, your Firewall and/or Router keep track of
connections in a (session) table, even though the actual connection
might be closed, it's possible that the table entry stays there due to
some stickyness.

If you have a lot of short sessions simultaneously you might exhaust
the maximum available number of sessions the session table of the
firewall/router can handle.

This situation might very well cause unexpected results in Indy causing
the AV.

But without the exact AV error message, very hard to say what else.

[Remy Lebeau (TeamB)]

Post by Michael Stieler
we have massive problems debugging one of our products
depending on Indy 10.

Which build number of Indy 10 are you actually using?

Post by Michael Stieler
The program is multi-threaded, and works fine for hours..
but then the program exits with an access violation and I am
not able to track it because it never happened in the IDE yet.

Have you tried using third-party tracers yet, such as MadExcept?

Post by Michael Stieler
The exception was thrown a few lines later at
IOHandler.Open;

That is not enough to diagnose your problem. Open() does a lot of work
internally. You are going to have to trace further than that.

Post by Michael Stieler
Since I can't understand why this could happen in a linear execution,
I guess it happens because of a multi-threading issue.

Do you have threads accessing the client while Connect() is still running?


Gambit

[Michael Stieler]

Post by Remy Lebeau (TeamB)

Post by Michael Stieler
we have massive problems debugging one of our products
depending on Indy 10.

Which build number of Indy 10 are you actually using?

As for the build number, I dont't know where to look for this.
The latest entry in IdTcpClient is
Rev 1.38 1/15/05 2:14:58 PM
made by you.

Post by Remy Lebeau (TeamB)

Post by Michael Stieler
The program is multi-threaded, and works fine for hours..
but then the program exits with an access violation and I am
not able to track it because it never happened in the IDE yet.

Have you tried using third-party tracers yet, such as MadExcept?

No, I didn't know this was possible or neccesary, but I will try it.

Post by Remy Lebeau (TeamB)

Post by Michael Stieler
The exception was thrown a few lines later at
IOHandler.Open;

That is not enough to diagnose your problem. Open() does a lot of work
internally. You are going to have to trace further than that.

The problem is, that IOHandler is nil at this point, I forgot to mention
this.

Post by Remy Lebeau (TeamB)

Post by Michael Stieler
Since I can't understand why this could happen in a linear execution,
I guess it happens because of a multi-threading issue.

Do you have threads accessing the client while Connect() is still running?

This is what I was thinking about. Connect() blocks until success or
failure, so my poll thread couldn't call during this time.
But if for example a Status Disconnected would be fired during
Connect(), this could lead to a call to Disconnect().

Hm.. so could this happen if I call Connect() although the connection is
already established because I think it isn't.. then inside the Connect()
function the connection brakes, the IOHandler is Freed and then the Line
IOHandler.Open() leads to the access violation?

I rember hearing about a (performance?) problem with calling
conn.Connected, so I used my own status variable for this. Wrong? This
of course can lead to out-of-sync-problems...

Thanks,
Michael

Post by Remy Lebeau (TeamB)
Gambit

[Remy Lebeau (TeamB)]

Post by Michael Stieler
As for the build number, I dont't know where to look for this.

Right-click on any Indy component in the form designer.

Post by Michael Stieler
The latest entry in IdTcpClient is
Rev 1.38 1/15/05 2:14:58 PM
made by you.

That is not helpful. The change history that is recorded in each source
file no longer applies. It was created by TeamCoherence, a version control
system that Indy hasn't used for a long while now. Indy made the switch to
StarTeam awhile back, and StarTeam does not record change history in the
source code directly like TeamCoherence did.

Post by Michael Stieler
The problem is, that IOHandler is nil at this point

Well, that would certainly explain the AV, but the real problem is that it
shouldn't be nil at that point. You already showed why - TIdTCPClient
created an IOHandler object internally if one doesn't already exist. So
this goes back to my earlier idea that you are not using the TIdTCPClient
correctly in your own threading code. You are probably doing something in
your worker thread that indirectly destroys the IOHandler while Connect() is
still trying to initialize it.

Post by Michael Stieler
This is what I was thinking about. Connect() blocks until success
or failure, so my poll thread couldn't call during this time.

Why is your polling thread even running at all while the client is still
trying to connect to the server? There is nothing to poll yet. Why are you
trying to poll and connect at the same time? Don't do that. Connect fully
first, then start polling if the connection succeeded.

Post by Michael Stieler
I rember hearing about a (performance?) problem with calling
conn.Connected, so I used my own status variable for this. Wrong?

Yes. You don't need a separate variable for that.


Gambit

[Michael Stieler]

Post by Remy Lebeau (TeamB)

Post by Michael Stieler
As for the build number, I dont't know where to look for this.

Right-click on any Indy component in the form designer.

Okay , we are using Version 10.1.6.

Post by Remy Lebeau (TeamB)

Post by Michael Stieler
The problem is, that IOHandler is nil at this point

Well, that would certainly explain the AV, but the real problem is that it
shouldn't be nil at that point. You already showed why - TIdTCPClient
created an IOHandler object internally if one doesn't already exist. So
this goes back to my earlier idea that you are not using the TIdTCPClient
correctly in your own threading code. You are probably doing something in
your worker thread that indirectly destroys the IOHandler while Connect() is
still trying to initialize it.

Yes, that's what I think. I'll have a look at how this could happen..

Post by Remy Lebeau (TeamB)

Post by Michael Stieler
This is what I was thinking about. Connect() blocks until success
or failure, so my poll thread couldn't call during this time.

Why is your polling thread even running at all while the client is still
trying to connect to the server? There is nothing to poll yet. Why are you
trying to poll and connect at the same time? Don't do that. Connect fully
first, then start polling if the connection succeeded.

The poll thread is running because it handles multiple connections.
The way it works shouldn't do anything during a Connect() command.

A simplified example of what the poll thread does:

for I := 0 to Length(conns) - 1 do
begin
try
if not conns[I].IsConnected then conns[I].Connect;
except
end;

if conns[I].IsConnectedAndLoggedIn then
begin
conns[I].SendCommandLn(command);
conns[I].ReadCommandResponseLn;
end;
end;

for i := 1 to 4 do
begin
If terminated then exit;
Sleep(500);
end;

and the TConnection.Connect function does

if conn.Connected then raise Exception.Create(..);
try
conn.Connect;
except
conn.Disconnect;
raise;
end;


if the connection was established the following event handler:

procedure TConnection.Connected(ASender : TObject);
begin
try
SendLoginSequence;
FLoggedIn := true;
except
Disconnect;
end;
end;


Thanks for spending your time,
Michael

[Remy Lebeau (TeamB)]

Post by Michael Stieler
Okay , we are using Version 10.1.6.

That is a very old version.

Post by Michael Stieler
The poll thread is running because it handles multiple connections.
The way it works shouldn't do anything during a Connect() command.

Based on what you have described so far, your thread is trying to access a
client before it is ready to be accessed. I suggest you maintain a
per-client flag that the thread can look for so it only tries to poll from
clients that are successfully connected. DO NOT DO ANYTHING with a client
that is still in the process of trying to connect to its respective server.
In fact, I would suggest your polling thread maintain a thread-safe dynamic
list of clients, and then don't even add a client to the list until it is
fully connected. If any thread detects the client is no longer connected,
remove the client from the list immediately so it can't be polled anymore.
<snip>

If the polling thread is the only one who ever calls Connect(), then there
is no way the IOHandler could be nil inside of Connect(). Your earlier
descriptions suggest that you are calling Connect() outside of the polling
thread. Is that actually the case?


Gambit