Can I read the cookies out from a WebBrowser

[unknown]

My intention is to fetch data, embedded in HTML documents, from a website
with a TIdHTTP. So, I have a form with a TIdHTTP dropped on it, and using
the GET method I can read the HTML documents into a string for further
processing. And so far all is OK.
The next step is to be able to read some files that requires that I am
logged in. The login page has the Username and Password fields as expected
and they are submitted to the server with the POST method over a HTTPS
connection. The server returns two cookies that then are used as
authorisation for the rest of the session over normal HTTP connection. This
I found out with the help of a sniffer demo (which unfortunately expired
and I cant use anymore).
Now I suppose that I could simulate a browser with the TIdHTTP using the SSL
libraries (which I already downloaded for the purpose) to be able to get
hold of the cookies, but before I go down that route I would like to ask
your opinions if the following would be possible:

Can I use a TWebBrowser for the actual logging in and then somehow fetch the
cookies from the TWebBrowser to continue using them with the TIdHTTP and
TIdCookieManager? Looking at the properties and methods of the TWebBrowser I
could not identify anything as related to the cookies, but maybe there is a
way. Btw, the cookies are not stored on disk, probably because they don't
have an expiry date.

I'm using D2005 and Indy 10.0.20. Reading other messages on this board I
know its an old version, but I would not right now like to go through the
upgrading procedure, if possible.

Best regards

[Johnnie Norsworthy]

Post by unknown
Can I use a TWebBrowser for the actual logging in and then somehow fetch
the cookies from the TWebBrowser to continue using them with the TIdHTTP
and TIdCookieManager? Looking at the properties and methods of the
TWebBrowser I could not identify anything as related to the cookies, but
maybe there is a way. Btw, the cookies are not stored on disk, probably
because they don't have an expiry date.

This is exactly what I have to do, as the login page I use executes some
script before posting and add some unique entries each time.

uses OleCtrls, SHDocVw, MSHTML_TLB,

Use TInternetExplorer.OnDocumentComplete to get the cookies.

var
s: String;
iDoc: IHTMLDocument2;
begin
try
iDoc := WebBrowser.Document as IHTMLDocument2;
s := iDoc.cookie;
except
s := '';
end;
// here I check the cookie string to see if my login cookie is there...
// if Pos('Auth',s)<>0 then ...
end;

[unknown]

Great! Thanks Johnny.
You obviously have some library (MSHTML_TLB) that I don't have; I had to do
some more research, but you encouraged me to continue on this route.
This is how I got a grip on those cookies:
s := WebBrowser1.OleObject.Document.Cookie;

BTW, funny that you are looking for the same identification string 'auth' as
I am ;-)

Thanks

Post by Johnnie Norsworthy

Post by unknown
Can I use a TWebBrowser for the actual logging in and then somehow fetch
the cookies from the TWebBrowser to continue using them with the TIdHTTP
and TIdCookieManager? Looking at the properties and methods of the
TWebBrowser I could not identify anything as related to the cookies, but
maybe there is a way. Btw, the cookies are not stored on disk, probably
because they don't have an expiry date.

This is exactly what I have to do, as the login page I use executes some
script before posting and add some unique entries each time.
uses OleCtrls, SHDocVw, MSHTML_TLB,
Use TInternetExplorer.OnDocumentComplete to get the cookies.
var
s: String;
iDoc: IHTMLDocument2;
begin
try
iDoc := WebBrowser.Document as IHTMLDocument2;
s := iDoc.cookie;
except
s := '';
end;
// here I check the cookie string to see if my login cookie is there...
// if Pos('Auth',s)<>0 then ...
end;

[Remy Lebeau (TeamB)]

Post by unknown
The next step is to be able to read some files that requires that I am
logged in. The login page has the Username and Password fields as
expected and they are submitted to the server with the POST method
over a HTTPS connection. The server returns two cookies that then
are used as authorisation for the rest of the session over normal HTTP
connection.

TIdHTTP handles the receiving and sending of cookies automatically.

Post by unknown
Can I use a TWebBrowser for the actual logging in and then somehow
fetch the cookies from the TWebBrowser to continue using them with
the TIdHTTP and TIdCookieManager?

After the page is fully loaded, you can query the TWebBrowser's Document
property for the IHTMLDocument2 interface, and then use its get_cookie()
method to retrieve individual cookies. You will have to know the cookie
name(s) ahead of time, as the browser does not provide any access to the
entire cookie collection in an indexable fashion.

Alternatively, you can use the InternetGetCookie/Ex() functions from the
WinInet API.

Alternatively, you can manually extract the information from the cookie
files directly. Use the SHGetSpecialFolderLocation() function to find where
IE keeps its Cookies folder on the machine. You would then have to locate
the individual files yourself, open them, read from them, and then populate
the TIdCookieManager. Indy has no support for automating that process for
you.

Post by unknown
Looking at the properties and methods of the TWebBrowser I could
not identify anything as related to the cookies

There are nothing in TWebBrowser itself. You would have to access the
document's DOM interfaces instead.

Post by unknown
Btw, the cookies are not stored on disk, probably because they don't have

an expiry date.

More likely that they are session cookies instead. Session cookies are kept
in memory and are discarded when the browser is closed.
InternetGetCookie/Ex() can retreive session cookies. And the
getIHTMLDocument2:get:_cookie() method can probably get them as well.


Gambit

[unknown]

Thank you Remy!
This was a good sumary of the options I have.
You may have seen in my answer to Johnnie that I found a way to get the
cookies from the webbrowser using the OleObject, but it appears now that I
do not get the additional fields 'domain' and 'path', which I assume I would
need when I populate TIdCookieManager. This is not really a problem in this
case because this small appplication is intended only to fetch data from a
specific site, so I can hard code the domain and path.
Just of curiosity, would either the WinInet API InternetGetCookie/Ex()
function or the get_cookie() method of the IHTMLDocument2 interface give me
these domain and path fiekds?

Regards

Post by Remy Lebeau (TeamB)

Post by unknown
The next step is to be able to read some files that requires that I am
logged in. The login page has the Username and Password fields as
expected and they are submitted to the server with the POST method
over a HTTPS connection. The server returns two cookies that then
are used as authorisation for the rest of the session over normal HTTP
connection.

TIdHTTP handles the receiving and sending of cookies automatically.

Post by unknown
Can I use a TWebBrowser for the actual logging in and then somehow
fetch the cookies from the TWebBrowser to continue using them with
the TIdHTTP and TIdCookieManager?

After the page is fully loaded, you can query the TWebBrowser's Document
property for the IHTMLDocument2 interface, and then use its get_cookie()
method to retrieve individual cookies. You will have to know the cookie
name(s) ahead of time, as the browser does not provide any access to the
entire cookie collection in an indexable fashion.
Alternatively, you can use the InternetGetCookie/Ex() functions from the
WinInet API.
Alternatively, you can manually extract the information from the cookie
files directly. Use the SHGetSpecialFolderLocation() function to find where
IE keeps its Cookies folder on the machine. You would then have to locate
the individual files yourself, open them, read from them, and then populate
the TIdCookieManager. Indy has no support for automating that process for
you.

Post by unknown
Looking at the properties and methods of the TWebBrowser I could
not identify anything as related to the cookies

There are nothing in TWebBrowser itself. You would have to access the
document's DOM interfaces instead.

Post by unknown
Btw, the cookies are not stored on disk, probably because they don't have

an expiry date.
More likely that they are session cookies instead. Session cookies are kept
in memory and are discarded when the browser is closed.
InternetGetCookie/Ex() can retreive session cookies. And the
getIHTMLDocument2:get:_cookie() method can probably get them as well.
Gambit

[Remy Lebeau (TeamB)]

it appears now that I do not get the additional fields 'domain'
and 'path', which I assume I would need when I populate
TIdCookieManager.

If no domain is specified in the cookie, then the cookie applies to the
domain of the document that sent the cookie. Likewise, if no path is
provided, the the cookie applies to the path of the document.

would either the WinInet API InternetGetCookie/Ex() function or
the get_cookie() method of the IHTMLDocument2 interface give
me these domain and path fiekds?

You are already working with the IHTMLDocument2 interface when you access
the Document of the TWebBrowser. The getter method for the Cookie property
is the get_cookie() method. So if that property is not already giving you
the domain and path, then there is your answer to that.

As for InternetGetCookie/Ex(), it retreives whatever data is available in
the specified cookie.


Gambit