How to handle gz files fetched with IdHTTP?

[Bo Berglund]

I need to build a data analysis application which gets its data from a
website as xml files. However the website has decided to compress the
files into gz format in order to improve transmission speed.
My problem now is how to handle these files once downloaded so I can
get at the xml code inside.

I have never coded any compression/decompression functionality except
a long time ago when I created zip files in Delphi. But that was done
using a zip dll from InfoZip (now no longer on the web) and it was
done the other way around...

Now I need to *un*-compress gz files in code.

How can this be done? I realize this NG might not be the right one to
ask in, so if you know of a better one then please tell me so.


Bo Berglund

[Stig Johansen]

Post by Bo Berglund
Now I need to *un*-compress gz files in code.
How can this be done?

Take a look at zlib.pas and the gzip header definition in rfc 1952
<http://www.faqs.org/rfcs/rfc1952.html>

[Bo Berglund]

Post by Stig Johansen

Post by Bo Berglund
Now I need to *un*-compress gz files in code.
How can this be done?

Take a look at zlib.pas and the gzip header definition in rfc 1952
<http://www.faqs.org/rfcs/rfc1952.html>

Way too complicated for me...
But I googled a bit and found some examples that finally led me to
download something called zlib123 from
http://www.dellapasqua.com/delphizlib/
Unfortunately when I try the suggested decoding using the below code I
get only an error box saying "Data error".

I use as the example file (channels.xml.gz) a file I have downloaded
from the web in gz format.
If I use 7Zip on this file it decodes fine into a channels.xml file.

procedure TForm1.DecodeFile(Src, Dst: string);
var
InputStream,
OutputStream: TFileStream;
DecompressionStream: TZDecompressionStream;
begin
InputStream := TFileStream.Create(Src, fmOpenRead);
OutputStream := TFileStream.Create(Dst, fmCreate);
DecompressionStream := TZDecompressionStream.Create(InputStream);
OutputStream.CopyFrom(DecompressionStream, 0);
DecompressionStream.Free;
OutputStream.Free;
InputStream.Free;
end;

What can I do?????

[Bo Berglund]

On Wed, 04 Jun 2008 17:08:25 +0200, Bo Berglund

Post by Bo Berglund

Post by Stig Johansen

Post by Bo Berglund
Now I need to *un*-compress gz files in code.
How can this be done?

Take a look at zlib.pas and the gzip header definition in rfc 1952
<http://www.faqs.org/rfcs/rfc1952.html>

Way too complicated for me...
But I googled a bit and found some examples that finally led me to
download something called zlib123 from
http://www.dellapasqua.com/delphizlib/
Unfortunately when I try the suggested decoding using the below code I
get only an error box saying "Data error".
I use as the example file (channels.xml.gz) a file I have downloaded
from the web in gz format.
If I use 7Zip on this file it decodes fine into a channels.xml file.
procedure TForm1.DecodeFile(Src, Dst: string);
var
InputStream,
OutputStream: TFileStream;
DecompressionStream: TZDecompressionStream;
begin
InputStream := TFileStream.Create(Src, fmOpenRead);
OutputStream := TFileStream.Create(Dst, fmCreate);
DecompressionStream := TZDecompressionStream.Create(InputStream);
OutputStream.CopyFrom(DecompressionStream, 0);
DecompressionStream.Free;
OutputStream.Free;
InputStream.Free;
end;
What can I do?????

I also tried this with a fresh download of the delphi zlib1.2.3 from
http://www.base2ti.com/zlib.htm
Same result, data error....
So zlibex cannot handle this gz file but 7Zip can.

procedure TForm1.DecompressFile(const inFile, outFile: String);
var
zStream : TZDecompressionStream;
inStream : TFileStream;
outStream: TFileStream;
size : Cardinal;
begin
// open the in file and get the original (decompressed)
// file size
// create the out file

inStream := TFileStream.Create(inFile,fmOpenRead);
outStream := TFileStream.Create(outFile,fmCreate);
zStream := TZDecompressionStream.Create(inStream);
try
try
inStream.Read(size,SizeOf(Cardinal));
// decompress
outStream.CopyFrom(zStream,size); <== creates exception!!!
except
on E: Exception do
ShowMessage(E.Message);
end;
finally
// clean up
zStream.Free;
outStream.Free;
inStream.Free;
end;
end;

[Stig Johansen]

Post by Bo Berglund

Post by Bo Berglund
What can I do?????

I also tried this with a fresh download of the delphi zlib1.2.3 from
http://www.base2ti.com/zlib.htm
Same result, data error....
So zlibex cannot handle this gz file but 7Zip can.

It's kind of difficult to copy paste source code to this machine, but i have
the (gzip) header definition definition as follows:
.....
unit gzipU;

interface

CONST
FTEXT = 1 ;
FHCRC = 2 ;
FEXTRA = 4 ;
FNAME = 8 ;
FCOMMENT = 16 ;

TYPE

tGZipFirstHeader = PACKED RECORD
ID1 : Byte ;
ID2 : Byte ;
CM : Byte ;
FLG : Byte ;
MTIME : INTEGER ;
XFL : Byte ;
OS : Byte ;
END ;
......

In my program, i read the header and afterwards the content into a string.
With (after setting the length):
I := FileRead(fileHandle,compressedstr[1],I) ;
Afterwards i just call:
compressedstr := ZReceiveFromBrowser(compressedstr) ;
from zlibEx (this is some code from around 2002 running on Linux, so no
guarantee).

The point is, that the 'inner data' is the same as deflate, and when the
headers is included, it's the same as gzip.

[Bo Berglund]

Post by Stig Johansen

Post by Bo Berglund

Post by Bo Berglund
What can I do?????

I also tried this with a fresh download of the delphi zlib1.2.3 from
http://www.base2ti.com/zlib.htm
Same result, data error....
So zlibex cannot handle this gz file but 7Zip can.

It's kind of difficult to copy paste source code to this machine, but i have
.....
unit gzipU;
interface
CONST
FTEXT = 1 ;
FHCRC = 2 ;
FEXTRA = 4 ;
FNAME = 8 ;
FCOMMENT = 16 ;
TYPE
tGZipFirstHeader = PACKED RECORD
ID1 : Byte ;
ID2 : Byte ;
CM : Byte ;
FLG : Byte ;
MTIME : INTEGER ;
XFL : Byte ;
OS : Byte ;
END ;
......
In my program, i read the header and afterwards the content into a string.
I := FileRead(fileHandle,compressedstr[1],I) ;
compressedstr := ZReceiveFromBrowser(compressedstr) ;
from zlibEx (this is some code from around 2002 running on Linux, so no
guarantee).
The point is, that the 'inner data' is the same as deflate, and when the
headers is included, it's the same as gzip.

Well, that might be good to know if one is to go deep into the
compression mystery, but all I want to do is to use an IdHTTP
component to retrieve the name.xml.gz files and have them converted to
xml so I can then process the contents...

I have googled along more now and tested yet another Delphi
implementation purportedly handling Gzipped files. I got a unit here:
http://www.torry.net/pages.php?s=99
The actual zipfile URL is:
http://www.torry.net/vcl/compress/std/delphiunzip.zip
It contains a single Delphi unit named lzwUnit, described as follows:

Description:
function unlzw(inStr,outStr:TStream): boolean;
- inStr: is a Stream that contains the complete LZW Data including
the Magic header.
- outStr: is a Stream where the Data have to be stored.

function unlzwFile(srcFile,dstFile:String): boolean;
- srcFile: is the Filename of the compressed file.
- dstFile: is the Filename where the decompressed data have to be
stored.

I downloaded a file channels.xml.gz and then I tried this code:

if not dlgOpen.Execute then exit;
Src := dlgOpen.FileName;
Dest := ChangeFileExt(Src,'');
unlzwFile(Src, Dest);

However it generates an error:
"Input not in compress format (read magic number 0x1F8B)"

When I check with a hex editor the first few bytes in the file are
0x1F 8B 08 00 ...

It seems *really* hard to find some way to download gz encoded files
from the web and convert to the contents(xml) in a Delphi app.
If one clicks the link directly in FireFox the browser seems to
uncompress directly to the screen so that the xml is visible.

If I use 7Zip on the gz file it also creates the correct xml file, it
seems only like these units I have tested totally fail. :-(

Isn't there some way in IdHTTP or some other Indy unit that allows a
gz URL to be automagically expanded on download just like FireFox is
doing so the result is the internal xml file???


/BoB

[Stig Johansen]

Post by Bo Berglund

Post by Stig Johansen
It's kind of difficult to copy paste source code to this machine, but i
.....
unit gzipU;
interface
CONST
FTEXT = 1 ;
FHCRC = 2 ;
FEXTRA = 4 ;
FNAME = 8 ;
FCOMMENT = 16 ;
TYPE
tGZipFirstHeader = PACKED RECORD
ID1 : Byte ;
ID2 : Byte ;
CM : Byte ;
FLG : Byte ;
MTIME : INTEGER ;
XFL : Byte ;
OS : Byte ;
END ;
......
In my program, i read the header and afterwards the content into a string.
I := FileRead(fileHandle,compressedstr[1],I) ;
compressedstr := ZReceiveFromBrowser(compressedstr) ;
from zlibEx (this is some code from around 2002 running on Linux, so no
guarantee).
The point is, that the 'inner data' is the same as deflate, and when the
headers is included, it's the same as gzip.

Well, that might be good to know if one is to go deep into the
compression mystery, but all I want to do is to use an IdHTTP
component to retrieve the name.xml.gz files and have them converted to
xml so I can then process the contents...

There is not much of mystery in there.
The tGZipFirstHeader definition is the _fixed_ part of a gzip file as
defined in the previous mentioned rfc.
So i read the fixed part, an afterwards the variable part, and _then_ the
compressed data block into "compressedstr" (a string in my example)
It should be the same whether data comes from a file or IdHTTP. (I use
synapse, so i don't know idHTTP).
The decompression is only this single line:
compressedstr := ZReceiveFromBrowser(compressedstr) ;
where ZReceiveFromBrowser is in unit zlibEx.pas
Afterwards compressedstr contains the uncompressed data.

Post by Bo Berglund
"Input not in compress format (read magic number 0x1F8B)"
When I check with a hex editor the first few bytes in the file are
0x1F 8B 08 00 ...

Theese bytes are correct according to the gzip rfc.

Post by Bo Berglund
It seems *really* hard to find some way to download gz encoded files
from the web and convert to the contents(xml) in a Delphi app.

My things are from about 2002, but i think i had a hard time back then,
figuring things out.

[Bo Berglund]

Post by Stig Johansen

Post by Bo Berglund
Well, that might be good to know if one is to go deep into the
compression mystery, but all I want to do is to use an IdHTTP
component to retrieve the name.xml.gz files and have them converted to
xml so I can then process the contents...

There is not much of mystery in there.
The tGZipFirstHeader definition is the _fixed_ part of a gzip file as
defined in the previous mentioned rfc.
So i read the fixed part, an afterwards the variable part, and _then_ the
compressed data block into "compressedstr" (a string in my example)
It should be the same whether data comes from a file or IdHTTP. (I use
synapse, so i don't know idHTTP).

But I have a single downloaded file on my disk so there are no
separate parts to "load". I just want to subject the *file* to the
decormpression and all I get is a "data error".
I also googled a discussion about just this where they talked about
setting some values in the constructor of the object like this:

TZDecompressionStream.Create(TFileStream.Create(gzfile,
fmOpenRead),12+32);

However there is no overloaded constructor thta accepts a second
parameter in my versions of ZLibEx.pas (I have downloaded both thta
are available, 1.1.4 and 1.2.3).

Post by Stig Johansen
compressedstr := ZReceiveFromBrowser(compressedstr) ;
where ZReceiveFromBrowser is in unit zlibEx.pas

ZReceiveFromBrowser does not exist in any of my two downloaded copies
of ZlibEx.pas...

Post by Stig Johansen
Afterwards compressedstr contains the uncompressed data.

Post by Bo Berglund
"Input not in compress format (read magic number 0x1F8B)"
When I check with a hex editor the first few bytes in the file are
0x1F 8B 08 00 ...

Theese bytes are correct according to the gzip rfc.

That should work then and yet it does not..

Post by Stig Johansen

Post by Bo Berglund
It seems *really* hard to find some way to download gz encoded files
from the web and convert to the contents(xml) in a Delphi app.

My things are from about 2002, but i think i had a hard time back then,
figuring things out.

Seems like the same is true 6 years later...


/BoB

[Stig Johansen]

Post by Bo Berglund

Post by Stig Johansen

Post by Bo Berglund
Well, that might be good to know if one is to go deep into the
compression mystery, but all I want to do is to use an IdHTTP
component to retrieve the name.xml.gz files and have them converted to
xml so I can then process the contents...

There is not much of mystery in there.
The tGZipFirstHeader definition is the _fixed_ part of a gzip file as
defined in the previous mentioned rfc.
So i read the fixed part, an afterwards the variable part, and _then_ the
compressed data block into "compressedstr" (a string in my example)
It should be the same whether data comes from a file or IdHTTP. (I use
synapse, so i don't know idHTTP).

But I have a single downloaded file on my disk so there are no
separate parts to "load". I just want to subject the *file* to the
decormpression and all I get is a "data error".
I also googled a discussion about just this where they talked about

I am not clear enough, i can see.
If i rewind my memory, we had this strange browser (IE) wich only supported
deflate, and not gzip.
On the other hand we had the other browsers, which supported gzip, but not
deflate.

I needed to support both, so thats where my jurney started.

I discovered that deflate was 'just' the compressed data without the
headers.

And gzip was just the compressed/deflated data with header and checksum.

So you can look at gzip roughly as:
gzip := header + compressed_data + trailer.

So when you do this (from your other post):
  Res := IdHTTP1.Get('http://tv.swedb.se/xmltv/channels.xml.gz');
Your Res contains the above construction.

What i am talking about is basically to exctract the compressed_data part
from the file/string/stream.

Post by Bo Berglund
TZDecompressionStream.Create(TFileStream.Create(gzfile,
fmOpenRead),12+32);

It looks like some hardcoded sizes of the header data.
The problem is that the header can contain variable parts, so you can not be
sure where exactly the compressed_part starts in the file/string/stream.

Post by Bo Berglund
ZReceiveFromBrowser does not exist in any of my two downloaded copies
of ZlibEx.pas...

Thinking about it, it may very well be my own addition to implement
decompression of deflate, but here it is:
.....
function ZReceiveFromBrowser(const s: string) : string;
var outBuf: Pointer; outBytes: Integer;
begin
ZDecompress2(pointer(s), length(s), outBuf, outBytes, 0);
SetLength(result, outBytes);
Move(pointer(outBuf)^, pointer(result)^, outBytes);
FreeMem(outBuf);
end;
.....

Post by Bo Berglund

Post by Stig Johansen
My things are from about 2002, but i think i had a hard time back then,
figuring things out.

Seems like the same is true 6 years later...

Yes but i downloaded this from your other post:
<http://tv.swedb.se/xmltv/channels.xml.gz>

And it decompresses fine with my programs as mentioned, so the data is
correct.

This is still on Linux which uses native zlib library.

[Bo Berglund]

Post by Stig Johansen
I am not clear enough, i can see.
If i rewind my memory, we had this strange browser (IE) wich only supported
deflate, and not gzip.
On the other hand we had the other browsers, which supported gzip, but not
deflate.
I needed to support both, so thats where my jurney started.
I discovered that deflate was 'just' the compressed data without the
headers.
And gzip was just the compressed/deflated data with header and checksum.
gzip := header + compressed_data + trailer.
  Res := IdHTTP1.Get('http://tv.swedb.se/xmltv/channels.xml.gz');
Your Res contains the above construction.
What i am talking about is basically to exctract the compressed_data part
from the file/string/stream.

Post by Bo Berglund
TZDecompressionStream.Create(TFileStream.Create(gzfile,
fmOpenRead),12+32);

Again:
However there is no overloaded constructor that accepts a second
parameter in my versions of ZLibEx.pas (I have downloaded both that
are available, 1.1.4 and 1.2.3).
So how can I enter the 12+32 parameter????

Post by Stig Johansen
It looks like some hardcoded sizes of the header data.
The problem is that the header can contain variable parts, so you can not be
sure where exactly the compressed_part starts in the file/string/stream.

Post by Bo Berglund
ZReceiveFromBrowser does not exist in any of my two downloaded copies
of ZlibEx.pas...

Thinking about it, it may very well be my own addition to implement
.....
function ZReceiveFromBrowser(const s: string) : string;
var outBuf: Pointer; outBytes: Integer;
begin
ZDecompress2(pointer(s), length(s), outBuf, outBytes, 0);
SetLength(result, outBytes);
Move(pointer(outBuf)^, pointer(result)^, outBytes);
FreeMem(outBuf);
end;
.....

Post by Bo Berglund

Post by Stig Johansen
My things are from about 2002, but i think i had a hard time back then,
figuring things out.

Seems like the same is true 6 years later...

<http://tv.swedb.se/xmltv/channels.xml.gz>
And it decompresses fine with my programs as mentioned, so the data is
correct.

Could you post the *complete* code you used to decompress successfully
the channels.xml.gz file, please?

And since you are on Linux, how can you use Delphi then????

[Stig Johansen]

Post by Bo Berglund
Could you post the *complete* code you used to decompress successfully
the channels.xml.gz file, please?

<danish>
Puha , Bo, Grundlovsdag herover på Sjælland, lidt rigeligt med øl, hvidvin,
rødvin, små grå, hårrødderne vender den forkerte vej m.v. men:
</danish>
I can see you have gotten Indy to work, but it *is* more or less the
complete code i have posted.
As mentioned, this piece of code decompresses a file, and not a buffer from
a HTTP request, but here is more code extraction:

I := FileRead(fileHandle,GZipFirstHeader,SIZEOF(GZipFirstHeader));
IF ( GZipFirstHeader.ID1 = 31 ) AND (GZipFirstHeader.ID2 = 139 ) THEN
BEGIN
IsGzip := TRUE ;
ModifiedTime := GZipFirstHeader.MTIME / 86400 {$IFDEF LINUX} + 25567
......
IF IsGzip AND NOT ReturnZip THEN BEGIN
PosStart := I ;
IF ( GZipFirstHeader.FLG AND FNAME ) <> 0 THEN BEGIN
I := FileRead(fileHandle,FileNameBuffer,255);
INC(PosStart,StrLen(FileNameBuffer) + 1); // +1 FOR 0
TERMINATION
FileSeek(woprfileHandle,PosStart,0 );
END ;
I := Fs - 8 - PosStart ; // FS is the File size.
SetLength(compressedstr,I) ;
I := FileRead(fileHandle,compressedstr[1],I) ;
compressedstr := ZReceiveFromBrowser(compressedstr) ;
......
Result :=
QueueBuffer(Pchar(compressedstr),Length(compressedstr),FALSE)
......

QueueBuffer is a routine to Queue data back to the browser, and
ZReceiveFromBrowser is the previously posted code added to ZlibEX.pas.

Post by Bo Berglund
And since you are on Linux, how can you use Delphi then????

Kylix, which was bundled to my version of D7.
Develop, test, debug with D7, (re)compile/deploy on Linux.

I had various attempts to checkout Linux. Back in 2002, it worked on Linux
as well, but compared to Windows, the speed was more or less the same, so
no benefit from Linux.

Then somewhere around 2004-2005, there was too much inconsitency (aka
missing compatibility) in Linux, so i didn't even bother to try again.

So now it's, lets call it 3. attempt to evaluate the matureness of Linux
(Only serverside).

The key issue here(to me) is NPTL Threads, which seems to be (very) much
faster than Windows.

I have built a little server with an old PII 200Mhz, 96MB RAM, and i can
easily handle 300+ simultaneous requests on that little beast.

Ok, not _entirely_ true, since i have raised the Maxconnections from the
default 32 to 128 in the webbroker part.

That issue is on my internal 'to do list', since there should be no such low
limit, but i have to investigate the impact on memory usage etc.

[Remy Lebeau (TeamB)]

Post by Stig Johansen
I needed to support both, so thats where my jurney started.

TIdHTTP in Indy 10 supports both.


Gambit

[Remy Lebeau (TeamB)]

Post by Bo Berglund
Isn't there some way in IdHTTP or some other Indy unit that allows
a gz URL to be automagically expanded on download just like
FireFox is doing so the result is the internal xml file???

See my other reply.


Gambit

[Remy Lebeau (TeamB)]

Post by Bo Berglund
I need to build a data analysis application which gets its data from
a website as xml files. However the website has decided to compress
the files into gz format in order to improve transmission speed.

There are two different aspects to that - is the server sending an actual
.gz file, or is it simply compressing the regular .xml file using a GZ
transformation during the data transfer? Those scenerios are handled quite
differently. In the former case, TIdHTTP would only be able to give you the
.gz file as-is, and then you would have to decompress it separately. In the
latter case, however, TIdHTTP in Indy 10 has built-in support for decoding
the GZ compression on the fly as the data is being downloaded, providing you
with the original uncompressed .xml file. So you need to figure out which
scenerio is actually occuring. Can you provide a trace log from a packet
sniffer that shows the actual HTTP reply data?


Gambit

[Bo Berglund]

On Wed, 4 Jun 2008 12:32:38 -0700, "Remy Lebeau \(TeamB\)"

Post by Remy Lebeau (TeamB)

Post by Bo Berglund
I need to build a data analysis application which gets its data from
a website as xml files. However the website has decided to compress
the files into gz format in order to improve transmission speed.

There are two different aspects to that - is the server sending an actual
.gz file, or is it simply compressing the regular .xml file using a GZ
transformation during the data transfer? Those scenerios are handled quite
differently. In the former case, TIdHTTP would only be able to give you the
.gz file as-is, and then you would have to decompress it separately. In the
latter case, however, TIdHTTP in Indy 10 has built-in support for decoding
the GZ compression on the fly as the data is being downloaded, providing you
with the original uncompressed .xml file. So you need to figure out which
scenerio is actually occuring. Can you provide a trace log from a packet
sniffer that shows the actual HTTP reply data?

Remy,
The actual URL for my test file (there are others from the same
source) is:
http://tv.swedb.se/xmltv/channels.xml.gz
If I just click this link I get a page with the actual code displayed
as xml.
If I choose RightClick/SaveLinkAs in FireFox I can save it to disk and
then it is a binary file of size about 2K.
7Zip is then able to "Extract Here" and bingo, I have a 19K regular
XML file (with Unix line endings).

I don't know the inner workings of such things as compression or
Internet transfers etc and I don't know what you would like me to do
regarding a "trace log from a packet sniffer". What is that?

I am working to improve my EPG program using TV program data from this
site, which is always in gz format.


/BoB

[Remy Lebeau (TeamB)]

The actual URL for my test file (there are others from the same source)
http://tv.swedb.se/xmltv/channels.xml.gz

Even though the URL looks like a .gz file, the server is actually delivering
a plain .xml file that is gzip-compressed dynamically as it is being
downloaded. So TIdHTTP in Indy 10 can decompress the file automatically for
you. Simply assign a TIdCompressorZLib component to the TIdHTTP.Compressor
property before calling TIdHTTP.Get().


Gambit

[Bo Berglund]

On Wed, 4 Jun 2008 18:12:37 -0700, "Remy Lebeau \(TeamB\)"

Post by Remy Lebeau (TeamB)

The actual URL for my test file (there are others from the same source)
http://tv.swedb.se/xmltv/channels.xml.gz

Even though the URL looks like a .gz file, the server is actually delivering
a plain .xml file that is gzip-compressed dynamically as it is being
downloaded. So TIdHTTP in Indy 10 can decompress the file automatically for
you. Simply assign a TIdCompressorZLib component to the TIdHTTP.Compressor
property before calling TIdHTTP.Get().

This sounds like *exactly* what I want to do, but then in BDS2006 I
got into trouble immediately...

1) There is no component named TIdCompressorZLib
2) I found one named TIdCompressorZLibEx so I put this on my form
3) Now I get a bunch of errors:
[Pascal Error] IdZLib.pas(1224): E1026 File not found: 'adler32.obj'
[Pascal Error] IdZLib.pas(1225): E1026 File not found: 'compress.obj'
[Pascal Error] IdZLib.pas(1226): E1026 File not found: 'crc32.obj'
[Pascal Error] IdZLib.pas(1227): E1026 File not found: 'deflate.obj'
[Pascal Error] IdZLib.pas(1228): E1026 File not found: 'infback.obj'
[Pascal Error] IdZLib.pas(1229): E1026 File not found: 'inffast.obj'
[Pascal Error] IdZLib.pas(1230): E1026 File not found: 'inflate.obj'
[Pascal Error] IdZLib.pas(1231): E1026 File not found: 'inftrees.obj'
[Pascal Error] IdZLib.pas(1232): E1026 File not found: 'trees.obj'
[Pascal Error] IdZLib.pas(1233): E1026 File not found: 'uncompr.obj'
[Pascal Error] IdZLib.pas(1234): E1026 File not found: 'zutil.obj'
These files are mentioned in IdZlib in comments looking like this:

{$L adler32.obj}
{$L compress.obj}
{$L crc32.obj}
{$L deflate.obj}
{$L infback.obj}
{$L inffast.obj}
{$L inflate.obj}
{$L inftrees.obj}
{$L trees.obj}
{$L uncompr.obj}
{$L zutil.obj}

I made a search from the top of my BDS2006 installation for
adler32.obj but it is not there..

So what do I do now? My Indy is what came with BDS2006 (10.1.5).


/BoB

[Stig Johansen]

Post by Bo Berglund
{$L adler32.obj}

.....
Actually it's a directive to link in the mentioned file(s).

Post by Bo Berglund
I made a search from the top of my BDS2006 installation for
adler32.obj but it is not there..
So what do I do now? My Indy is what came with BDS2006 (10.1.5).

I know there are various places where you can download the set of .obj
files.
I have tried to Google, but i can't remember what to Google with.
I don't remember where i got them from, perhaps it was on my D7 CD.

[Remy Lebeau (TeamB)]

You should only be getting those errors if you are trying to re-compile
Indy. Otherwise, the .obj files are already compiled into the Indy binaries
that Borland shipped.

Post by Bo Berglund
My Indy is what came with BDS2006 (10.1.5).

You are using an outdated Indy 10 version. There have been many changed to
Indy 10 since the 10.1.5 release. The current version is 10.2.3 now.


Gambit

[Bo Berglund]

On Wed, 4 Jun 2008 12:32:38 -0700, "Remy Lebeau \(TeamB\)"

Post by Remy Lebeau (TeamB)

Post by Bo Berglund
I need to build a data analysis application which gets its data from
a website as xml files. However the website has decided to compress
the files into gz format in order to improve transmission speed.

There are two different aspects to that - is the server sending an actual
.gz file, or is it simply compressing the regular .xml file using a GZ
transformation during the data transfer? Those scenerios are handled quite
differently. In the former case, TIdHTTP would only be able to give you the
.gz file as-is, and then you would have to decompress it separately. In the
latter case, however, TIdHTTP in Indy 10 has built-in support for decoding
the GZ compression on the fly as the data is being downloaded, providing you
with the original uncompressed .xml file. So you need to figure out which
scenerio is actually occuring. Can you provide a trace log from a packet
sniffer that shows the actual HTTP reply data?

I tried this:
var
Res: string;
begin
Res := IdHTTP1.Get('http://tv.swedb.se/xmltv/channels.xml.gz');

This works, but Res contains the *compressed* data.
What I now had in mind was to supply the Res string to some decoding
function such as discussed here and get the plaintext out of it.

This is done with Delphi7 where I have only Indy 9.0.10 installed...


/BoB

[Bo Berglund]

On Thu, 05 Jun 2008 00:14:56 +0200, Bo Berglund

Post by Bo Berglund
var
Res: string;
begin
Res := IdHTTP1.Get('http://tv.swedb.se/xmltv/channels.xml.gz');
This works, but Res contains the *compressed* data.
What I now had in mind was to supply the Res string to some decoding
function such as discussed here and get the plaintext out of it.
This is done with Delphi7 where I have only Indy 9.0.10 installed...

Fired up BDS2006 with Indy 10 and got the exact same result.


/BoB

[Remy Lebeau (TeamB)]

Post by Bo Berglund
Fired up BDS2006 with Indy 10 and got the exact same result.

Because you did not configure TIdHTTP for decompressing yet. See my other
reply.


Gambit

[Remy Lebeau (TeamB)]

<snip>

Post by Bo Berglund
This works, but Res contains the *compressed* data.

See my other reply.

Post by Bo Berglund
What I now had in mind was to supply the Res string to some
decoding function such as discussed here and get the plaintext
out of it.

Yes, you will have to do that if you do not allow Indy to decompress the
data for you automatically.

Post by Bo Berglund
This is done with Delphi7 where I have only Indy 9.0.10 installed...

You will have to upgrade to Indy 10.2.3 if you want to take advantage of
TIdHTTP's automated decompressing feature.


Gambit

[Bo Berglund]

On Wed, 4 Jun 2008 18:14:03 -0700, "Remy Lebeau \(TeamB\)"

Post by Remy Lebeau (TeamB)
Yes, you will have to do that if you do not allow Indy to decompress the
data for you automatically.

Post by Bo Berglund
This is done with Delphi7 where I have only Indy 9.0.10 installed...

You will have to upgrade to Indy 10.2.3 if you want to take advantage of
TIdHTTP's automated decompressing feature.

Then we have the problem of upgrading Indy in BDS2006 again (there are
other threads about this). I have 10.1.5 as delivered with BDS2006....
Exactly how is this upgrade done?


/BoB

[Bo Berglund]

On Wed, 4 Jun 2008 18:14:03 -0700, "Remy Lebeau \(TeamB\)"

Post by Remy Lebeau (TeamB)

Post by Bo Berglund
This is done with Delphi7 where I have only Indy 9.0.10 installed...

You will have to upgrade to Indy 10.2.3 if you want to take advantage of
TIdHTTP's automated decompressing feature.

Remy,
thank you very much for your help! Really appreciated! :-)

I have now installed Indy 10 (snapshot) in Delphi7 and tested the
IdHTTP.GET after assigning the compressor to the component.
As you claimed it worked directly!

Now I only have to figure out the best way to switch between Indy 9
and Indy 10 on my D7 installation so I can maintain the old code.
Might go to using a virtual machine for development and then I could
probably use multiple dev machines depending on if I am doing
maintenance or new dev. We'll see.
Thanks again, now I can concentrate on the main job of downloading all
the data files and extracting the TV EPG for my system. :-)


/BoB