latest 64 bit gzip archive (hdsentinel-019c-x64.gz) is malfo

[HaroldFinch]

Background: I have a Windows laptop with a corrupted disk and/or file system that I need to diagnose.
I created a bootable USB flash drive running the latest Linux Mint Cinnamon (20.2) that I want to put disk diagnostic tools on.
The first tool that I want to use is the Linux version of Hard Disk Sentinel.
(Link: https://www.hdsentinel.com/hard_disk_sentinel_linux.php)
In particular, in the previous link's "Download Hard Disk Sentinel Linux" section, I want to use "Hard Disk Sentinel 64-bit Linux console version - executable, gzip-compressed".
(Link: https://www.hdsentinel.com/hdslin/hdsentinel-019c-x64.gz)

I found that the latest gzip archive (hdsentinel-019c-x64.gz) is malformed.

In particular, when I try to extract it in Linux Mint I get different (wrong) results whereas I can extract it in Windows (cygwin or 7-zip).

This is not a problem with my version of Linux Mint, since I tested a different gzip archive file and found that Mint extracted it just fine.

Therefore, the issue seems to be with hdsentinel-019c-x64.gz: it must have been created with a tool that violates the gzip spec or something?

Full details follow.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What I first tried in Linux Mint:

I opened a browser, went to
https://www.hdsentinel.com/hard_disk_sentinel_linux.php
and downloaded the file
hdsentinel-019c-x64.gz

I immediately tried to extract the file inside that archive by executing
$ gzip -dk hdsentinel-019c-x64.gz

[hdsentinel]

Thanks, but I can confirm that the file is not "malformed" of course, the file is 100% correct.

In particular, when I try to extract it in Linux Mint I get different (wrong) results whereas I can extract it in Windows (cygwin or 7-zip).

Sorry for the troubles, but this confirms that the archive file is absolutely correct and the problem is with this Linux distro.

One user reported that when he attempted to download the file, Linux (incorrectly) thinks that it is "double" compressed.
So when he downloaded the file and used gzip to extract, he received a

HDSentinel

file - but it is not the executable but a compressed file again. After renaming that to hdsentinel.gz and extracting again, then things worked correctly.

[HaroldFinch]

[Dear Moderator: when composing my first post, I accidentally clicked Submit when I meant to click on Preview. Crap!!! So, only a fraction of my text is there.

Below is the complete post that I originally intended. If you can replace my original post with the text below, that would be great. I ask because I see no way to edit my original post at this point.

And please read all of it before concluding that Linux Mint is defective! I offer evidence that there is something wrong with the HDS .gz archive.

Finally, if you could give me the privilege to use url tags to make proper hyperlinks, I am sure that everyone reading this would be grateful.]




Background: I have a Windows laptop with a corrupted disk and/or file system that I need to diagnose.

So I created a bootable USB flash drive running the latest Linux Mint Cinnamon (20.2) that I want to put disk diagnostic tools on.

The first tool that I want to use is the Linux version of Hard Disk Sentinel.
(Link: https://www.hdsentinel.com/hard_disk_sentinel_linux.php)

In particular, in the previous link's "Download Hard Disk Sentinel Linux" section, I want to use "Hard Disk Sentinel 64-bit Linux console version - executable, gzip-compressed".
(Link: https://www.hdsentinel.com/hdslin/hdsentinel-019c-x64.gz)

That last link downloads the gzip archive file hdsentinel-019c-x64.gz which I think is malformed.

In particular, when I try to extract it in Linux Mint I get different (wrong) results whereas I can extract it in Windows (partly correctly with cygwin, fully correctly with 7-zip).

This is likely not a problem with my version of Linux Mint or cygwin, since I also tested a different gzip archive file and found that Mint and cygwin extracted it perfectly.

Therefore, the issue seems to be with hdsentinel-019c-x64.gz: it must have been created with a tool that violates the gzip spec or something?

Full details follow.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What I first tried in Linux Mint:

I opened a browser, went to
(Link: https://www.hdsentinel.com/hard_disk_sentinel_linux.php)
and downloaded the file
hdsentinel-019c-x64.gz

I immediately tried to extract the file inside that archive by executing

Code: Select all

$ gzip -dk hdsentinel-019c-x64.gz

Here is the result:

Code: Select all

$ ls -al
...
-rw-rw-r--  1 mint mint 1589849 Aug 12 00:16 hdsentinel-019c-x64
-rw-rw-r--  1 mint mint 1589737 Aug 12 00:16 hdsentinel-019c-x64.gz

Note that the file that was extracted is named hdsentinel-019c-x64 (NOT HDSentinel) and that its size (1589849) is barely bigger than the gzip archive that it came from (1589737). These are all bad signs. To no surprise then, when I tried to execute it

Code: Select all

sudo ./hdsentinel-019c-x64

that failed with an error message.

What the heck is going on with that archive?

Here is what the gzip command thinks it contains:

Code: Select all

$ gzip -lv hdsentinel-019c-x64.gz
method  crc     date  time           compressed        uncompressed  ratio uncompressed_name
defla c0f770b6 Aug 12 00:16             1589737             1589849   0.0% hdsentinel-019c-x64

The output above agrees exactly with what gzip extracted above, but it is definitely NOT what that archive is supposed to contain, see below.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What I next tried in a different Windows 10 computer:

I opened a cygwin bash shell and first examined the contents of hdsentinel-019c-x64.gz:

Code: Select all

$ gzip -lv hdsentinel-019c-x64.gz
method  crc     date  time           compressed        uncompressed  ratio uncompressed_name
defla a757f917 Aug 11 14:56             1589849             3222032  50.7% hdsentinel-019c-x64

Bad: it still thinks that the file to extracted is named hdsentinel-019c-x64 (NOT HDSentinel). Good: it now thinks that its uncompressed size (3222032) is much bigger than the gzip archive that it came from (1589849).

In cygwin, I extracted it using

Code: Select all

$ gunzip -k hdsentinel-019c-x64.gz

which left the file

Code: Select all

hdsentinel-019c-x64

which is the wrong file name (should be HDSentinel) but its size is correct (3222032).

I next opened hdsentinel-019c-x64.gz in 7-zip. There, I see that the compressed file has the proper name HDSentinel, and using 7-zip I extracted that file.

I then compared the 7-zip extraction (HDSentinel) against the cygwin extraction (hdsentinel-019c-x64) and found that they were byte-byte identical.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Summary for hdsentinel-019c-x64.gz:
--Linux Mint gets the extracted file name and file contents wrong
--cygwin gets the extracted file name wrong but the file contents correct
--7-zip gets the extracted file name and file contents correct

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You may be wondering if there is something buggy with my cygwin or Linux Mint.

I do not think so.

I used 7-zip to create a totally new gzip archive, and I found that both cygwin and Linux Mint were able to extract the file perfectly (both its name and contents).

I conclude that there is something wrong just with the archive file hdsentinel-019c-x64.gz: was it created with a tool that did not follow the proper gzip format?

[hdsentinel]

Thanks for your message.

I'm sorry for the troubles, but yes, exactly what you see confirms the bug in Linux Mint.

> I opened a browser, went to
> (Link: https://www.hdsentinel.com/hard_disk_sentinel_linux.php)
> and downloaded the file
> hdsentinel-019c-x64.gz
> -rw-rw-r-- 1 mint mint 1589849 Aug 12 00:16 hdsentinel-019c-x64
> -rw-rw-r-- 1 mint mint 1589737 Aug 12 00:16 hdsentinel-019c-x64.gz[/code]

Yes, this is wrong.
The file hdsentinel-019c-x64.gz is 1589849 files long on the server. So when you download that file, you should immediately get the file with that size.
It seems Linux Mint somehow "added" to the file and made it corrupted. The archive has only one file (HDSentinel) itself.
So what you see is surely not related to the file - but related to how Linux Mint downloaded/packed incorrectly.

The archive file hdsentinel-019c-x64.gz should be 1589849 bytes long when you download it - and when extracted, the file HDSentinel inside should be 3222032 bytes.

The problem is in the browser of Linux Mint which does not handle correctly if the server has enabled compressed transfers (as recommended/forced by Google to optimize delivery). It seems Linux Mint somehow "double" compress the already compressed .gz archive during download and this is why you need to extract first to get the REAL archive (which you need to extract again).

When you download the same file under Windows, you get the proper size (1589849 bytes) which confirms that the Linux Mint is causing the problem.

> I conclude that there is something wrong just with the archive file hdsentinel-019c-x64.gz: was it created with a tool that did not follow the proper gzip format?

Absolutely and surely NOT. Exactly to prevent this (and similar) the same tool used for many-many years (Total Commander, and even same version) to create the archive.
The problem is NOT with the file, but the fact how Linux Mint downloads it from the server.

[hdsentinel]

Probably similar to this issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1470011

[hdsentinel]

I uploaded differently, so you can try these links to get the REAL file:

https://www.hdsentinel.com/hdslin/hdsentinel-019c-x64/HDSentinel (3222032 bytes)

OR

https://www.hdsentinel.com/hdslin/hdsentinel-019c-x64/hdsentinel-019c-x64.zip (1589938 bytes)

[hdsentinel]

Please see attached screenshot, captured under Linux Mint.
This confirms the bug related to Mozilla Firefox:



If you download the file with Firefox, the file is wrong (double compressed).
If you download the file with wget, the file is correct.

So the best is to use the command on the screenshot:

wget https://www.hdsentinel.com/hdslin/hdsentinel-019c-x64.gz

to download the file.

[HaroldFinch]

(Crap, second time I have to totally retype this, got logged out after a few minutes of inactivity and my entire draft was eliminated when I re-entered my credentials. If you could fix this forum software to not do that, many would be grateful...)

hdsentinel: thanks for your many replies.

I concur that the version of firefox bundled with the Linux Mint must be mangling to download of hdsentinel-019c-x64.gz. And if the buzilla report for 1470011 that you cite is the issue, then maybe every version of firefox over the last 3+ years is affected since that bug is open. Can't believe that such a serious bug has gone on for so long...

If I use your suggested wget command in my Linux Mint, I now get correct size file downloaded (1589849 bytes).

But the decompressed file name inside hdsentinel-019c-x64.gz still comes out as hdsentinel-019c-x64; in the console output below look under "uncompressed_name".

I reported the same issue already with cygwin on Windows.

And I noted that the exact same hdsentinel-019c-x64.gz file that I download on Windows (using Brave, note firefox!), when opened using 7-zip has the correct decompressed file name of HDSentinel.

Not sure what is going on there...

Here is my Linux Mint console that proves the above statements:

Code: Select all

mint@mint:~/Documents$ wget https://www.hdsentinel.com/hdslin/hdsentinel-019c-x64.gz
--2021-08-18 22:06:20--  https://www.hdsentinel.com/hdslin/hdsentinel-019c-x64.gz
Resolving www.hdsentinel.com (www.hdsentinel.com)... 79.172.252.41
Connecting to www.hdsentinel.com (www.hdsentinel.com)|79.172.252.41|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1589849 (1.5M) [application/x-gzip]
Saving to: ‘hdsentinel-019c-x64.gz’

hdsentinel-019c-x64.gz                                      100%[=========================================================================================================================================>]   1.52M   217KB/s    in 8.2s   

2021-08-18 22:06:29 (189 KB/s) - ‘hdsentinel-019c-x64.gz’ saved [1589849/1589849]


mint@mint:~/Documents$ ls -al
total 1568
drwxr-xr-x  2 mint mint    4096 Aug 18 22:06 .
drwxr-xr-x 16 mint mint    4096 Aug 18 21:58 ..
-rw-rw-r--  1 mint mint 1589849 Feb 16  2021 hdsentinel-019c-x64.gz


mint@mint:~/Documents$ gzip -lv hdsentinel-019c-x64.gz
method  crc     date  time           compressed        uncompressed  ratio uncompressed_name
defla a757f917 Feb 16 14:22             1589849             3222032  50.7% hdsentinel-019c-x64

[wergo]

I have been using HDsentinel for ages and even have a 2014 windows licence key! In the meantime I moved to Linux (using Mint XFCE 20.1 right now) and found that my good old HDSentinel has been moving to version 0.19.

I just wanted to confirm the apparentlty malformed file. I'll try the wget approach now.

Thanks for a great programme!

[1024mb]

One question about this. In the download page there is only 019c available but if I change the url to 019d or even 019e I receive different files, are these newer versions? Why are they unpublished? Should I just stay with 019c?
Thanks.

[hdsentinel]

Those contain some very minor changes, specifically used to test/examine some special systems/situations. You can use them - but no need as generally those versions are same as 019c.