Discussion:
Ominous smartd messages ....
(too old to reply)
William A. Mahaffey III
1970-01-01 00:00:00 UTC
Permalink
I have a workstation, recently (within the last 12 mos.) purpose built
that is my dev. box for in house software (CFD code & user interface).
It has 8 1 TB HDD's in a ZFS unmirrored pool. All data is backed up
nightly by rsunc to other boxen on my LAN & weekly by compressed tar to
those boxen as well. This A.M., I noticed some messages on the console
from smartd about 1 of the HDDs. I tailed my syslog file & see the
following:


[***@devbox, /etc, 9:42:37pm] 360 % tail -50 /var/log/messages ;
hwclock -r ; date
Jul 26 05:11:34 devbox kernel: pid 39865 (time), uid 1110: exited on
signal 6
Jul 26 05:22:29 devbox kernel: pid 41916 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 26 05:22:41 devbox kernel: pid 41953 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 26 05:22:52 devbox kernel: pid 41990 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 26 22:51:42 devbox healthd: A value of 1.64 for CPU #0 Core Voltage
with a range of (1.65 <= n <= 2.30)
Jul 27 06:30:56 devbox kernel: pid 71015 (memcheck-amd64-free), uid
1110: exited on signal 6
Jul 27 06:30:56 devbox kernel: pid 71014 (time), uid 1110: exited on
signal 6
Jul 27 06:41:44 devbox kernel: pid 73053 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 27 06:41:55 devbox kernel: pid 73090 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 27 06:42:07 devbox kernel: pid 73127 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 28 05:10:56 devbox kernel: pid 2029 (memcheck-amd64-free), uid 1110:
exited on signal 6
Jul 28 05:10:56 devbox kernel: pid 2028 (time), uid 1110: exited on signal 6
Jul 28 05:21:41 devbox kernel: pid 4067 (PreBFCGL.opteron.TE), uid 1110:
exited on signal 6 (core dumped)
Jul 28 05:21:51 devbox kernel: pid 4104 (PreBFCGL.opteron.TE), uid 1110:
exited on signal 6 (core dumped)
Jul 28 05:22:02 devbox kernel: pid 4141 (PreBFCGL.opteron.TE), uid 1110:
exited on signal 6 (core dumped)
Jul 29 06:30:46 devbox kernel: pid 33158 (memcheck-amd64-free), uid
1110: exited on signal 6
Jul 29 06:30:47 devbox kernel: pid 33157 (time), uid 1110: exited on
signal 6
Jul 29 06:41:39 devbox kernel: pid 35197 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 29 06:41:51 devbox kernel: pid 35234 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 29 06:42:01 devbox kernel: pid 35271 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 30 05:10:54 devbox kernel: pid 64110 (memcheck-amd64-free), uid
1110: exited on signal 6
Jul 30 05:10:54 devbox kernel: pid 64109 (time), uid 1110: exited on
signal 6
Jul 30 05:21:44 devbox kernel: pid 66149 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 30 05:21:55 devbox kernel: pid 66186 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 30 05:22:06 devbox kernel: pid 66223 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 31 16:18:28 devbox kernel: pid 96421 (memcheck-amd64-free), uid
1110: exited on signal 6
Jul 31 16:18:28 devbox kernel: pid 96420 (time), uid 1110: exited on
signal 6
Jul 31 16:29:15 devbox kernel: pid 98457 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 31 16:29:26 devbox kernel: pid 98494 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Jul 31 16:29:36 devbox kernel: pid 98531 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Aug 1 06:31:17 devbox kernel: pid 26641 (memcheck-amd64-free), uid
1110: exited on signal 6
Aug 1 06:31:17 devbox kernel: pid 26640 (time), uid 1110: exited on
signal 6
Aug 1 06:42:45 devbox kernel: pid 28681 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Aug 1 06:42:58 devbox kernel: pid 28718 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Aug 1 06:43:09 devbox kernel: pid 28755 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Aug 1 23:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 1 23:39:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 00:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 00:39:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 01:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 01:39:58 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 02:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 02:39:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 03:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 03:39:58 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 05:11:06 devbox kernel: pid 57518 (memcheck-amd64-free), uid
1110: exited on signal 6
Aug 2 05:11:06 devbox kernel: pid 57517 (time), uid 1110: exited on
signal 6
Aug 2 05:21:55 devbox kernel: pid 59557 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Aug 2 05:22:07 devbox kernel: pid 59594 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
Aug 2 05:22:16 devbox kernel: pid 59642 (PreBFCGL.opteron.TE), uid
1110: exited on signal 6 (core dumped)
hwclock: Command not found.
Tue Aug 2 22:28:28 MCDT 2016
You have new mail.
[***@devbox, /etc, 10:28:28pm] 361 % uname -a
FreeBSD devbox 9.3-RELEASE-p33 FreeBSD 9.3-RELEASE-p33 #0: Wed Jan 13
17:55:39 UTC 2016
***@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
[***@devbox, /etc, 10:28:53pm] 362 %


I am asking about the smartd messages from overnight Aug 1-2. The rest
gives a bit of context about the duty cycle of the machine, which is
pretty light except for rebuild code nightly & run a suite of scripted
tests, some of which obviously don't complete successfully :-). The
HDD's were brand new 2.5" SATA3 7200 RPM HGST's, which I have had good
luck with in other builds (I have about 25 in service, all purchased new
from NewEgg for each build). Case is well ventilated, drives are cool,
etc. From a shell script to nicely format smartctl output:


[***@devbox, /etc, 10:39:28pm] 364 % hddtemp /dev/ada[0-7]
SMART supported, SMART enabled
drive /dev/ada0: HGST HTS721010A9E630, S/N: JR10046P1EEJZN, Temp. 27
degC, min/max, cycle: 17/36, lifetime: 17/36, lifetime avg. 25 degC
drive /dev/ada1: HGST HTS721010A9E630, S/N: JR10046P1E5ZWN, Temp. 27
degC, min/max, cycle: 17/37, lifetime: 17/37, lifetime avg. 25 degC
drive /dev/ada2: HGST HTS721010A9E630, S/N: JR10046P1EKG0N, Temp. 27
degC, min/max, cycle: 16/36, lifetime: 16/36, lifetime avg. 24 degC
drive /dev/ada3: HGST HTS721010A9E630, S/N: JR10046P1EK72N, Temp. 27
degC, min/max, cycle: 17/37, lifetime: 17/37, lifetime avg. 25 degC
drive /dev/ada4: HGST HTS721010A9E630, S/N: JR10046P1E649N, Temp. 25
degC, min/max, cycle: 15/32, lifetime: 15/33, lifetime avg. 23 degC
drive /dev/ada5: HGST HTS721010A9E630, S/N: JR10046P1E69HN, Temp. 25
degC, min/max, cycle: 14/31, lifetime: 14/31, lifetime avg. 22 degC
drive /dev/ada6: HGST HTS721010A9E630, S/N: JR10046P1EK92N, Temp. 25
degC, min/max, cycle: 14/31, lifetime: 14/32, lifetime avg. 22 degC
drive /dev/ada7: HGST HTS721010A9E630, S/N: JR10046P1E6A4N, Temp. 25
degC, min/max, cycle: 14/32, lifetime: 14/32, lifetime avg. 23 degC
[***@devbox, /etc, 10:39:43pm] 365 %


My question is: Are these messages benign, or am I in the market for
more hardware ? *ANY* more questions, please ask. TIA & have a good one.
--
William A. Mahaffey III

----------------------------------------------------------------------

"The M1 Garand is without doubt the finest implement of war
ever devised by man."
-- Gen. George S. Patton Jr.
Jon Radel
2016-08-03 04:18:49 UTC
Permalink
Post by William A. Mahaffey III
It has 8 1 TB HDD's in a ZFS unmirrored pool.
Do I understand this to mean you have 8 TB with no redundancy? Pretty
brutal if your data starts rotting away. :-(
Post by William A. Mahaffey III
Aug 1 23:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
My question is: Are these messages benign, or am I in the market for
more hardware ? *ANY* more questions, please ask. TIA & have a good one.
Somewhere between, YMMV, depends on your luck and the phase of the moon.
That's useless, so I'll expand:

I've read reasonable sounding commentary from people running very, very
large collections of hard drives that there is a high enough correlation
between this error and the drive going to heck sooner rather than later
that they take this as a sign to replace. [Can't find reference right now.]

I've read reasonable sounding commentary that by writing to the bad
sector(s) you can force the HD to reallocate the sectors (assuming
remaining capacity to do that) and then use ZFS redundancy and a good
scrub to recover the data you trashed with dd (might not apply to you).
Then there's a good chance that all will be well indefinitely. See
https://dekoder.wordpress.com/2014/10/08/fixing-freenas-currently-unreadable-pending-sectors-error/
for an example of this school of thought. Actually, given that the
messages stopped again, I suspect reallocation already happened, but a
smartd test should tell you for certain. (You didn't have a scrub
running between 8/2 03:39 and 04:09 by any chance?)

Personally, so long as only one drive in my pool is doing this I tend to
replace but not rush. I'd rush more if the sector count started
creeping up, up, up.
--
--Jon Radel
***@radel.com
David Christensen
2016-08-03 03:59:19 UTC
Permalink
... 8 1 TB HDD's in a ZFS unmirrored pool. ...
Aug 1 23:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 1 23:39:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 00:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 00:39:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 01:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 01:39:58 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 02:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 02:39:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 03:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 03:39:58 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
... The
HDD's were brand new 2.5" SATA3 7200 RPM HGST's, which I have had good
luck with in other builds (I have about 25 in service, all purchased new
from NewEgg for each build). Case is well ventilated, drives are cool,
etc. ...
My question is: Are these messages benign, or am I in the market for
more hardware ? *ANY* more questions, please ask. TIA & have a good one.
HBA's can fail.


Cables can fail; usually, it's the connection/ connector. Re-seating
cables can fix problems. Connections should feel solid. If not,
install a new cable and recycle the old one.


I assume there is some utility on BSD to read and pretty-print the
smartd information (?).


I would download the HGST Windows Drive Fitness Test (WinDFT) and use it
to test all the drives. Ideally, by removing your BSD system drive,
installing a Windows system drive, installing WinDFT, and running WinDFT
before, during, and after you mess with the hardware.


David
Matthew Seaman
2016-08-03 12:18:37 UTC
Permalink
Post by William A. Mahaffey III
My question is: Are these messages benign, or am I in the market for
more hardware ? *ANY* more questions, please ask. TIA & have a good one.
It doesn't look particularly good. Do you run the smartd selftests at
all? Worth giving that a go --

# smartctl -t long /dev/ada5

This can be done while the disk is in use without upsetting anything.
It will scan the disk for unreadable areas. The disk does have a number
of spare sectors it can use instead of any broken ones, but it generally
needs to see a failed write to the affected area to trigger the
substitution mechanism. Once you run out of substitute sectors, the
disk is basically toast and should be replaced. With modern drives,
seeing that mechanism in use at all typically means the drive is on the
downward spiral and you should plan on replacing it PDQ.

Cheers,

Matthew
William A. Mahaffey III
1970-01-01 00:00:00 UTC
Permalink
Post by Matthew Seaman
Post by William A. Mahaffey III
My question is: Are these messages benign, or am I in the market for
more hardware ? *ANY* more questions, please ask. TIA & have a good one.
It doesn't look particularly good. Do you run the smartd selftests at
all? Worth giving that a go --
# smartctl -t long /dev/ada5
This can be done while the disk is in use without upsetting anything.
It will scan the disk for unreadable areas. The disk does have a number
of spare sectors it can use instead of any broken ones, but it generally
needs to see a failed write to the affected area to trigger the
substitution mechanism. Once you run out of substitute sectors, the
disk is basically toast and should be replaced. With modern drives,
seeing that mechanism in use at all typically means the drive is on the
downward spiral and you should plan on replacing it PDQ.
Cheers,
Matthew
Thanks, I kicked that off just now, it says it will take 169 min. to
complete, so we'll see. No more new messages overnight, maybe my luck is
good ....
--
William A. Mahaffey III

----------------------------------------------------------------------

"The M1 Garand is without doubt the finest implement of war
ever devised by man."
-- Gen. George S. Patton Jr.
William A. Mahaffey III
1970-01-01 00:00:00 UTC
Permalink
Post by David Christensen
... 8 1 TB HDD's in a ZFS unmirrored pool. ...
Aug 1 23:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 1 23:39:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 00:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 00:39:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 01:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 01:39:58 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 02:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 02:39:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 03:09:59 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
Aug 2 03:39:58 devbox smartd[835]: Device: /dev/ada5, 8 Currently
unreadable (pending) sectors
... The
HDD's were brand new 2.5" SATA3 7200 RPM HGST's, which I have had good
luck with in other builds (I have about 25 in service, all purchased new
from NewEgg for each build). Case is well ventilated, drives are cool,
etc. ...
My question is: Are these messages benign, or am I in the market for
more hardware ? *ANY* more questions, please ask. TIA & have a good one.
HBA's can fail.
No separate HBA, 8 SATA3 slots on the mbd. Definitely hope that's *NOT*
the problem :-/ ....
Post by David Christensen
Cables can fail; usually, it's the connection/ connector. Re-seating
cables can fix problems. Connections should feel solid. If not,
install a new cable and recycle the old one.
I assume there is some utility on BSD to read and pretty-print the
smartd information (?).
No, I wrote a small script to do that ....
Post by David Christensen
I would download the HGST Windows Drive Fitness Test (WinDFT) and use it
to test all the drives. Ideally, by removing your BSD system drive,
installing a Windows system drive, installing WinDFT, and running WinDFT
before, during, and after you mess with the hardware.
David
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
No separate system drive, ZFS root as per wiki. That last option sounds
destructive, right ? Any other options short of that ? Thanks & TIA ....
--
William A. Mahaffey III

----------------------------------------------------------------------

"The M1 Garand is without doubt the finest implement of war
ever devised by man."
-- Gen. George S. Patton Jr.
Brandon J. Wandersee
2016-08-03 14:00:38 UTC
Permalink
Post by Jon Radel
I've read reasonable sounding commentary from people running very, very
large collections of hard drives that there is a high enough correlation
between this error and the drive going to heck sooner rather than later
that they take this as a sign to replace. [Can't find reference right now.]
While there's no way to know from the error message alone just what will
happen to the disk in the coming days, the general reasoning is this:
sectors are not physically segregated. They all sit on the same
platter. Several bad sectors occuring in a short period might be a sign
of physical fault in the platter, and if that fault is real then stress
from the platter spinning will likely cause that fault to spread. So
some people conclude that the appearance of several bad sectors in a
short period should just be a signal to replace the disk immediately.
--
:: Brandon J. Wandersee
:: ***@gmail.com
:: --------------------------------------------------
:: 'The best design is as little design as possible.'
:: --- Dieter Rams ----------------------------------
Jon Radel
2016-08-03 14:09:32 UTC
Permalink
Post by Brandon J. Wandersee
Post by Jon Radel
I've read reasonable sounding commentary from people running very, very
large collections of hard drives that there is a high enough correlation
between this error and the drive going to heck sooner rather than later
that they take this as a sign to replace. [Can't find reference right now.]
While there's no way to know from the error message alone just what will
sectors are not physically segregated. They all sit on the same
platter. Several bad sectors occuring in a short period might be a sign
of physical fault in the platter, and if that fault is real then stress
from the platter spinning will likely cause that fault to spread. So
some people conclude that the appearance of several bad sectors in a
short period should just be a signal to replace the disk immediately.
If I remember the discussion well enough (sad that I can't find it) my
use of "correlation" was precise. They actually manage enough drives
(thousands) and kept enough records to allow for statistical analysis
which indicate that this smartd error correlates very well with failure
within [I wish I could remember] timeframe.

Do please excuse the utter lack of footnotes. :-(
--
--Jon Radel
***@radel.com
Arthur Chance
2016-08-03 15:59:13 UTC
Permalink
Post by Jon Radel
Post by Brandon J. Wandersee
Post by Jon Radel
I've read reasonable sounding commentary from people running very, very
large collections of hard drives that there is a high enough correlation
between this error and the drive going to heck sooner rather than later
that they take this as a sign to replace. [Can't find reference right now.]
While there's no way to know from the error message alone just what will
sectors are not physically segregated. They all sit on the same
platter. Several bad sectors occuring in a short period might be a sign
of physical fault in the platter, and if that fault is real then stress
from the platter spinning will likely cause that fault to spread. So
some people conclude that the appearance of several bad sectors in a
short period should just be a signal to replace the disk immediately.
If I remember the discussion well enough (sad that I can't find it) my
use of "correlation" was precise. They actually manage enough drives
(thousands) and kept enough records to allow for statistical analysis
which indicate that this smartd error correlates very well with failure
within [I wish I could remember] timeframe.
Do please excuse the utter lack of footnotes. :-(
I think everyone is probably thinking of Backblaze. This is their latest
summary of drive statistics

https://www.backblaze.com/blog/hard-drive-failure-rates-q2-2016/

And this is their take on which SMART metrics matter

https://www.backblaze.com/blog/hard-drive-smart-stats/
--
Moore's Law of Mad Science: Every eighteen months, the minimum IQ
necessary to destroy the world drops by one point.
Ken Moffat
2016-08-03 16:39:37 UTC
Permalink
Post by William A. Mahaffey III
Post by Matthew Seaman
Post by William A. Mahaffey III
My question is: Are these messages benign, or am I in the market for
more hardware ? *ANY* more questions, please ask. TIA & have a good one.
It doesn't look particularly good. Do you run the smartd selftests at
all? Worth giving that a go --
# smartctl -t long /dev/ada5
This can be done while the disk is in use without upsetting anything.
It will scan the disk for unreadable areas. The disk does have a number
of spare sectors it can use instead of any broken ones, but it generally
needs to see a failed write to the affected area to trigger the
substitution mechanism. Once you run out of substitute sectors, the
disk is basically toast and should be replaced. With modern drives,
seeing that mechanism in use at all typically means the drive is on the
downward spiral and you should plan on replacing it PDQ.
Cheers,
Matthew
Thanks, I kicked that off just now, it says it will take 169 min. to
complete, so we'll see. No more new messages overnight, maybe my luck is
good ....
I agree with Matthew, and since your custom script seemed to ignore
all the items which I regard as relevant, after the long test has
finished run smartctl -a on that drive and look at things like
Reallocated Sector Count and Pending (if that is still present).
plus any other error fields.

ĸen
--
`I shall take my mountains', said Lu-Tze. `The climate will be good
for them.' -- Small Gods
Andrea Venturoli
2016-08-03 18:24:53 UTC
Permalink
Post by William A. Mahaffey III
Post by David Christensen
I would download the HGST Windows Drive Fitness Test (WinDFT) and use it
to test all the drives. Ideally, by removing your BSD system drive,
installing a Windows system drive, installing WinDFT, and running WinDFT
before, during, and after you mess with the hardware.
That last option sounds
destructive, right ? Any other options short of that ? Thanks & TIA ....
You can try with Ultimate Boot CD: you can boot HGST DFT from there
without the hassle of going through Windows.

You'll run the test, find the drive faulty, try repair, test again:
that's (usually) not destructive.

In case the last test gives positive results, watch the disk closely:
sometimes it will go on for years, sometimes you'll start getting those
error again within days.

If that's the case, replace it.

bye
William A. Mahaffey III
1970-01-01 00:00:00 UTC
Permalink
Post by Matthew Seaman
Post by William A. Mahaffey III
My question is: Are these messages benign, or am I in the market for
more hardware ? *ANY* more questions, please ask. TIA & have a good one.
It doesn't look particularly good. Do you run the smartd selftests at
all? Worth giving that a go --
# smartctl -t long /dev/ada5
This can be done while the disk is in use without upsetting anything.
It will scan the disk for unreadable areas. The disk does have a number
of spare sectors it can use instead of any broken ones, but it generally
needs to see a failed write to the affected area to trigger the
substitution mechanism. Once you run out of substitute sectors, the
disk is basically toast and should be replaced. With modern drives,
seeing that mechanism in use at all typically means the drive is on the
downward spiral and you should plan on replacing it PDQ.
Cheers,
Matthew
OK, after completion of the above command, I got the following:

[***@devbox, /etc, 2:18:44pm] 372 % smartctl -l selftest /dev/ada5
smartctl 6.4 2015-06-04 r4109 [FreeBSD 9.3-RELEASE-p33 amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 6671
167940256

[***@devbox, /etc, 2:18:57pm] 373 %


What does this mean ?
--
William A. Mahaffey III

----------------------------------------------------------------------

"The M1 Garand is without doubt the finest implement of war
ever devised by man."
-- Gen. George S. Patton Jr.
William A. Mahaffey III
1970-01-01 00:00:00 UTC
Permalink
Post by Ken Moffat
Post by William A. Mahaffey III
Post by Matthew Seaman
Post by William A. Mahaffey III
My question is: Are these messages benign, or am I in the market for
more hardware ? *ANY* more questions, please ask. TIA & have a good one.
It doesn't look particularly good. Do you run the smartd selftests at
all? Worth giving that a go --
# smartctl -t long /dev/ada5
This can be done while the disk is in use without upsetting anything.
It will scan the disk for unreadable areas. The disk does have a number
of spare sectors it can use instead of any broken ones, but it generally
needs to see a failed write to the affected area to trigger the
substitution mechanism. Once you run out of substitute sectors, the
disk is basically toast and should be replaced. With modern drives,
seeing that mechanism in use at all typically means the drive is on the
downward spiral and you should plan on replacing it PDQ.
Cheers,
Matthew
Thanks, I kicked that off just now, it says it will take 169 min. to
complete, so we'll see. No more new messages overnight, maybe my luck is
good ....
I agree with Matthew, and since your custom script seemed to ignore
all the items which I regard as relevant, after the long test has
finished run smartctl -a on that drive and look at things like
Reallocated Sector Count and Pending (if that is still present).
plus any other error fields.
ĸen
OK:


[***@devbox, /etc, 2:22:57pm] 373 % smartctl -a /dev/ada5
smartctl 6.4 2015-06-04 r4109 [FreeBSD 9.3-RELEASE-p33 amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: HGST Travelstar 7K1000
Device Model: HGST HTS721010A9E630
Serial Number: JR10046P1E69HN
LU WWN Device Id: 5 000cca 7dcd41800
Firmware Version: JB0OA3J0
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Aug 3 14:23:03 2016 MCDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection:
Disabled.
Self-test execution status: ( 121) The previous self-test completed
having
the read element of the test
failed.
Total time to complete Offline
data collection: ( 45) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 169) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control
supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 091 091 062 Pre-fail
Always - 3538944
2 Throughput_Performance 0x0005 100 100 040 Pre-fail
Offline - 0
3 Spin_Up_Time 0x0007 100 100 033 Pre-fail
Always - 2
4 Start_Stop_Count 0x0012 100 100 000 Old_age
Always - 5
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail
Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail
Always - 0
8 Seek_Time_Performance 0x0005 100 100 040 Pre-fail
Offline - 0
9 Power_On_Hours 0x0012 085 085 000 Old_age
Always - 6678
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 5
191 G-Sense_Error_Rate 0x000a 100 100 000 Old_age
Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age
Always - 2
193 Load_Cycle_Count 0x0012 034 034 000 Old_age
Always - 668237
194 Temperature_Celsius 0x0002 230 230 000 Old_age
Always - 26 (Min/Max 14/31)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age
Always - 8
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age
Always - 0
223 Load_Retry_Count 0x000a 100 100 000 Old_age
Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 6671
167940256

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[***@devbox, /etc, 2:23:04pm] 374 %


Reallocated sectors zero (0) .... so where am I ? Thanks & TIA & have a
good one.
--
William A. Mahaffey III

----------------------------------------------------------------------

"The M1 Garand is without doubt the finest implement of war
ever devised by man."
-- Gen. George S. Patton Jr.
Matthew Seaman
2016-08-03 20:13:00 UTC
Permalink
Post by William A. Mahaffey III
What does this mean ?
That there's a bad spot on the disk, which may also mean that you've got
a corrupted filesystem -- depends if the bad spot was in use by zfs or
not. 'zpool scrub' should tell you if the filesystem is corrupted.

Time to replace the drive. You should be able to convert the vdev that
contains the failing drive into a mirror temporarily, and sync the data
without downtime beyond maybe a few reboots to install the new disk
(assuming you have space to plug the new drive in without unplugging any
of the old ones). Failing that, you're going to need to rebuild the
zpool from scratch and restore your data from backup.

Also, the fact that you have how ever many terabytes of data with no
resilience just makes me feel on edge -- and it's not even my data.

Strongly recommend rebuilding your zpool as a RAIDZ of 8 drives -- yes,
you'll end up with less usable space, but you and your data will survive
failure of a drive and a 'zpool scrub' will be able to fix things even
if a bad spot on one drive has scrambled some of your data.
William A. Mahaffey III
1970-01-01 00:00:00 UTC
Permalink
Post by Matthew Seaman
Post by William A. Mahaffey III
What does this mean ?
That there's a bad spot on the disk, which may also mean that you've got
a corrupted filesystem -- depends if the bad spot was in use by zfs or
not. 'zpool scrub' should tell you if the filesystem is corrupted.
Can I do that 'zpool scrub' live ?
Post by Matthew Seaman
Time to replace the drive. You should be able to convert the vdev that
contains the failing drive into a mirror temporarily, and sync the data
without downtime beyond maybe a few reboots to install the new disk
(assuming you have space to plug the new drive in without unplugging any
of the old ones). Failing that, you're going to need to rebuild the
zpool from scratch and restore your data from backup.
No spare SATA slots :-/ ....
Post by Matthew Seaman
Also, the fact that you have how ever many terabytes of data with no
resilience just makes me feel on edge -- and it's not even my data.
Strongly recommend rebuilding your zpool as a RAIDZ of 8 drives -- yes,
you'll end up with less usable space, but you and your data will survive
failure of a drive and a 'zpool scrub' will be able to fix things even
if a bad spot on one drive has scrambled some of your data.
I was/am already thinking along those lines, w/ 1 complication. I have
another box (NetBSD 6.1.5) w/ a RAID5 that I wound up building w/
mis-aligned disk/RAID blocks in spite of a fair amount of effort to
avoid that. I/O writes are horrible, 15-20 MB/s. My understanding is
that RAIDZn is like RAID5 in many ways & that you always want 2^n+1
(3,5,9, ...) drives in a RAID5 to mitigate those misalignments,
presumably in a RAIDZ also. Is that so w/ RAIDZ as well ? If so, I lose
more than a small amount of total storage, which is why I went as I did
when I built the box whenever that was.
--
William A. Mahaffey III

----------------------------------------------------------------------

"The M1 Garand is without doubt the finest implement of war
ever devised by man."
-- Gen. George S. Patton Jr.
Ken Moffat
2016-08-03 20:35:17 UTC
Permalink
Post by Ken Moffat
I agree with Matthew, and since your custom script seemed to ignore
all the items which I regard as relevant, after the long test has
finished run smartctl -a on that drive and look at things like
Reallocated Sector Count and Pending (if that is still present).
plus any other error fields.
ĸen
(pruning, and unwrapping)
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED > WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 8
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
At this point it hasn't managed to read 8 sectors, I assume it only
reallocates afteri either succeding to read or giving up.

I had this once on a RAID-1, I swapped the drive out as soon as it
was convenient, and meanwhile the other drive in the mirror provided
the data. Before that, on daily tests the position of the first
failure did change once, so I think mine managed to read one of the
failing sectors.

A couple of years ago, 1TB spinning drives were the smallest
commonly available from several retail suppliers - and not
particularly reliable. I think I've had two 1TB drives (different
brands) fail in the last 2 years. So, as with all drives, expect
failures.

ĸen
--
`I shall take my mountains', said Lu-Tze. `The climate will be good
for them.' -- Small Gods
Brandon J. Wandersee
2016-08-04 00:50:33 UTC
Permalink
Post by William A. Mahaffey III
Post by Matthew Seaman
Post by William A. Mahaffey III
What does this mean ?
That there's a bad spot on the disk, which may also mean that you've got
a corrupted filesystem -- depends if the bad spot was in use by zfs or
not. 'zpool scrub' should tell you if the filesystem is corrupted.
Can I do that 'zpool scrub' live ?
Ordinarily, yes. A scrub might lower performance a little while it's
underway, but it's safe to use the system while you do it. However,
depending on how much data you have on that pool, a scrub can take a
long time to finish. A scrub of the measely ~1.8Tb on my pool takes the
better part of five hours to complete. The risk I would worry about in this
particular situation is whether leaving the system running long enough
for a scrub to complete would result in more sectors on the disk failing,
in an area already passed over by the scrub. If that happened, you'd
wind up with more corrupted files (assuming there already are corrupted
files in the first place due to a filesystem problem). Finding and
fixing those would mean running another scrub, taking up twice the time.

Ordinarily, then, I'd recommend running the scrub after replacing the
disk. In this particular situation, if you want to try get out of this
with absolutely no corrupted files, then if at all possible use `zfs
send | zfs receive` to clone the existing pool to a new pool on another
machine, and run the scrub there. The problem is that if you intend to
recreate your current pool in a RAIDZ layout you'll need to back up your
data, and if you back up your data using rsync (as you have been) and
then restore it to the new pool using rsync, the checksums for the
previously good files will be lost and the corrupted files will be given
new checksums. ZFS won't realize they're corrupted. Bear in mind,
though, that none of this is to say that any of your files currently are
corrupted or will be corrupted. This is just a "best approach to
worst case" as I see it.
Post by William A. Mahaffey III
I was/am already thinking along those lines, w/ 1 complication. I have
another box (NetBSD 6.1.5) w/ a RAID5 that I wound up building w/
mis-aligned disk/RAID blocks in spite of a fair amount of effort to
avoid that. I/O writes are horrible, 15-20 MB/s. My understanding is
that RAIDZn is like RAID5 in many ways & that you always want 2^n+1
(3,5,9, ...) drives in a RAID5 to mitigate those misalignments,
presumably in a RAIDZ also. Is that so w/ RAIDZ as well ? If so, I lose
more than a small amount of total storage, which is why I went as I did
when I built the box whenever that was.
I don't have enough knowledge/experience with RAIDZ to answer your
specific questions, but if nothing else you could still combine the disks
into mirrored vdevs, which are more flexible than RAIDZ, but slightly
less robust. You'd have a maximum of half the storage space and more
redundancy than you do now (though significantly less redundancy than
with a RAIDZ setup).
--
:: Brandon J. Wandersee
:: ***@gmail.com
:: --------------------------------------------------
:: 'The best design is as little design as possible.'
:: --- Dieter Rams ----------------------------------
William A. Mahaffey III
1970-01-01 00:00:00 UTC
Permalink
Post by Brandon J. Wandersee
Post by William A. Mahaffey III
Post by Matthew Seaman
Post by William A. Mahaffey III
What does this mean ?
That there's a bad spot on the disk, which may also mean that you've got
a corrupted filesystem -- depends if the bad spot was in use by zfs or
not. 'zpool scrub' should tell you if the filesystem is corrupted.
Can I do that 'zpool scrub' live ?
Ordinarily, yes. A scrub might lower performance a little while it's
underway, but it's safe to use the system while you do it. However,
depending on how much data you have on that pool, a scrub can take a
long time to finish. A scrub of the measely ~1.8Tb on my pool takes the
better part of five hours to complete. The risk I would worry about in this
particular situation is whether leaving the system running long enough
for a scrub to complete would result in more sectors on the disk failing,
in an area already passed over by the scrub. If that happened, you'd
wind up with more corrupted files (assuming there already are corrupted
files in the first place due to a filesystem problem). Finding and
fixing those would mean running another scrub, taking up twice the time.
Ordinarily, then, I'd recommend running the scrub after replacing the
disk. In this particular situation, if you want to try get out of this
with absolutely no corrupted files, then if at all possible use `zfs
send | zfs receive` to clone the existing pool to a new pool on another
machine, and run the scrub there. The problem is that if you intend to
recreate your current pool in a RAIDZ layout you'll need to back up your
data, and if you back up your data using rsync (as you have been) and
then restore it to the new pool using rsync, the checksums for the
previously good files will be lost and the corrupted files will be given
new checksums. ZFS won't realize they're corrupted. Bear in mind,
though, that none of this is to say that any of your files currently are
corrupted or will be corrupted. This is just a "best approach to
worst case" as I see it.
Post by William A. Mahaffey III
I was/am already thinking along those lines, w/ 1 complication. I have
another box (NetBSD 6.1.5) w/ a RAID5 that I wound up building w/
mis-aligned disk/RAID blocks in spite of a fair amount of effort to
avoid that. I/O writes are horrible, 15-20 MB/s. My understanding is
that RAIDZn is like RAID5 in many ways & that you always want 2^n+1
(3,5,9, ...) drives in a RAID5 to mitigate those misalignments,
presumably in a RAIDZ also. Is that so w/ RAIDZ as well ? If so, I lose
more than a small amount of total storage, which is why I went as I did
when I built the box whenever that was.
I don't have enough knowledge/experience with RAIDZ to answer your
specific questions, but if nothing else you could still combine the disks
into mirrored vdevs, which are more flexible than RAIDZ, but slightly
less robust. You'd have a maximum of half the storage space and more
redundancy than you do now (though significantly less redundancy than
with a RAIDZ setup).
When you say mirrored vdevs, are you alluding to a RAID10-ish setup ? My
zpool man pages says that's a nogo for me (FreeBSD 9.3R), maybe for
newer .... My various boxen do a fair amount of stuff overnight,
automated. I will try the 'zpool scrub' tomorrow during the day with the
machine deliberately lightly loaded.
--
William A. Mahaffey III

----------------------------------------------------------------------

"The M1 Garand is without doubt the finest implement of war
ever devised by man."
-- Gen. George S. Patton Jr.
David Christensen
2016-08-04 04:24:05 UTC
Permalink
Post by William A. Mahaffey III
No separate HBA, 8 SATA3 slots on the mbd. Definitely hope that's *NOT*
the problem :-/ ....
Agreed, but bear in mind that it is a possibility if you exhaust the
other possibilities and problems persist.
Post by William A. Mahaffey III
No separate system drive, ZFS root as per wiki.
I find it useful to have my O/S on a small, fast, dedicated drive. I
got lucky, and bought a used 16 GB SSD on Amazon for $10. I installed
FreeBSD 10.1 on that with encrypted ZFS root using the installer, and
then dd'd the image to a chosen HDD.


It might be possible to install Windows onto an external HDD/SSD. Note
that you can install Windows and play with it for 30 days without
activating.


If you can't test the 8 @ 1 TB HDD's in-place, then you'll need to move
them into a Windows machine with HGST Windows Drive Fitness Test
(WinDFT). Make sure that you record HBA port number, cable serial
number (I number mine with Sharpie), and drive serial number as you pull
the drives, so you can restore things exactly after testing. In the
Windows machine, make sure to connect to an HBA, not a USB, Firewire,
eSATA, or other external docking bay. Test each drive with its matching
cable.
Post by William A. Mahaffey III
[HGST Windows Drive Fitness Test (WinDFT)] sounds
destructive, right ? Any other options short of that ? Thanks & TIA ....
Manufacturer diagnostic utility disks usually include tests that don't
modify the on-disk data. But, some include a utility to wipe (HDD) or
Secure Delete (SSD) the disk; which would definitely be destructive to
your data.


In any case, make sure you have backups.


David
David Christensen
2016-08-04 04:49:26 UTC
Permalink
Post by William A. Mahaffey III
Can I do that 'zpool scrub' live ?
I try to scrub when the machine is not in use.
Post by William A. Mahaffey III
No spare SATA slots :-/ ....
How about expansion slots -- e.g. PCI, PCIe?
Post by William A. Mahaffey III
Post by Matthew Seaman
Also, the fact that you have how ever many terabytes of data with no
resilience just makes me feel on edge -- and it's not even my data.
+1
Post by William A. Mahaffey III
Post by Matthew Seaman
Strongly recommend rebuilding your zpool as a RAIDZ of 8 drives -- yes,
you'll end up with less usable space, but you and your data will survive
failure of a drive and a 'zpool scrub' will be able to fix things even
if a bad spot on one drive has scrambled some of your data.
I was/am already thinking along those lines, w/ 1 complication. I have
another box (NetBSD 6.1.5) w/ a RAID5 that I wound up building w/
mis-aligned disk/RAID blocks in spite of a fair amount of effort to
avoid that. I/O writes are horrible, 15-20 MB/s. My understanding is
that RAIDZn is like RAID5 in many ways & that you always want 2^n+1
(3,5,9, ...) drives in a RAID5 to mitigate those misalignments,
presumably in a RAIDZ also. Is that so w/ RAIDZ as well ? If so, I lose
more than a small amount of total storage, which is why I went as I did
when I built the box whenever that was.
How much data do you have on the 8 @ 1 TB machine?


How are you doing backups? Archives? Images?


How much data are you willing to lose?


What is your usage environment -- e.g. student/ hobbyist, SOHO, SMB,
large corporate?


What are your budget and schedule?


David

Loading...