In 2013 Backblaze saw ~14% avg failure rate at 3 years 3 months. Drive useful life: 4 years, roughly.
In 2021, Backblaze saw ~14% failure rate at 7 years 9 months. Drive useful life: about 6 years.
That gave us what was a stable-seeming, old "replace drives at 5 years, or you're getting risky" rule of thumb (unless you prefer to replace on failure, of course).
In 2025, Backblaze saw ~5% failure rate at 10 years 3 months. Drive useful life: ~10 years, roughly!
I wonder if any of the improvements in failure rates were caused by changes in how backblaze operates the drives? Like, have they learned things about cooling or improved write leveling or better power management or something.
Could be better controllers, or better driver firmware, or something.
Just curious where in the chain the improvements come from.
I find the upgrade quite surprising. I would have expected them to be mature already in 2013. Or maybe there is change in selection. Blackblaze did go for cheap drives before. But with most users moving to SSD as default drive maybe overall HDDs have moved to more reliable segment. As in lower quality drives are just not made for desktops in mass volumes.
Part of it was drive quality. The 2011 flood that impacted Western Digital[1] caused them to buy a lot of lower-quality Seagate drives which affected their stats for years.
But I think part of it is the modern densities have less tolerance for manufacturing issues, so if the disks work they work well. Kinda like the Mars rovers which only had to work 90 days but they really had to work, so they ended up working for years.
The term "bathtub curve" came from the fact that typically hard disks had a lot of early deaths in the first year right out of the factory, hence a graph of hard drive death age would have a high side on the left (brand new drives dying) and a high side on the right (old drives dying), and during the average lifetime the drives were reliable, so the curve was shaped like a bathtub (high sides, flat bottom).
This lead to a lot of people doing things like intensive burn-in cycles for a week to test new drives before putting them in their machines.
When there are no longer early deaths (the "left" side of the bathtub), it's just "old things wear out" which doesn't have a special word for it.
Well, until the tail of the bathtub, which still seems to be there, but more of a wide dyke rather then a sharp and tall bathtub-edge. Which is father away than it used to be.
Drives still have a meaningful average working life is what I see in te data. There's just barely any infant mortality left.
I remember back in the late nineties / early 2000s drive failures really were pretty common. During my childhood I think I had three. I've never had a catastrophic failure since then, really came a long way.
My father was deep into photography in that era and digitized all his photos. He was understandably paranoid about drive failures. The solution? ZIP disks everywhere! Cue massive family stress when the disk containing his competition entry was nowhere to be found and definitely in one of the kids rooms. It was, of course, found 3 days later in the glove box of his truck (thank you for reading my lowercase-t trauma dump).
> I remember back in the late nineties / early 2000s drive failures really were pretty common.
Yup. And most people wouldn't bother but typically you could RMA HDDs: the terms were quite generous. I was in Europe and did learn, as a teenager, how to RMA HDDs to get shiny replacements (IIRC for example for Seagate HDDs I'd send them to the Netherlands and then after a few weeks I'd get a new one by the mail).
This is broadly consistent with my own experience at smaller scale. Out of a dozen or so 2TB HGST drives in use for 10+ years (mostly Ultrastar SATA, with a few Deskstars that may or may not be mechanically identical), I've had only one failure, and that was of a bare drive that was outside my control, so I have no idea how it was handled. And this was in a dusty environment with no air conditioning, not a datacenter.
Likewise with SSDs. Given their slightly more write intensive than ordinary desktop workloads, I'm reasonably confident that the capacitors in my 10-year-old enterprise SSDs (HGST SAS with Intel flash) will fail before the flash.
> That gave us what was a stable-seeming, old "replace drives at 5 years, or you're getting risky" rule of thumb (unless you prefer to replace on failure, of course).
That's not exactly what this means. This is like life expectancy at birth vs life expectancy at current age. (which is the thing where people act like "only kings got old" in the past, when while there is some truth to it also doesn't mean "you would be dead by 30" when you already reached adulthood. The biggest difference we got regarding life expectancy at birth by far was doctors and nurses cleaning their hands, followed by things like vaccines for childhood diseases)
Manufacturing errors are likely to hit early. So replacing a drive means you might replace a perfectly healthy drive with one that will eat your data.
Just use SMART and you have a way better insight of how well your drive does than any such advice.
And also you likely won't run your drives in the same conditions as Backblaze, so things might vary in both directions.
There is shipping, storage, running conditions, series, manufacturing, and many variables. So you have a way better understanding of your drive than what Backblaze can provide if you just look at your drive's SMART data and run a self test.
I think that's because there is differences in comparison. When an HDD reached 5 (or X) years it's much more likely to live to 10 (or y) years than one that died at 2 (or Z) years already. ;)
There is both the mathematical truth to it, but also the physical/chemical one. If something eg. deformed or had a slight manufacturing mistake the wear will be quicker. If everything works perfectly and eg. there are little vibrations (can be everything from earthquake to noise) then HDDs also tend to live longer and run faster. Of course the shipping process also can take its toll and vary greatly.
I think in the world of computers it's all to easy to forget that things largely aren't binary and "logically perfect". Even with advancements in manufacturing and materials drives are physical things in the real world influenced by many things inside and outside of the manufacturer's and user's control.
Hard drives.... I remember those things... didn't I used to use them before the Great HDD and SSD Extinction of 2025-2026?
Seriously, I had to replace 2 hard drives in my NAS this year like a bad luck lottery and my hard drives have doubled in price. I was terrified that my NVMe was failing as well.
As I see it, the prices are basically what they were in 2020. Not ideal, but survivable. I had a hard drive in 2020, and probably so did you. Since 2020, they dropped in half, and then they doubled again.
My take on hard drive economics is that the cost of an HDD is relatively constant, modulo the platter count, while capacity increases over time, as the components are pretty much the same in each generation.
There's one exception to this - HAMR added additional components, and a corresponding increase in BOM cost, but there's probably a learning curve on this.
However the recent AI data center boom is probably hitting HDDs in two ways - (a) general demand for anything datacenter-related, and (b) high DRAM prices, with most HDDs containing 256-512MB of DDR3. (I haven't physically examined the newest drives - they may be DDR4 and go beyond 512MB; also the 40TB supposedly includes NAND flash, adding yet more cost)
It's not the economics of manufacturing rust-coated glass disks and more than chips have the economics of molten sand. Both are actually the economics of very precise devices requiring very specialised knowledge and equipment to make that can't be readily scaled up.
For the second link, you'll have to ignore the horribly tasteless AI slop image for the site background (someone should e-mail them about it or something), the site itself is legit, bought some drives from them: https://blog.kronis.dev/blog/buying-some-drives-from-datablo...
Those have also shot up though. The past couple years is the first time I can recall cost per GB increasing rather than decreasing in a sustained manner. And it's been all forms of storage.
Never buy a recertified drive, talking from my own experience, my recertified Seagate Exos X16 failed exactly after a year. They are very cheap for a reason.
It failed for a reason before, so it will likely fail again before the end of its usual lifecycle.
This has not been my experience at all. All of my NAS drives are recertified enterprise models. I've never had one fail. (But then I've only had a single spinning disk fail on me ever. Generally I retire them because they become too small to justify occupying a slot. Slots are scarce.)
A really long time (at least by residential standards). Close to 24/7 for many years.
If a single HDD failing is traumatic then you're doing something wrong. At absolute minimum you should be in raid1. Backups should amount to snapshots that can be restored in a single step.
> If a single HDD failing is traumatic then you're doing something wrong. At absolute minimum you should be in raid1. Backups should amount to snapshots that can be restored in a single step.
I wasn't storing anything important on it anyway, mostly movies ripped from Blu-rays. So its failure wasn't traumatic. However I expected it to last at least three years, but it died after exactly 365 days of use. The seller refused to send a replacement, and legally speaking, they were in the right.
Melodramatic mutch? The first new IDE HDD I ever bought died within a week. The first new USB HDD I ever bought was dead on arrival. My first server SSD became read only after a year.
Better telling everyone not to buy this things. /s
I usually don't spite people or companies but in this case I hope all these AI slop bullshit machines fail & fail hard.
So that we can get hardware at sane prices for actually useful thing. And to serve as a warning to never ever do such a stupidly disruptive shit ever again.
It's not the AI tech itself, it's that a select set of people have been given far too much money - anything they decide to invest in could be like this. AI isn't even the first - GPU prices were driven up by Ethereum miners, and just like AI, Nvidia flirted with exiting the customer sector and only selling to miners.
I've been tracking 12TB HDD prices for replacing a faulty one in my NAS for the past 6 months, I bought 4x 12TB HDD in 2024 for a total of 13k SEK (around 1300 USD) and right now a single one of the same brand/model is selling for 6750-7000 SEK (~675-700 USD).
It is 2x the price of just 2 years ago, it's too bad.
Something that's interested me is how hard drive capacities have continuously gone up, but read and write speeds have not really changed. To the point where it would take multiple days to read these new high capacity drives.
I wonder what the use case for them is, they aren't ideal for RAID as they would take far too long to rebuild the array in a drive failure, but they also aren't great for long term cold storage. Perhaps they are ideal for periodic backups where you want to take a daily 1TB backup of a server or CCTV where you don't really care about backing up the data as it's all deleted on schedule anyway.
It doesn't matter much to me if the overall process of replacing a dead drive can be done in 40 minutes or if it drags out to 40 hours. The storage system is still alive, still available for reading and writing, and still redundant for that entire time. It can take as long as it takes.
My break-glass-in-case-of-emergency offsite backups are done with restic over the network, so I only had to send 1 initial copy. That first copy took quite a long time. But restic is pretty clever about how it handles things; all subsequent daily backups are what might best be described as incremental-ish. They contain only recent changes and are pretty small with the usual churn rate on this particular pile of data.
(This matters because it keeps the cost of keeping it running quite low, which means that backups actually happen instead of being just a good idea. :) )
---
I'd probably pick different methods if I had a real budget for this stuff, but even then: Why would I be motivated to care about how long it takes to replace a hard drive?
So move the bar up to raidz3 or the equivalent? This adds cost, of course, but enhancing redundancy isn't free.
It was something I kept in-mind when I made the decision to use raidz2 for my own purposes: It tolerates two concurrent drive failures before local data loss happens, which I feel is acceptable for my purposes. raidz3 moves that tolerance to three concurrent drive failures.
I mean: What's the alternative for storing X amount of data? A greater number of relatively small drives? That's expensive in its own ways, too.
And adding complexity in that way tends to nosedive metrics like MTBF: If a system relies upon one widget and this system has an MTBF of n, then a system that relies upon two such widgets has an MTBF of n/2.
If the drives all have the same failure rate, then: A system with raidz2 and 10x 10TB drives will tend experience more disk failures than a system with raidz2 and 6x 20TB drives. Both systems can store ~80TB of data, but the one with 10TB drives has a worse ratio of redundancy and includes more parts that will eventually fail.
KISS, etc. :)
> and the longer it takes the more likely another drive will fail during that time.
You mean the old light-weight RAID-5-esque fear? A drive fails and redundancy drops to zero. So the failed drive is replaced or the hot spare is rotated in (or whatever), and the rebuild process begins. Another drive crashes during the rebuild because of the stress of actually-being-read and now the data on the array is presumed to be trash.
That process seems to actually work more like this: Old data that hasn't been read for years finally gets accessed (because RAID rebuild), and that's when some other drive starts reporting errors. But those errors were already there; they just weren't detected until it was way too late.
It doesn't really seem to work that way with filesystems like ZFS. The contents, whether old and stale or brand new, can (and should) be read routinely and checksummed to confirm integrity as part of a regularly-performed scrub.
The weak drives that can't stand to be read will be discovered during this read-only exercise, but they were already dead. We just didn't know it until we ran a scrub. So we scrub early, and often.
That's the same as with RAID, except: With RAID, the same disk still dies in the same way. We just don't know it was dead until a recovery is in-process. RAID, as commonly practiced, can therefore function as a problem-multiplier in ways that other systems can avoid.
Spinning rust is still the only cost effective answer to redundant 10+ tb storage. If you have 4+ disks, use raid 6 or raid 10 and you get decent fault tolerance too. Raid 10 also increases write performance but it's a bit less robust imo and more expensive per gb.
As for speed it's mainly tied to rpm and the market decided no one wants higher failure rates / lower mtbf for marginal speed gains.
If you need speed and capacity, throw an ssd or two in front of a raid disk array for caching with primocache or whatever.
When you're regularly working with largely sequential datasets with sizes ranging from 100GB to >1TB that you're not processing at speeds in excess of 100MB/sec, hard drives are very much good enough.
Especially now that the same 4TB NVMe SSDs I was buying for $300 are now selling for over $1,000.
In my case, the source of the data is S3, so there's no need for redundancy, as the hard drive is effectively a local cache for cloud storage and the work isn't nearly so time-critical that an occasional failure would be more than a mild annoyance to a customer, if that.
Same for archival backups. Faster recovery time than Glacier Deep Archive, less cost over a period of years, simpler and less expensive than tape for modest volume.
I'm not doubting the usefulness of hard drives in general, just the ones above 20TB where it would take you most of a week to write to the drive. Hard drive fill times have been steadily going up to the point it becomes almost impractical to use these 30tb drives.
I guess if you have it all backed up in S3 it's fine, but if you put this thing in a NAS and one drive dies, your NAS is going to spend the whole week rebuilding where it's running at 100%, and another drive dying during this means data loss.
Can't you just not run a Raid 6 or whatever ? If it takes a week to rebuild but is just making a new redundant copy of the old failed drive then it doesn't seem like a huge deal ?
It depends? In something like a 800TB Ceph cluster a failure of a single 20TB drive is not felt at all if configured redundantly enough. Then it does not matter how long replacing it takes, only how many bucks each TB costs and/or how much electricity each drive needs.
If the cluster as whole can stomach the drive failures without losing data and is overall fast enough, there is no need to run many small drives where running fewer big drives is cheaper.
For one, home servers can go on maintenance mode (read only?) for a day or two, it will be fine most of the time.
In bigger applications, you have enough redundancy to allow one or two nodes to go down for a while. If a company had terabytes upon terabytes of critical data, having a multi-tiered storage would make sense, even of a full backup restore would take a days.on some tiers.
RAID 5 being too risky doesn't invalidate the whole concept. You can use RAID 6 with two parity drives instead of one, or something higher level like ZFS or Ceph. If you have a trustworthy backup and can tolerate an amount of downtime risk, you can even run RAID 0 or mergerfs on your main array. Many ways to arrange storage.
I imagine that there is a ton of data in the "would be nice to retain but not so critical as to warrant a constant backup" category. Alternatively stuff that requires large storage, but only occasional small writes, so incremental backups are feasible.
You use them for backups, data archiving, or object storage with triple parity RAID or erasure coding. The combined bandwidth of dozens to hundreds of magnetic hard drives is pretty decent.
Speeds do go up with density. Take an empty 4 TB drive and an empty 16 TB drive, and then write and read 2 TB of data to both of them. Since the platters on the 16 TB one are that much denser, you go over a lot more blocks in a single spin than the 4 TB one.
The 16 TB drive won't be anywhere close to 4x faster than the 4 TB drive, though. Maybe 20% (~200 MB/s vs ~230-250 MB/s). That's because most of the capacity increase comes from more tracks and more platters, not linear density, which determines throughput.
I would say 250-260MB/s for a 16TB drive, and when 4TB drives were newer they were more like 160-180MB/s. That's about a 50% increase.
Doing the math, you're probably going from 5 to 8-9 platters, so 2.2-2.5 areal density, and a naive square root suggests it should be about 50% faster. So maybe track width is taking a fair share after all?
Though when I look at more sizes, it seems like 12TB drives were also hitting 260MB/s and speed has almost entirely stalled since, with 24TB being only a hair faster.
That's a very logical theory, but it doesn't really seem to hold up in real-world data transfer speeds.
For instance: 7200 RPM SATA drives have been stuck in the peak-performance realm of between about 250 and 300 megabytes per second for a decade, even though maximum capacity has more than tripled over that time.
I have 2 3TB WD green desktop drives in my nas that have been running for 12 and 13 years power on time. Both have the disable head parking fix applied. Over the same period I've had 4 seagate nas drives of varying capacity fail. I really have to retire those 2 drives.
I bought two WD Red 6TB drives for a NAS, and both failed within two years. That’s unusually bad in my experience. Across two other NAS systems I’ve used for 15–20 years, I’ve only replaced drives three times.
One failed drive was replaced with the same model, which doesn’t inspire much confidence. The other couldn’t be replaced, so I received a refund of the original €198 purchase price from 1.5 years ago. The exact same 6TB drive now costs around €340, a pretty steep increase. For the second drive I went with Seagate, fingers crossed.
6TB Red, you say? That was about the time of their SMR scandal where they labeled junk drives for NAS use.
There’re still a couple of years left on my personal boycott timer before I’ll risk trusting them again. Every manufacturer has a bad batch sometimes, but that was egregious lying to customers.
I also had a 6TB red that failed. It was SMR. The other 6TB reds I had were and are still fine, but they weren’t SMR. I didn’t even know what SMR was but that drive had awful performance til the day it died
My worst one was a 8 TB Seagate IronWolf from Amazon. Bought at the same time as an 8 TB WD Red, as I wanted to spread the risk. The WD is still running about six years later, but the IronWolf died in less than 1.5 years. Thankfully Amazon gave me a full refund, so I threw in a shucked 12 TB WD Elements that's now been running for about five years.
I'm surprised Backblaze doesn't use host-managed SMR. I would expect that they'd already write data sequentially for speed and have different storage tiers (for "hot" data likely to be changed/deleted soon vs. data expected to stay for a long time). At least for large files in the cold tier SMR would make a lot of sense.
Until very recent price changes, at the scale that they purchase hard drives, I don't think that there would have been significant cost savings in purchasing SMR drives. Particularly after taking into account the additional hours of software development required to write custom host managed SMR stuff to deal with the performance problems.
Now that they appear to be using mostly enterprise SATA drives, I imagine there would be considerable cost savings, assuming they pull off a migration without serious problems. But that's a big assumption, and the risk certainly justifies a cautious approach so long as they remain quite profitable without SMR.
This seems like a poorly executed analysis of very interesting underlying data.
With this degree of average age variability they can't compare failure rates.
They should show us Kaplan-Meier (survival) curves not failure in this quarter.
Unless they set some datacenter on fire, not much in Q2 should be different vs Q1 for the same hard drive other than it being older.
Sure, internally, I'd do a better analysis controlled for age that shows if there are common failure rate increases but here they focus on model failure rates which is completely confounded by age differences.
I published the same criticism in relation to one of their earlier summaries. Backblaze responded with something along the lines of (paraphrasing liberally)
> We're giving you this data for free and providing a brief summary of what you should expect to see in it. We do not get paid to teach our readers advanced statistics, nor is that what we want to do. We encourage you to take the raw data and explain to people how you think it should be analysed.
which seems like a fair response. I'm very, very grateful that they release the data freely the way they do. It reflects well on them and more companies should do that.
A long time ago I would collect the magnets before discarding. Then it occurred to me that I've never had any use for them, so they were just more junk in a box.
They're cool, but I don't bother anymore - usually when I need neodymium magnets I need a specific size or shape anyway.
Since Backblaze publishes the stats, Seagate somehow doesn’t dare to put awfully bad drives on the market - ehm they did have a baaad track record. At least the ones monitored by BB seem to be performing well.
thx for posting this. Personally I am a fan of Western Digital Black Label spinning drives. Say what you will! I have details, and basically they work for me. AFAIK in 2026 and back into 2025, there are very large, new high-spec Western Digital Black Label disks in the catalog.. that I just cannot order. It says "all orders are pre-allocated until further notice" or similar ? Meanwhile, the same disks I bought ten years ago, same size, spec, performance and rating, as also available and I can buy those.. for perhaps twenty percent more than I paid long ago. This is new to me, this situation where an ordinary civilian cannot purchase the large drives, exclusive to "select customers" with opaque wording and no end in sight. The happy escalator of increasing spec and similar consumer prices, has ended.
Why am I reading the very newest Backblaze drive report -- they fired the over-forty engineer who started this, apparently.. and I as a consumer, cannot buy the same drives that might be listed there -- have not looked yet. The entire situation is fundementally changed. Info welcome
No inside information, but I imagine some models are made at the request of OEMs and hyperscalers who buy entire production runs, manufacturers and distributors would prefer not to drop ship in low quantities, and retailers don't like stocking lots of substantially similar products, especially now that internal hard drives are somewhat niche at retail.
I've had a lot of nvme form factor drives (whether in a usb stick or das or on my motherboard) fail in the last 5 years.
I assume heat. I have one of the fastest 2tb usb sticks you can get and it gets super hot in the aluminum case its in and that drive only lasted me a year. My tiny little 2tb only-used-for-time-machine drive i bought with my m1 max died years ago. I have a $1200 (now) msrp 4tb wd black i used for a month before it completely died. I don't think i have a single working usb stick right now out of probably 5 of them.
I have 4-6 samsung 970-990 pros none of those have died.
Well i guess with the price of drives right now I learned one thing the drives i got in the last 5 years will probably hold out for a while longer. Maybe there is some hope of waiting out this supply shortage.
It makes me a little physically sick that writing, something I love, is now tainted by the constant nagging thought, "could somebody mistake that sentence as AI?"
Critics of this line of worrying tend to fall into the same category of irrationality/bias as the "just be yourself" advice-givers.
It is an interesting thought about how these AI-cliches will affect human-written text that will subsequently be used in training - thus perpetuating the cycle.
i guess, but to me its not so much "this is ai this is terrible" and more "i'm sick of hearing this phrase all the time". it's not that I never want to see an em-dash ever again, its just that i don't want them used every other sentence.
I feel like there's no way an AI wasn't involved in this paragraph "The appeal is straightforward: more terabytes per drive slot, in the same rack space, at roughly similar drive-level power...."
In 2021, Backblaze saw ~14% failure rate at 7 years 9 months. Drive useful life: about 6 years.
That gave us what was a stable-seeming, old "replace drives at 5 years, or you're getting risky" rule of thumb (unless you prefer to replace on failure, of course).
In 2025, Backblaze saw ~5% failure rate at 10 years 3 months. Drive useful life: ~10 years, roughly!
Thanks for all the HDD improvements, industry!
Could be better controllers, or better driver firmware, or something.
Just curious where in the chain the improvements come from.
But I think part of it is the modern densities have less tolerance for manufacturing issues, so if the disks work they work well. Kinda like the Mars rovers which only had to work 90 days but they really had to work, so they ended up working for years.
[1]: https://spectrum.ieee.org/a-flood-on-the-diskdrive-market
Backblaze even showed with their statistics that the "bathtub curve" is dead - new drives have a barely lower failure rate than middle-aged drives https://www.backblaze.com/blog/drive-failure-over-time-the-b...
This lead to a lot of people doing things like intensive burn-in cycles for a week to test new drives before putting them in their machines.
When there are no longer early deaths (the "left" side of the bathtub), it's just "old things wear out" which doesn't have a special word for it.
Drives still have a meaningful average working life is what I see in te data. There's just barely any infant mortality left.
Yup. And most people wouldn't bother but typically you could RMA HDDs: the terms were quite generous. I was in Europe and did learn, as a teenager, how to RMA HDDs to get shiny replacements (IIRC for example for Seagate HDDs I'd send them to the Netherlands and then after a few weeks I'd get a new one by the mail).
Pretty crap.
Likewise with SSDs. Given their slightly more write intensive than ordinary desktop workloads, I'm reasonably confident that the capacitors in my 10-year-old enterprise SSDs (HGST SAS with Intel flash) will fail before the flash.
That's not exactly what this means. This is like life expectancy at birth vs life expectancy at current age. (which is the thing where people act like "only kings got old" in the past, when while there is some truth to it also doesn't mean "you would be dead by 30" when you already reached adulthood. The biggest difference we got regarding life expectancy at birth by far was doctors and nurses cleaning their hands, followed by things like vaccines for childhood diseases)
Manufacturing errors are likely to hit early. So replacing a drive means you might replace a perfectly healthy drive with one that will eat your data.
Just use SMART and you have a way better insight of how well your drive does than any such advice.
And also you likely won't run your drives in the same conditions as Backblaze, so things might vary in both directions.
There is shipping, storage, running conditions, series, manufacturing, and many variables. So you have a way better understanding of your drive than what Backblaze can provide if you just look at your drive's SMART data and run a self test.
There is both the mathematical truth to it, but also the physical/chemical one. If something eg. deformed or had a slight manufacturing mistake the wear will be quicker. If everything works perfectly and eg. there are little vibrations (can be everything from earthquake to noise) then HDDs also tend to live longer and run faster. Of course the shipping process also can take its toll and vary greatly.
I think in the world of computers it's all to easy to forget that things largely aren't binary and "logically perfect". Even with advancements in manufacturing and materials drives are physical things in the real world influenced by many things inside and outside of the manufacturer's and user's control.
And not because it failed, but because I was doing a system overhaul and replaced all my storage with M.2 NVMe drives.
Seriously, I had to replace 2 hard drives in my NAS this year like a bad luck lottery and my hard drives have doubled in price. I was terrified that my NVMe was failing as well.
There's one exception to this - HAMR added additional components, and a corresponding increase in BOM cost, but there's probably a learning curve on this.
However the recent AI data center boom is probably hitting HDDs in two ways - (a) general demand for anything datacenter-related, and (b) high DRAM prices, with most HDDs containing 256-512MB of DDR3. (I haven't physically examined the newest drives - they may be DDR4 and go beyond 512MB; also the 40TB supposedly includes NAND flash, adding yet more cost)
I think for the folks in the US this was a site that works: https://www.goharddrive.com/default.php
Whereas for those in the EU there's this: https://datablocks.dev/
For the second link, you'll have to ignore the horribly tasteless AI slop image for the site background (someone should e-mail them about it or something), the site itself is legit, bought some drives from them: https://blog.kronis.dev/blog/buying-some-drives-from-datablo...
It failed for a reason before, so it will likely fail again before the end of its usual lifecycle.
idk, it was such a traumatic experience for me that I decided to pass on recertified HDDs.
If a single HDD failing is traumatic then you're doing something wrong. At absolute minimum you should be in raid1. Backups should amount to snapshots that can be restored in a single step.
I wasn't storing anything important on it anyway, mostly movies ripped from Blu-rays. So its failure wasn't traumatic. However I expected it to last at least three years, but it died after exactly 365 days of use. The seller refused to send a replacement, and legally speaking, they were in the right.
Better telling everyone not to buy this things. /s
I usually don't spite people or companies but in this case I hope all these AI slop bullshit machines fail & fail hard.
So that we can get hardware at sane prices for actually useful thing. And to serve as a warning to never ever do such a stupidly disruptive shit ever again.
Yeah I'd prefer it to be even cheaper but the prices for HDDs aren't too bad.
It is 2x the price of just 2 years ago, it's too bad.
When I bought these they were peak GB/$, which is a fair couple of years ago.
But for me hdd prices feel like when people were panicking about crypto going down to prices that hadn't been seen in... days.
Now RAM and GPU prices, yeah. Those are bloody expensive.
I wonder what the use case for them is, they aren't ideal for RAID as they would take far too long to rebuild the array in a drive failure, but they also aren't great for long term cold storage. Perhaps they are ideal for periodic backups where you want to take a daily 1TB backup of a server or CCTV where you don't really care about backing up the data as it's all deleted on schedule anyway.
I keep my important stuff with ZFS raidz2.
It doesn't matter much to me if the overall process of replacing a dead drive can be done in 40 minutes or if it drags out to 40 hours. The storage system is still alive, still available for reading and writing, and still redundant for that entire time. It can take as long as it takes.
My break-glass-in-case-of-emergency offsite backups are done with restic over the network, so I only had to send 1 initial copy. That first copy took quite a long time. But restic is pretty clever about how it handles things; all subsequent daily backups are what might best be described as incremental-ish. They contain only recent changes and are pretty small with the usual churn rate on this particular pile of data.
(This matters because it keeps the cost of keeping it running quite low, which means that backups actually happen instead of being just a good idea. :) )
---
I'd probably pick different methods if I had a real budget for this stuff, but even then: Why would I be motivated to care about how long it takes to replace a hard drive?
So move the bar up to raidz3 or the equivalent? This adds cost, of course, but enhancing redundancy isn't free.
It was something I kept in-mind when I made the decision to use raidz2 for my own purposes: It tolerates two concurrent drive failures before local data loss happens, which I feel is acceptable for my purposes. raidz3 moves that tolerance to three concurrent drive failures.
I mean: What's the alternative for storing X amount of data? A greater number of relatively small drives? That's expensive in its own ways, too.
And adding complexity in that way tends to nosedive metrics like MTBF: If a system relies upon one widget and this system has an MTBF of n, then a system that relies upon two such widgets has an MTBF of n/2.
If the drives all have the same failure rate, then: A system with raidz2 and 10x 10TB drives will tend experience more disk failures than a system with raidz2 and 6x 20TB drives. Both systems can store ~80TB of data, but the one with 10TB drives has a worse ratio of redundancy and includes more parts that will eventually fail.
KISS, etc. :)
> and the longer it takes the more likely another drive will fail during that time.
You mean the old light-weight RAID-5-esque fear? A drive fails and redundancy drops to zero. So the failed drive is replaced or the hot spare is rotated in (or whatever), and the rebuild process begins. Another drive crashes during the rebuild because of the stress of actually-being-read and now the data on the array is presumed to be trash.
That process seems to actually work more like this: Old data that hasn't been read for years finally gets accessed (because RAID rebuild), and that's when some other drive starts reporting errors. But those errors were already there; they just weren't detected until it was way too late.
It doesn't really seem to work that way with filesystems like ZFS. The contents, whether old and stale or brand new, can (and should) be read routinely and checksummed to confirm integrity as part of a regularly-performed scrub.
The weak drives that can't stand to be read will be discovered during this read-only exercise, but they were already dead. We just didn't know it until we ran a scrub. So we scrub early, and often.
That's the same as with RAID, except: With RAID, the same disk still dies in the same way. We just don't know it was dead until a recovery is in-process. RAID, as commonly practiced, can therefore function as a problem-multiplier in ways that other systems can avoid.
As for speed it's mainly tied to rpm and the market decided no one wants higher failure rates / lower mtbf for marginal speed gains.
If you need speed and capacity, throw an ssd or two in front of a raid disk array for caching with primocache or whatever.
Especially now that the same 4TB NVMe SSDs I was buying for $300 are now selling for over $1,000.
In my case, the source of the data is S3, so there's no need for redundancy, as the hard drive is effectively a local cache for cloud storage and the work isn't nearly so time-critical that an occasional failure would be more than a mild annoyance to a customer, if that.
Same for archival backups. Faster recovery time than Glacier Deep Archive, less cost over a period of years, simpler and less expensive than tape for modest volume.
I guess if you have it all backed up in S3 it's fine, but if you put this thing in a NAS and one drive dies, your NAS is going to spend the whole week rebuilding where it's running at 100%, and another drive dying during this means data loss.
I’ve got a Synology that’s mostly full of 20TB drives. As you say, adding a drive takes about a week. Expanding the array is even longer.
If the cluster as whole can stomach the drive failures without losing data and is overall fast enough, there is no need to run many small drives where running fewer big drives is cheaper.
Every application will have different trade offs
In bigger applications, you have enough redundancy to allow one or two nodes to go down for a while. If a company had terabytes upon terabytes of critical data, having a multi-tiered storage would make sense, even of a full backup restore would take a days.on some tiers.
Most of the data on my NAS is of that form.
Doing the math, you're probably going from 5 to 8-9 platters, so 2.2-2.5 areal density, and a naive square root suggests it should be about 50% faster. So maybe track width is taking a fair share after all?
Though when I look at more sizes, it seems like 12TB drives were also hitting 260MB/s and speed has almost entirely stalled since, with 24TB being only a hair faster.
For instance: 7200 RPM SATA drives have been stuck in the peak-performance realm of between about 250 and 300 megabytes per second for a decade, even though maximum capacity has more than tripled over that time.
One failed drive was replaced with the same model, which doesn’t inspire much confidence. The other couldn’t be replaced, so I received a refund of the original €198 purchase price from 1.5 years ago. The exact same 6TB drive now costs around €340, a pretty steep increase. For the second drive I went with Seagate, fingers crossed.
There’re still a couple of years left on my personal boycott timer before I’ll risk trusting them again. Every manufacturer has a bad batch sometimes, but that was egregious lying to customers.
*AFR%-wise
https://dropbox.tech/infrastructure/smr-what-we-learned-in-o...
With this degree of average age variability they can't compare failure rates.
They should show us Kaplan-Meier (survival) curves not failure in this quarter.
Unless they set some datacenter on fire, not much in Q2 should be different vs Q1 for the same hard drive other than it being older.
Sure, internally, I'd do a better analysis controlled for age that shows if there are common failure rate increases but here they focus on model failure rates which is completely confounded by age differences.
> We're giving you this data for free and providing a brief summary of what you should expect to see in it. We do not get paid to teach our readers advanced statistics, nor is that what we want to do. We encourage you to take the raw data and explain to people how you think it should be analysed.
which seems like a fair response. I'm very, very grateful that they release the data freely the way they do. It reflects well on them and more companies should do that.
What else is there to do?
They're cool, but I don't bother anymore - usually when I need neodymium magnets I need a specific size or shape anyway.
Why am I reading the very newest Backblaze drive report -- they fired the over-forty engineer who started this, apparently.. and I as a consumer, cannot buy the same drives that might be listed there -- have not looked yet. The entire situation is fundementally changed. Info welcome
I assume heat. I have one of the fastest 2tb usb sticks you can get and it gets super hot in the aluminum case its in and that drive only lasted me a year. My tiny little 2tb only-used-for-time-machine drive i bought with my m1 max died years ago. I have a $1200 (now) msrp 4tb wd black i used for a month before it completely died. I don't think i have a single working usb stick right now out of probably 5 of them.
I have 4-6 samsung 970-990 pros none of those have died.
"That question is becoming more relevant as higher-capacity drives become more (dare we say) load bearing."
Critics of this line of worrying tend to fall into the same category of irrationality/bias as the "just be yourself" advice-givers.