PGP Advisory Board
The Strange Rise and Fall of Hardware Disk Encryption
Tuesday, April 14th, 2009
It is perhaps a mistake to even talk about hardware as opposed to software anymore. Many devices that we normally think of as hardware are actually computer systems with a CPU, memory, mass storage, and so on. Routers, cell phones, music players, etc. are every bit as much software as hardware. On the other end of the spectrum, software systems are often wrappers around hardware. This is most true with graphics. There’s more compute power in a computer’s graphics card than there is in its CPU. Almost all of our devices are a mix of software and hardware.
This includes our disk drives, which are a mix of hardware and software as much as anything else is. Years ago, a disk controller would be thought of as a piece of hardware, but today it’s a computer system all of its own. In the case of NAS systems and RAID arrays, they are entirely small servers posing as disk drives. (Perhaps most amazingly, the battery in your laptop is also not merely hardware. There’s a computer in your battery that controls the way it charges and discharges.)
Recently, hardware disk encryption has appeared in disk products because the CPU on the disk controller has gotten fast enough that it has enough compute power to be able to encrypt disk data with minimal overhead. The result is encryption appearing in the hardware device itself.
Hardware disk encryption has some potential advantages over software encryption because it’s being done in the drive. The total system of the computer and this disk drive are a small multiprocessing system with a big processor in the computer and a small processor on the disk. At least in theory, two processors can be faster than one. At least in theory, the two systems can be more secure than one. But in theory, theory is the same as practice, but in actual practice this turns out not to be the case.
Let’s look at some of what’s going on.
Sometimes, when the hardware folks use word encrypt, the word doesn’t mean quite what I think it means. Some testers, particularly Heise Online, have been good at finding that some disk drives that advertise AES encryption use AES to encrypt your password, but
only an XOR to encrypt the disk
itself, or
just some linear algebra with your password stored in plaintext.
This is deplorable. It’s nothing but crypto snake oil. I have one of the drives that only XOR myself, but I didn’t use their encryption on it. I bought it because it was very, very cheap and not for its alleged security. If I’d bought it for the security, I’d be justifiably upset.
A number of other drives use real encryption, but in ECB mode. ECB, or “Electronic Code Book” encryption is the weakest form of encryption there is. It merely encrypts every sixteen-byte block with the key, and does not tie blocks together. This leaks a lot of information as all blocks that have the same plaintext have the same ciphertext. For files like XML files that have fixed headers, you can deduce which encrypted disk blocks are the starts of files, for example. With highly structured data, like some images, the encryption can literally be seen through.
You can find a Wikipedia article with excellent pictures to show the weakness of ECB mode here. Unfortunately, even
many reputable vendors aren’t forthcoming
about what mode they’re using. Presently, it is safe to assume that an encrypting drive is using ECB or worse, unless there’s documentation to say otherwise.
This gives customers the problem of knowing what is in the drive. Encryption is complex, and without the manufacturer saying precisely what is being used, it is impossible for consumers to know if they are getting good crypto, mediocre crypto, or bad crypto. I can’t figure it out, myself — if I could, I’d be happy to offer guidance of which drives are worth using and which aren’t. Unfortunately, even the new Opal specification from TCG mandates AES, but does not specify the mode of operation a drive must use. It is still possible for an Opal-compliant drive to use AES and still use AES in a way that yields a completely insecure drive.
Even the disks that are actually using encryption (even just ECB mode) increase the latency in the disk I/O by 15-25%. A rule of thumb on this is that it effectively lowers the speed of your disk by one notch on disk speed. A 7200 RPM disk with hardware encryption will run at about the speed of a 5400 RPM disk without encryption. A 5400 RPM encrypting drive will run about at the same speed as a 4200 RPM without encryption. This isn’t bad, performance-wise. Most users won’t notice it unless you put two drives side-by-side. Nonetheless, if you hear marketing claims that hardware disk encryption has no overhead, they’re false. You may not need to worry about the overhead, but it’s there, no matter what the marketing may say.
This means that any customer looking at hardware disk encryption cannot know what’s in the system because the manufacturers aren’t telling us. Even the recent standards like Opal give no guarantee that the crypto will be good. The marketing claims about performance are demonstrably false, so there is much reason for skepticism, even with systems coming from otherwise-reputable manufacturers.
The alternative is to use software encryption. We have our own PGP Whole Disk Encryption, but there are many software disk encryption systems. Software encryption is much faster than hardware encryption. This shouldn’t be a surprise. Which CPU is going to be faster, the one in your disk or in your laptop?
Software encryption has better security characteristics, as well; since there is plenty of extra compute power available on any modern PC. We developers can more secure (and thus slower) cryptography. We can also use better key management, better password management, and a better user experience. All of us software people are building our systems so that the overhead of what we do is not noticeable, and since we have more compute power available, we can put better encryption on the drive.
On the minus side, the disadvantage of software encryption is that the keys have to be in memory.
However, there are a number of hardware trends that are changing in the systems that have disk encryption, and these trends in hardware overall favor software encryption.
- Disks are getting faster. Primarily, this is coming from solid-state disks (SSDs). Since the speed of the disk dominates a disk encryption system, faster disks means better performance but worse benchmarks. If you double the speed of the disk, encryption will be a larger percentage of the whole system. This is good for users, but will cause much consternation in people who try to make sense of benchmarks, as an overall improvement can be shown as worsening performance if you do the benchmarks wrong.
- Multi-core systems are proliferating. This is good for software because disk encryption is very parallelizable. Normal disk encryption can benefit by up to eight threads for small I/Os, and more for larger I/Os. The more cores a CPU gets, the faster software encryption will be.
- Intel is putting AES in the CPU. Starting later in 2009, and with an improved architecture in 2010, this will be coming on most standard CPUs. It will increase the speed of AES by up to 10x over doing it in pure software. If you combine multi-core with this, then there can be huge improvements. It is not inconceivable that in the next half-dozen years, software disk encryption might speed up by 100X or more!Most interestingly, the large performance gains will happen in large reads or writes. On small reads and writes, either the hardware improvements or the AES instruction will run at maximum speed, but not both.
- Video cards are becoming general purpose processors. The cryptographer Debra Cook first showed the world how to speed up crypto in a graphics card, and both specialized and general programming systems that use the spare power of graphics cards are coming soon.NVIDIA has its system called CUDA, and they are working with other people on a platform-independent parallel programming system that The Khronos Group administers called OpenCL. Systems that are parallelized can be sped up by some 40-50x when done in the graphics card. This is great news for disk encryption because it is very suitable to parallel processing, and software has many ways to make it faster soon.
- Software is also inevitably cheaper. This is always true — the marginal cost of software is close to zero. Hardware can never do that. The cheapest storage is never going to have hardware in it, because it’s always cheaper to leave out the encryption hardware. Software is always buy-once, use everywhere.
- Lastly, the base computer platform is coming with more key management hardware. Software encryption will always have the drawback that it has to have the encryption keys in memory, but there are developments coming to make that more secure. Right now, there are plans coming from the hardware manufacturers like Intel and AMD, the BIOS people like Phoenix, as well as groups like the TCG. These are presently less well-formed than the performance improvements, but it’s coming too.
The net effect of these trends is that the advantages that hardware encryption has over software encryption are quite likely at their greatest today. Yes, the encrypting drives will get better, but so will the services that are available on the core computer platform itself. The exponential gains that we get from Moore’s Law means that the benefit will come faster and greater on the CPU and its subsystems than it will in the disk drives themselves. These exponential gains will affect the general-purpose systems quicker and more effectively than the special-purpose systems. That’s just where all the money and engineering is going.
These trends mean that ironically, the best days of hardware encryption are these present days. In less than a year, they will be in the rear-view mirror, with the advantage going geometrically to software. Doubly ironically, this trend has as its root cause cheap crypto hardware and cheap parallel processing. Hardware disk encryption, we hardly knew ye.
Hello Jon,
I think it’s interesting that you are declaring the “fall” of hardware disk encryption just as most of the major HDD vendors are announcing self-encrypting drives (both HDD and SSD).
I’d like to address some of the points you make in this blog posting, as there is a lot of misinformation. This is “Part 1″ of my reply, as I don’t have time to address everything today.
First, while it is true that most “hardware” devices are a mix of hardware and software (or “firmware” as it’s generally called in the embedded world), there is usually a clear separation within the devices regarding which functions get implemented in true hardware (usually for performance reasons) and which functions are implemented in firmware running on the device’s microcontroller. The microcontrollers used on modern drives are NOT fast enough to handle encryption running at the full data rate of the drive (at least not *real* encryption, e.g. AES… maybe it could handle the snake oil examples you give, but probably not at the full data rate). For AES encryption at the full data rate, a true hardware AES engine needs to be added into the drive’s electronics (either as an additional chip on the PCBA or integrated into the drive’s main controller chip… I’ve seen it done both ways).
If the encryption is implemented in true hardware in the data path of the drive, and if the encryption engine runs faster than the maximum data rate of the drive, there is ABSOLUTELY ZERO performance impact, contrary to your assertions that in-disk encryption increases the latency by 15-20%. OK, maybe I shouldn’t say ZERO, because there is *some* latency added by the introduction of an AES hardware engine – one cipher block’s worth. AES-256 requires 14 rounds of encryption per cipher block, so assuming one round per clock cycle, and being fast enough to keep up with the maximum data rate of the drive, the added latency is less than 200 nanoseconds.
Regarding all of the deplorable snake oil disk encryption products that you mention, I agree with you that these are deplorable. But I have never heard of any of these companies before. They’re probably out to make a quick buck while investing as little as possible in R&D, and sell their products to those who don’t know any better. On the other hand, the major drive vendors who have announced self-encrypting drives (Fujitsu, Toshiba, Hitachi, Seagate, and Samsung) are all large companies with reputations to defend, and with significant R&D budgets. We consult with security experts from other industries. We scour the available literature on encryption and security (of which there is no shortage). At Fujitsu we take security very seriously, and I’m quite confident that the other major drive manufacturers do, too.
Regarding your assertion, “it is safe to assume that an encrypting drive is using ECB or worse, unless there’s documentation to say otherwise,” I’d like to point out that the NIST AES Validation List offers a reliable source of cipher mode documentation for many of the drive vendors making self-encrypting drives. You’ll note that the ECB “mistake” has been corrected by at least one company in newer-generation product.
Is it possible that mistakes will be made in hardware disk encryption? Of course. I think the use of ECB by certain vendors was one of those mistakes. When such mistakes are identified, they get corrected in follow-on products, similar to how mistakes/vulnerabilities are corrected in software crypto products.
As far as the requirements for the Opal specification, you are correct that Opal does not mandate a certain cipher mode. However, it DOES mandate that the drive’s cipher mode is DISCOVERABLE by applications running on the host computer (the “Mode” column of the K_AES tables reports the cipher mode). This was somewhat of a compromise… It allows for legacy silicon that only implements ECB to still be “Opal-compliant”, and it also allows the software application managing the drive to decide if the drive meets a minimum security requirement. If the drive reports ECB, the software application is free to use software-based encryption instead of relying on the drive’s encryption.
I would also like to point out that an Opal-compliant drive is just one component of a disk encryption solution. Another critical component is the host-side software that must manage the security features of the drive, perform the password/key management, and provide a good user experience with a nice familiar-looking UI (hint, hint).
Finally (at least for “Part 1″ or my reply), I’d like to point out that there is no shortage of software-based snake oil crypto. But just because software snake oil crypto exists, I’m not going to declare the “fall of software disk encryption”. We all know that there are reputable companies, such as PGP, who take the security seriously and do a fine job of implementation. It’s no different in the self-encrypting drive market. There will be snake oil products, and there will be exceptionally secure products.
Best regards,
Darren Lasko
Principal Engineer
Advanced Development Group, Storage Products
Fujitsu Computer Products of America
Darren Lasko
Wednesday, April 29th, 2009 at 4:02 pm
Thanks for your reply, Darren.
I’m noting trends in the technology and speculating from them. I am indeed speculating, but my speculations are grounded in industry trends — the increase in cores and security hardware appearing in general-purpose hardware.
However, I’m going to stand by my own observed facts. The 15-25% latency is a number that we measured in our own lab by buying encrypting and non-encrypting versions of some drives and measuring it. The latency is there, we’ve seen it, and so have other reviewers. I don’t think that’s a bad number, myself. As I said in my first article, that’s around what you’d get if you bought a 5400 rpm drive as opposed to a 7200 rpm drive. It’s fast enough for nearly all purposes.
But in your comments you exhibit what I see to be the real problem — a lack of transparency in communication, which is not a technical issue at all. It’s marketing.
You said, “ABSOLUTELY ZERO” and put it in all caps. Then you admit that it’s not absolutely zero. And it isn’t. We’ve measured it, and it’s about 15-25%. We think it’s good enough, it’s just not ABSOLUTELY ZERO, which is what the claims of the hardware people are, and those claims are just not true.
Incidentally, I don’t want to go down a rathole of measuring crypto speed, but your calculations are off there. AES runs typically between 10 and 30 clock cycles per byte, depending on the CPU and how well the code was written. Let’s pick 10 because it’s the most favorable, and it’s easy to do math with 10. There are roughly 500 characters in a disk block (again, I pick this because it is favorable to the disk and easy to do math with), and so it’s 5000 clock cycles to encrypt a disk block. If we assume a 200 MHz processor on the controller and a 10ms normal latency, we get that the crypto adds 10% latency. The numbers are identical if you have a 400 MHz processor and 20 clocks per byte (which is very common, especially on an ARM).
Note that while I’m licking my finger and sticking it in the wind by picking some reasonable numbers that are err in being favorable to the computation, I get very close to numbers we’ve seen in the field.
I’ll repeat myself and say that these aren’t bad numbers! Users aren’t typically going to say, “gosh this disk is a turtle” with that sort of added latency.
The problem is that it’s nigh impossible for me, an educated expert in the field, to know what’s in my disk drive, and I pity anyone else who really wants to know. I encourage and urge you to come out with really geeky, detailed spec sheets. For the sake of the industry, stop saying ABSOLUTELY ZERO, well, not really ZERO, just really, really small.
We bought some drives from a reputable manufacturer that I will not name. It was not your company, but it was one of the companies that you listed. When we looked at the enciphered data on the disk, we noted that it had repeating patterns of eight-byte patterns. This lead us to believe that not only was the drive using ECB, which we’d been assured it was not, but that it was not using AES, which has a sixteen-byte block. Our bet is that it was using 3DES. We think we were shipped the wrong model number, but we don’t know, and we only found out by doing lab work.
Clarity is needed. I’m not dinging you nor your company, but the industry as a whole.
Now for my speculations about performance, as I said when I started, these come from industry trends. Intel says that in 2010, they’ll have AES on-CPU running at less than one clock per byte, assuming you have sufficient parallelization in the application. Disk encryption is ideal for this parallelization.
The very fast x86 implementations of AES run at 11-12cpb, and if you get to one and below with eight cores, that’s a pretty easy 100x improvement. Yeah, Intel has to deliver, but the smart money bets on that all the time.
Similarly, the roadmap from NVIDIA has them beating Intel for computes-per-watt and thus similar gains from highly parallelizable applications (like disk encryption).
It is not daft to think that a 4KB write will take 4-ish microseconds in software in just two to three years, thanks to the CPU and GPU guys. It’s been true for ages that general-purpose machines eclipse special-purpose machines because they have money and developers pushing them. It was a surprise to me when I connected all the dots, so I wrote about it.
And yeah, it’s speculation, as well as an interesting observation. If so-called software is one-hundred times faster than so-called hardware, then the advantage tips away from hardware to software. If Intel, NVIDIA, and others fail to deliver, then in a few years I’ll look back at my speculation and say, “meh; looked good at the time.”
Jon
Jon
Thursday, April 30th, 2009 at 5:03 pm
Hello Jon,
I think that maybe you missed one of the main points of the reply I
posted. We are not doing the encryption using the drive controller’s
microcontroller. It’s a dedicated hardware encryption engine built
into the data path of the controller chip. Therefore, software
benchmarks of 10-30 clocks per byte don’t apply. Using a dedicated
hardware encryption engine, AES-256 takes 14 clock cycles to
encrypt/decrypt one cipher block (16 bytes). For AES-128, it takes 10
clock cycles per 16 bytes. The added latency is just the time it
takes to encrypt/decrypt a single cipher block (14 clock cycles for
AES-256). If the AES engine is running fast enough to keep up with
the drive’s maximum data rate (which it is), then this latency is well
under 200 ns.
When I said, “ABSOLUTELY ZERO”, and then clarified with 200 ns, I knew
I should have put it into context a little better. My point was that
the added latency is so minuscule, it’s completely undetectable. Let
me provide some additional context now:
Disk drives have inherent latency due to seek time (the head moving
across the platter to get to the correct track) and rotational latency
(the time it takes for the media to rotate under the head until the
correct sector is reached). For current 7200 RPM (high-end) 2.5″
laptop drives, typical seek times are 10-13 ms, and average rotational
latency is 4.17 ms (the time for a half rotation at 7200 RPM).
To make the calculations as pessimistic as possible, let’s completely
throw out the latency due to seek time, and pretend that the drive has
only 4.17 ms of rotational latency. Adding in the 200 ns of latency
due to the AES engine brings the total latency to 4.1702 ms. This is a
difference of less than 0.005%. So while maybe I shouldn’t have said
“ABSOLUTELY ZERO”, the truth is that the difference is well within the
noise and is completely undetectable.
You asked for really geeky data sheets. You can access the specs for
our self-encrypting 7200 RPM drive here:
http://www.fujitsu.com/us/services/computing/storage/hdd/mobile/mhz2320cj-sata.html
Here are the specs for our non-encrypting 7200 RPM drive of the same
product generation:
http://www.fujitsu.com/us/services/computing/storage/hdd/mobile/mhz2320bj-sata.html
They both spec Average Latency at 4.17 ms. It wouldn’t make any sense
to spec “4.1702 ms” for the self-encrypting drive, as the extra 200 ns
is in the noise.
Can I please see your geeky spec sheets for PGP Whole Disk Encryption?
I checked the Overview / Technical Specifications / FAQ on the PGP WDE
page, but I can’t find any data for added latency, data throughput,
etc. I also noticed that the WDE Technical Specifications mention
that AES-256 is used, but there’s no indication of the cipher mode
used.
I want to understand more about the 15-25% added latency that you
measured in a certain undisclosed encrypting drive. How long ago did
you do this comparison? Was this comparison done with a stand-alone
self-encrypting drive, or was it using an encrypting enclosure? Are
you sure you were comparing apples to apples? If you simply take a
5400 RPM self-encrypting drive and compare it to a 5400 RPM
non-encrypting drive, you’re not necessarily comparing apples to
apples, because you need to make sure the drives are from the same
manufacturer and of the same product generation. As areal densities
increase from generation to generation, so does the sustained data
rate, even with no change in rotational speed of the platters. So if
you compared the self-encrypting drive to a newer generation of
non-encrypting drive, you definitely would have seen a performance
difference.
If you’re certain that you were comparing self-encrypting and
non-encrypting drives from the same manufacturer, of the same product
generation, with the same interface (SATA/PATA), and the same
rotational speed, then my guess is that the product in question was a
first-generation product where they didn’t get the design quite right.
But since it was probably the only game in town at the time, it was
probably good enough.
Regarding AES built into Intel CPUs, I think that this will be great
for accelerating many different types of applications that use AES,
including software-based disk encryption when self-encrypting drives
aren’t in the platform. However, no matter how fast AES can be
executed in the CPU, it cannot match the performance of a
self-encrypting drive. To understand why that is, you need to
consider not just the CPU, but the entire system and how data flows
within the system. Consider the normal flow of data on a disk read
when software-based disk encryption is not being performed:
1. CPU configures the SATA port for a read access, providing an
address to a buffer in system memory for the data to land
2. SATA port initiates the read command to the HDD (or SSD, or
whatever…)
3. Data streams directly from the chipset’s SATA port into system
memory via Direct Memory Access (DMA)
When using a self-encrypting drive, the above process doesn’t change,
because the data is decrypted on-the-fly by the drive and lands in
system memory in cleartext. The application that requested the data
can start using it immediately after step 3.
However, if you get the CPU involved because it needs to decrypt the
data, it looks more like this:
1. CPU configures the SATA port for a read access, providing an
address to a buffer in system memory for the data to land
2. SATA port initiates the read command to the HDD
3. Encrypted data streams directly into system memory via DMA
4. CPU reads a block of data from the buffer in system memory
5. CPU decrypts the data block
6. CPU writes the data block back to system memory
(repeat 4-6 until the entire buffer is decrypted)
Even if step 5 is very fast (or instantaneous) due to specialized
encryption instructions in the CPU, you’re still adding an additional
round-trip between main memory and the CPU for every piece of data
read/written from/to disk. It can never be as efficient as putting a
hardware encryption engine directly in the normal data path.
Jon, you say that you are not dinging me or my company, but the
industry as a whole. However, your experience seems to be based on:
1. Very poor encryption performed by drive enclosures, which aren’t
even products from the major drive vendors.
2. An early generation self-encrypting drive from a single drive
vendor.
Do you think you’re giving “the industry as a whole” a fair shake?
I encourage you to evaluate the self-encrypting drives that are
available today. In fact, I will gladly send you one of our currently
shipping self-encrypting drives along with its non-encrypting sister
product. You will not be able to measure a difference in performance,
and it’s not because we are dumbing-down our non-encrypting drive. I
also hope that websites such as h-online.com take a hard look at our
products for vulnerabilities.
Best regards,
Darren Lasko
Principal Engineer
Advanced Development Group, Storage Products
Fujitsu Computer Products of America
Darren Lasko
Monday, May 4th, 2009 at 3:15 pm
Technical support for these self-encrypting HDDs is very poor. You are told by the Support dept to enter a password in the bios. Under which security heading? Eventually you discover they mean the “HDD password” entry which is only present in some modern laptops. The only way you can confirm that the HDD is “encrypted” is by taking out the HDD from the laptop, putting it in an enclosure and then running it via a USB cable. How do you then know that it is not just “locked” via a normal HDD password int he normal way but it is actually ENCRYPTED? You need to see encryption to believe it, not just take it on trust.
Francis Beaumont
Wednesday, August 5th, 2009 at 4:08 am
I am fairly late in the game with this thread, but I had a question about the use of “free cycles” on the CPU – What if there are *no* free cycles left on the CPU? Sure, there are people out there with quad-core machines that tear up a great game of Minesweeper (and have 3 idle cores), but what about the people who operate at the limits of system performance?
I do on-site videography and post-edit in the field with a 3.06Ghz/8GB Macbook Pro. When doing recursive clip editing and tansitioning with compressed encoding, I literally do not have any free cycles available; the CPU and the drive run full-tilt for a good couple of hours after a shoot (Thank goodness for travel adapters). The laptop also gets amazingly hot, due to the CPU chewing on multiple clips around 25GB each and the flood of I/O read/write operations, but it gets the job done in a timely fashion.
I think that using ‘free cycles’ off of the CPU is a wonderful way of minimizing an encryption scheme’s performance impact to a nearly un-noticable level for regular users, but what kind of performance can I expect out of a software-encrypted filesystem if Final Cut Studio is hogging both cores and all 8GB of RAM? Benchmarks involving large files (over 500MB) that I have found sparsely scattered around the web are not promising, even though I know benchmarks are not always an indicator of real-world performance. I would think, however, that the high utilization of CPU, RAM and hard drive of any professional-level video editing suite is certainly not classified as ’standard use’, and is probably more closely represented by the numbers of a synthetic benchmark program designed to torture-test a system. Because of this, I am very concerned that I will take an enormous performance hit if I go with whole disk encryption.
Although now mostly in the field, I have had a fair amount of exposure to various server-level equipment and storage in the past. As a stop-gap measure, we tried sofware-based RAID when the ancient (hardware-based) Dell PERC2 card went bad in one of our older servers. It had dual 3.06 Xeons, 4GB RAM, 4 146GB drives in RAID-5 on CentOS, running only as a fileserver. It had plenty of free cycles, but performance under software RAID was dismal, even when we reconfigured the set in the less taxing striped and mirrored modes.
Perhaps this is apples and oranges, but it does show that sofware-based methods can be held captive to overall cpu availability (or take priority over your applications’ processes) and not all hardware-based solutions are slow by nature.
I’d like to be able to protect my footage(copyrights) via WDE in the case of a laptop theft, but I need to know what kind of performance I can expect under extreme CPU and I/O stress, not what someone else says should be ‘normal use’.
Ed
Tuesday, August 18th, 2009 at 8:03 pm