Constraints

These details of the scenario have been given:

• data image processing server
• images are either 25GB or 50GB
• images are immutable once processed
• 20 TB usable space
• continuous stream of new images
• 400 GB/week average accumulation
• remain online for 6 months
• migrate offline after 6 months, keep indefinitely
• media migrates offline with the data

The following parameters were specified:

• total cost cannot exceed a $30,000 budget
• offline (archival) service is not our cost
• migrated backup media (archives) are our cost to replace
• implementation from "go" must finish in 2 months

Assumptions

For the purposes of this exercise, an intentionally vague problem description and parameters were given. As such, the following assumptions have been made:

• we are covering only simple hardware and media failures
• our backup host is separate from the storage host
• we treat the entire storage host as read-only untouchable
• only already-processed images need to be backed up
• we do not consider host system backups for our purposes
• the storage host itself is considered secure from outset
• we are unconcerned with tampering/break-in of involved hosts
• our segment has no offroute and complete electrical isolation
• we trust other nodes on local segment within airgap perimeter
• we trust human traffic that can bypass the airgap
• we do not have to encrypt backups, they are assumed safe
• storage available to backup host from its vfs (nfs or scsi)
• filenames to backup will match a given glob pattern in vfs
• matching backup files known-consistent (process + mv)
• backup media technology is not specified and can be chosen
• processed images are not compressible (encrypted)
• ~400GB * 24 weeks == ~9600GB needed for online backups (10TB)
• 400GB must be replaced every week

These assumptions can be corrected for if erroneous.

Hardware

Each hardware piece is detailed below. Rationale and implementation are provided in later sections. A total and recurring hardware cost conclude this section.

Backup Enclosure

The backup media will be directly attached commodity SATA disks in a JBOD configuration. We will deal with any drive device concatenation in the operating system layer.

The storage array chosen is a Promise 12-bay SAS/SATA enclosure selected for compatibility with the standard SAS HBA in our backup server and simple redundancy features like dual power supplies. Specifically, their VTJ310sS product. I/O modules (which come every four disks) can be directly cascaded into one large JBOD, with individual disks accessible from the operating system. The data sheet is here:

http://www.promise.com/upload/datasheet/VTrak_J-Class_DS_std.pdf

We could upgrade to a dual-port model if required for multi-path I/O with the purchase of an additional I/O module. As well, a 16-bay model is available.

At superwarehouse.com, the price of this enclosure with 12 blanks was:

$ 2278

Backup Server

The solution chosen as the backup server is a commodity PC hardware platform x86 machine such as a Dell PowerEdge R200 series server, running CentOS 5.x. We choose Linux for obvious reasons, and CentOS because it's cheap, stable, well known and supported. The CentOS 5 kernel contains drivers for the HBA we are using to interact with the enclosure (which is NOT the embedded HBA used for the system, but an add-in card).

Given our assumptions, we are mounting the storage server via NFSv4, which supports sufficiently large files and filesystems.

A simple pricing on dell.com gives us a single-core 2GHz x86 CPU with 1GB RAM, dual embedded gigabit NICs, dual 160GB primary disks (we will RAID1 our system disks as standard for any production server not part of a farm or booting off network), and embedded SAS HBA for talking to the system disks:

$ 838

To ensure driver compatibility with CentOS 5 and hardware compatibility with our array, we use a second HBA to access the disk enclosure. An Adaptec ASC-1045 SAS/SATA HBA will plug into the 4x PCI-Express port of our Dell R200 and support the native 3Gbit transfer speed, supporting up to 128 SATA targets on the bus for a cost of:

$ 194

We will use XFS for the filesystem, which is stable, has a mature Linux implementation, and will support 16TB filesystems on 32-bit platforms (recent kernels actually increase this figure even on 32-bit, but we do not need to consider that).

Backup Media

We will fill our array with 12 1TB SATA disks, which are compatible with our enclosure hardware. The model used is the Seagate Barracuda 7200RPM 3Gb/s ES.2 SATA drive.

These disks are priced at $160 in 1TB version. Since we average 400GB/week, our media cost will run $80/week. Upfront media cost:

$ 1920

to get 12 disks online (10TB usable), with a recurring cost:

$ 80/week

as we rotate disks into long-term storage and replace them.

Total Cost

Capital expenditure for this project:

Enclosure: $2278
Media:     $1920
Server:    $1092
----------------
Total:     $5290

Recurring expenses are anticipated at:

$    80 (weekly) OR
$ 3,800 (biannually)

Given that we had a budget of $30,000, we could run the solution for three years before exhausting the initial budget and requiring additional funds.

Note that this does not include the cost of offsite backup service, which was specified as coming out of the budget of a different group.

Implementation

Our backup requirements are dead simple. There is nothing that changes about our setup, files are large, incompressible, and have static sizes of 25GB and 50GB, which are convenient units for storage on 1TB disks.

Language

Since nothing complicated is needed here -- we are only backing up one host, can use a static NFS mount, we have random access to everything, random erase, the entire 6-month window is online at once from a single filesystem, etc -- a simple Bourne shell script is used to implement the backups: one run with arbitrary frequency to backup new files (selected based on age or lack of existence on backup media), and the other to implement archival.

There is no need for anything complicated and a shell script will be easy to implement and maintain. We don't even need to keep state: it's done in the filesystem (struct stat). If necessary, we could conceivably store extra data in extended attributes of the filesystem (which XFS does support) or in a simple text file.

Logistics

We use 11 of our 12 available 1TB disks for the online backup XFS volume. Only 10 are usable, since we use capacity gained by the 11th disk as the RAID5 parity stripe across our volume, to accommodate simple one-disk failure. We use Linux MD to implement this, and put LVM2 on top in case we want to do funny things later like snapshots or whatever, but our plan does not require this because the processing server will use the atomic rename() system call to ensure that only fully processed files end up matching the glob pattern used in the script to select files for backup.

Archival

The last remaining disk is used to rotate out the oldest files to long-term storage (archival). This will have a separate filesystem on it from the online storage, so the rotation process will use a simple copy every two weeks to rotate (or with whatever frequency we want to spread the copy load across time). We will know what files to select based on file mtimes and our run interval.

Copying 1TB of data at the 100MB/s sustained transfer rate of our disks would take roughly three hours, which is acceptable once every two weeks. Since our average is 400GB/week, running every 2 weeks gives us 200GB of variance, and probably enables us to just run every 2 weeks out of cron and not have to notify anyone. Someone will just use the sneakernet to pull the drive, swap it with a new one, and bring the drive out of the airgap to give to Iron Mountain with its 800-1000GB of data.

Caveats

The question begs, "why use disks?" 500 or 1000 GB drives are very cheap nowadays, and the rapid growth of disk storage obsoleted tape solutions many years back for new implementations. Cheap disk storage has random access, near-instant seek, supports arbitrary numbers of concurrent I/O streams and very high throughput. Tapes are appropriate where existing capital investment in tape hardware, media, or software is large, or at truly large scale (i.e. room-sized) HSM libraries. Those situations were not deemed to apply in this scenario.

Tape solutions like LTO4 do still have a cost advantage, but it is insignificant at this quantity, especially when the low cost of the JBOD enclosure is compared to tape changer. We do not really have a need for any of the benefits offered by hard disk based backup media in this scenario, but the cost was, if not a wash, superior at these levels when factoring in the autoloader. Because the existing "corporate approach" in the sample problem used an autoloader, that was not considered as a cost of a tape-based solution. But we are well within our budget with either solution, and the ability to re-use components of this backup solution for other purposes makes disk-based backups a superior choice, in my estimation.

It may be worth considering a tape solution, but the cost of disks seems very cheap as-is and provides many other benefits: simple implementation, no indices required, no (or less) state. Somewhere, we still have to keep a log so we know which disks have what, and someone has to put a physical label on the media, but this is trivial.

If we used tapes, it would still be a simple enough scenario to use 'mt' commands on the drive and 'mtx' commands with the changer in a script, but even a simple AMANDA setup would be trivial. Indexing does become more important with tapes to recover the data with any kind of speed and efficiency. It's only really "online" with an index.