The technique of removing duplicate information to conserve disk storage space called data deduplication or dedup has gained traction in the IT industry. Companies that deploy the technology are able to achieve data compression ratios as high as 99 to 1. This in turn allows them maximize their existing storage utilization and hold off on purchasing expaned capacity, thus saving thousands in capital expenditures.
Dedup has spread outside of the data storage realm in recent years. Organizations are using it to cut down on their WAN bandwidth costs by compressing network traffic prior to transmission.
You should not confuse data deduplication with traditional file compression technology used in popular formats like WinZip, ARJ, and RAR files which rely on the LZMA algorithm. The two technologies vary most on the scale of data being compressed. Data Dedup seeks to remove big blocks of data such as a 3MB Word attachment that is emailed to a dozen employees on a distribution list. Instead of storing twelve instances of the document on the mailbox server’s storage it is able to store just one and insert a pointer for the other eleven. Simple file compression technologies work in a similar way but within individual files rather than entire disk drives.
If your Information Technology budget has been cut or frozen this year, Dedup may be the only way in which you can keep up with the exploding growth of data. Most companies experience data growth rates of 35%-50% per year as the majority of communications are now digital. By becoming an early adopter of Data Deduplication you can save yourself the headaches of having to expand your SAN every year just to accommodate new data.