Moving Large Datasets over High-Performance Long Distance Networks
SESSION: Storage Challenge Presentations
EVENT TYPE: Storage Challenge
TIME: 11:30AM - 12:00PM
SESSION CHAIR: Alan Sussman
ABSTRACT: Using programs like FTP, GridFTP, scp, and bbcp to move multi-terabyte datasets over 10+Gbit networks is very inefficient and impractical beyond a few gigabytes. A data storage performance analysis tool, XDD, has been modified to perform the data movement task between storage systems using a long distance (up to 8,600 miles) intermediate 10Gbit Network at rates that approach wire-speed. The issues and solutions to the problems involved in performing this feat are covered in this project. Basically, we are building a transfer tool that is resistant to distance-based latency and on the disk and network sides to maintain efficient high-bandwidth data transfer.