Bulk data transfer
Bulk data transfer
Bulk data transfer definition
Bulk data transfer is the process of sending a lot of data from one computer system to another, typically over the internet or other networks. The amount of data is usually so large that traditional transfer methods may not work.
See also: data transfer, managed file transfer, content delivery network
Common methods of bulk data transfer
- File Transfer Protocol (FTP) and Secure File Transfer Protocol (SFTP). FTP is a standard protocol for sending files between a server and a client over a network, such as the internet. SFTP is its safer extension that encrypts data for secure transfer.
- rsync. A software application used for syncing files and directories between several locations. It keeps the amount of data to a minimum by only sending the differences between the source and the destination files.
- GridFTP. GridFTP is a protocol based on FTP and specifically designed for high-speed transfer of large volumes of data. It’s often used in scientific research communities.
- Aspera. IBM’s own data transfer protocol was designed to be highly reliable and secure. Aspera uses a patented technology called FASP (Fast and Secure Protocol) to ensure fast and safe data transfer. It’s especially useful for long distances and unstable networks.
- Data Transfer Nodes (DTNs). DTNs are high-performance systems specially designed for bulk data transfer over high-speed networks.
Bulk data transfer uses
- Data migration. Sometimes businesses or services need to move their data from one storage system to another or from an old system to a new one. This could happen during a system upgrade or when a company switches to a new data storage provider. In any case, it often involves transferring enormous amounts of data.
- Data backup and recovery. Companies often use bulk data transfer to back up their data to an offsite location or the cloud. This allows for quick data recovery in case of a system failure or data loss event.
- Distributed computing and big data. Companies working with distributed systems and big data often send large volumes of data for processing and analysis.
- Content delivery networks (CDNs). CDNs use widespread networks of servers for fast delivery of large amounts of data to end users. They often use bulk data transfer to send data to various points within the network.
- Scientific research. Scientific communities often need to transfer large datasets for collaborative research and analysis. That’s especially relevant for researchers working in astronomy, genomics, and climatology.