ZeroShred: High-Performance P2P File Transfer

ZeroShred is a high-performance file transfer utility designed to maximize network throughput. It achieves transfer speeds exceeding 700 MB/s on modern hardware by utilizing kernel-level zero-copy techniques and hardware-accelerated parallel hashing.

Architecture and Data Flow

The following diagram illustrates the parallelized, zero-copy architecture that enables ZeroShred's high performance:

sequenceDiagram
    participant F as File System
    participant K as Kernel Space (Sendfile)
    participant N as Network Interface
    participant H as Hashing Thread (CommonCrypto)

    Note over F, N: Data Path (Zero-Copy)
    F->>K: system call initiate
    K->>N: direct DMA transfer
    
    Note over F, H: Integrity Path (Parallel)
    F->>H: concurrent read
    H->>H: hardware-accelerated SHA-256
    
    Note over N, H: Finalization
    N-->>H: wait for transfer complete
    H->>N: send file trailer (hash)

Loading

Performance Optimizations

ZeroShred implements several infrastructure-level optimizations:

• Zero-Copy Path: Utilizes sendfile(2) to transfer data directly from the kernel buffer cache to the network socket, eliminating userspace context switches and memory copies.
• Pipelined Hashing: Hashing operations run in a dedicated background thread, decoupled from the network I/O loop to prevent CPU-bound bottlenecks.
• Hardware Acceleration: Uses the macOS CommonCrypto framework to leverage instructions for SHA-256.
• Optimized Socket Buffers: Configures 4MB send and receive buffers to prevent TCP congestion and maintain peak throughput.
• Coarsened Chunks: Uses 1MB chunk sizes to minimize the frequency of system calls.

Benchmark Results

Tests conducted on local loopback (macOS M-series):

File Size	Peak Throughput	Integrity Status
64 MB	530 MB/s	PASS
128 MB	760 MB/s	PASS
512 MB	780 MB/s	PASS

Getting Started

Prerequisites

• macOS or Linux
• CMake 3.10+
• C11 compatible compiler
• Pthreads library

Build

mkdir -p build && cd build
cmake ..
make

Usage

Receiver:

./zeroshred recv

Sender:

./zeroshred send <receiver_ip> <file_path>

Demo (Quick Start)

A demo script is provided to showcase high-speed transfer and integrity verification:

./demo_transfer.sh

This will:

• Generate a 512MB random file (if not present)
• Start the receiver in the background
• Run the sender and show a live progress bar with speed and ETA
• Verify the transfer and print a summary

Demo & Benchmark Example

Below is a real run showing the demo and benchmark output, including speed, integrity, and comparison with HTTP transfer:

---

Clean up

To remove all test/demo files, logs, and build artifacts:

./scripts/clean.sh

Supported File Types

• Any regular file (binary, text, media, etc.)
• Large files (tested up to 10GB+)
• Filenames with UTF-8 and most special characters (except control chars)

Note: Directory transfer is not supported (single file per invocation).

How to Send a File

1. Build the project:

   mkdir -p build && cd build
   cmake ..
   make

2. Start the receiver (on the destination machine):

   ./build/zeroshred recv

3. Send a file (from the source machine):

   ./build/zeroshred send <receiver_ip> <file_path>

   • Replace <receiver_ip> with the IP address of the receiver.
   • Replace <file_path> with the path to the file you want to send.

4. Demo: To see a full demo with progress bar and speed:

   ./demo_transfer.sh

Implementation Details

• Memory Management: All large buffers are heap-allocated to avoid stack overflow issues (particularly on macOS thread stack limits).
• Resource Safety: Robust error handling with unified cleanup paths ensures descriptors and memory are released in all failure scenarios.
• Experimental Resume Support: Progress tracking via .part.meta files allows for resuming interrupted transfers (feature is experimental).
• Signal Handling: Configured with SO_NOSIGPIPE to prevent process termination on abrupt peer disconnection.

License

This project is licensed under the MIT License. See the LICENSE file for details.