ArthurHeymans/emacs-tramp-rpc: High-performance TRAMP backend using JSON-RPC instead of shell parsing

A high-performance TRAMP backend for Emacs that uses a binary RPC server instead of parsing shell command output.

Traditional TRAMP works by piping shell commands over SSH and parsing their output. This approach is robust but slow, especially for operations that require many round-trips (like directory listings or VC operations).

TRAMP-RPC replaces this with a lightweight Rust server that runs on the remote host. Emacs communicates with it using MessagePack-RPC over SSH, resulting in significantly faster file operations.

• Fast file operations via binary RPC protocol (2-57x faster than shell-based TRAMP)
• Async process support (make-process, start-file-process)
• Full VC mode integration (git, etc.)
• Automatic binary deployment (download or build from source)
• Support for Linux and macOS (x86_64 and aarch64)
• Batch/pipelined requests for reduced round-trip latency
• PTY support for terminal emulators (vterm, eat)

• Emacs 30.1 or later
• msgpack.el package (installed automatically from MELPA)
• SSH access to remote hosts
• Remote host running Linux or macOS (x86_64 or aarch64)

(use-package tramp-rpc
  :ensure t)

1. Clone this repository:

   git clone https://github.com/ArthurHeymans/emacs-tramp-rpc.git
       

2. Add to your Emacs init file:

   (add-to-list 'load-path "/path/to/emacs-tramp-rpc/lisp")
   (require 'tramp-rpc)
       

Access remote files using the rpc method:

/rpc:user@host:/path/to/file

On first connection, the server binary is automatically obtained and deployed:

1. Download from GitHub Releases (fastest, ~1MB download)
2. Build from source if Rust is installed and download fails

The binary is cached locally in ~/.emacs.d/tramp-rpc/ and deployed to ~/.cache/tramp-rpc/ on the remote host.

┌─────────────┐   SSH/MessagePack-RPC  ┌──────────────────┐
│   Emacs     │ ◄────────────────────► │ tramp-rpc-server │
│ (tramp-rpc) │                        │     (Rust)       │
└─────────────┘                        └──────────────────┘

The server exposes RPC methods for:

User connects via /rpc:host:/path
         │
         ▼
Remote already has binary? ──yes──► Done
         │ no
         ▼
Check local cache (~/.emacs.d/tramp-rpc/VERSION/ARCH/)
         │
         ├─ Found ──────────────────► Transfer to remote
         │
         ▼
Download from GitHub Releases
         │
         ├─ Success ────────────────► Cache locally, transfer to remote
         │
         ▼
Build with cargo (if Rust installed)
         │
         ├─ Success ────────────────► Cache locally, transfer to remote
         │
         ▼
Error with helpful instructions

If automatic deployment fails, download manually from GitHub Releases and extract to:

~/.emacs.d/tramp-rpc/VERSION/ARCH/tramp-rpc-server

For example:

~/.emacs.d/tramp-rpc/0.1.0/x86_64-linux/tramp-rpc-server

# Build for current platform
nix build

# Cross-compile for specific target
nix build .#tramp-rpc-server-x86_64-linux
nix build .#tramp-rpc-server-aarch64-linux

# Development shell with all tools
nix develop

cd server
cargo build --release

The binary will be at target/release/tramp-rpc-server.

# Install target
rustup target add aarch64-unknown-linux-gnu

# Build (requires appropriate linker)
cargo build --release --target aarch64-unknown-linux-gnu

;; Prefer building from source over downloading (default: nil)
(setq tramp-rpc-deploy-prefer-build t)

;; Local cache directory (default: ~/.emacs.d/tramp-rpc/)
(setq tramp-rpc-deploy-local-cache-directory "~/.cache/tramp-rpc-binaries")

;; Remote installation directory (default: ~/.cache/tramp-rpc)
(setq tramp-rpc-deploy-remote-directory "~/.local/bin/tramp-rpc")

;; Disable automatic deployment (default: t)
(setq tramp-rpc-deploy-auto-deploy nil)

;; Use different GitHub repo for downloads
(setq tramp-rpc-deploy-github-repo "myuser/my-fork")

;; Download timeout in seconds (default: 120)
(setq tramp-rpc-deploy-download-timeout 60)

Run M-x tramp-rpc-deploy-status to see:

• Current version
• Local architecture
• Whether Rust/cargo is available
• Cached binaries
• Download URLs

If you experience issues with diff-hl in dired buffers on remote hosts:

(setq diff-hl-disable-on-remote t)

The server binary is deployed using standard SSH (sshx method). Ensure you can connect to the remote host with:

ssh -o BatchMode=yes user@host echo success

If GitHub downloads fail (corporate firewall, etc.), you can:

1. Install Rust and let tramp-rpc build locally:

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
       

2. Download manually and place in the cache directory (see above)
3. Pre-deploy to remote hosts using your own method

TRAMP-RPC uses MessagePack-RPC over stdin/stdout with length-prefixed binary framing.

TRAMP-RPC originally used JSON-RPC with newline-delimited messages. This was changed to MessagePack-RPC for several reasons:

The switch eliminates encoding overhead for file transfers and fixes edge cases with non-UTF8 filenames that are valid on Unix filesystems.

MessagePack is a binary serialization format that provides:

• Native binary data support (no base64 encoding needed for file content)
• ~33% smaller messages compared to JSON
• Faster serialization/deserialization
• Proper distinction between null and false values

Each message is prefixed with a 4-byte big-endian length:

Request (conceptual structure):

((version . "2.0")
 (id . 1)
 (method . "file.stat")
 (params . ((path . "/etc/passwd"))))

Response (conceptual structure):

((version . "2.0")
 (id . 1)
 (result . ((type . "file")
            (size . 2847)
            (mode . 420))))

File content, paths, and process I/O are transmitted as raw binary (MessagePack bin type), eliminating encoding overhead and ensuring correct handling of:

• Non-UTF8 filenames
• Binary file content
• Arbitrary byte sequences in process output

TRAMP-RPC significantly outperforms traditional TRAMP for most operations:

Batch operations provide additional 3-6x speedup by combining multiple requests into a single round-trip.

For detailed benchmarks and an in-depth technical comparison with original TRAMP, see Technical Comparison.

TRAMP-RPC includes a comprehensive test suite using Emacs ERT (Emacs Lisp Regression Testing).

./test/run-tests.sh --protocol

Or directly with Emacs:

emacs -Q --batch -l test/tramp-rpc-mock-tests.el \
  --eval "(ert-run-tests-batch-and-exit \"^tramp-rpc-mock-test-protocol\")"

./test/run-tests.sh --mock

This runs protocol tests plus server integration tests that communicate directly with the RPC server (no SSH needed).

TRAMP_RPC_TEST_HOST=your-remote-host ./test/run-tests.sh --remote

Or:

emacs -Q --batch \
  -l test/tramp-rpc-tests.el \
  --eval "(setq tramp-rpc-test-host \"your-remote-host\")" \
  --eval "(ert-run-tests-batch-and-exit \"^tramp-rpc-test\")"

• test/tramp-rpc-tests.el – Full ERT test suite for remote operations
• test/tramp-rpc-mock-tests.el – CI-compatible tests (no SSH required)
• test/run-tests.sh – Test runner script

The GitHub Actions workflow runs:

1. Protocol Tests – MessagePack-RPC encoding/decoding (no server)
2. Server Integration Tests – Direct server communication (Rust binary)
3. Full Test Suite – (main branch only) Complete tests via SSH to localhost

This project is licensed under the GNU General Public License v3.0 or later – see the LICENSE file for details.

Contributions welcome! Please ensure code passes cargo clippy and cargo test before submitting.

For Emacs Lisp changes, also run the test suite:

./test/run-tests.sh --mock