Withdrawals Infrastructure

Overview of Syndicate Appchain withdrawals infrastructure and guidance for running your own

This process is under active development. To help improve this guide or for technical support running your own Appchain withdrawals infrastructure, join our Discord. To report a bug, you can create a new issue in https://github.com/SyndicateProtocol/syndicate-appchains/issues.

Withdrawals Overview

The Syndicate withdrawal system uses a TEE-based architecture to securely validate and submit withdrawal proofs across multiple chains. This system ensures secure, verifiable withdrawals while maintaining decentralization and trustlessness through cryptographic proofs and TEE-based verification.

For details on how the withdrawals system works, refer to the official project README.

Here we will discuss the specific infrastructure and configuration needed to support withdrawals for a Syndicate Appchain. This section is just an overview but further down is a guide for running these components yourself.

Synd-Enclave [docs]

This service performs the core logic for withdrawal proof validation. It is recommended to run this binary inside of an isolated Trusted Execution Environment ("TEE") with restricted network access. A simple co-located proxy server routes jsonrpc requests to the enclave via a secure vsock connection within the same VM.

This TEE infrastructure for the Synd-Enclave can be expensive — proof validation is CPU- and memory-intensive. As a rough starting point, plan for something like 16–32 vCPU, 64–128 GB RAM, and fast NVMe SSD storage (and scale up if you see slow proof validation or request backlogs). However, a single enclave can support multiple Syndicate Appchains as long as they share the same settlement chain.

Synd-Proposer [docs]

This service is responsible for extracting the Appchain root state and submitting assertions to the settlement chain. Synd-Proposer is a lightweight, stateless application that runs well as a docker container or kubernetes pod, but it is not horizontally scalable. Only one Synd-Proposer should be active per Appchain at a time. Unlike the Synd-Enclave, the Synd-Proposer cannot be shared across Syndicate Appchains; it is configured for one single Appchain and requires a dedicated wallet with settlement chain funds to operate.

Source chains

The withdrawals system needs access to L1, L2, and L3 source chains to obtain the raw data needed for withdrawal verification. Generally, you can provide paid or public RPC URLs for the L1/L2 chains and expose an RPC URL for your Syndicate Appchain as the L3. However this is NOT the case for the sequencing chain L2! The Syndicate Appchain stack relies on a specific fork of Arbitrum Nitro that supports the custom 'synd' API. This fork/API is typically not supported by mainstream providers like Alchemy. Therefore, running an additional sequencing chain readonly RPC node is also required to support withdrawals.

TEE Attestation Proofs [docs]

The withdrawals system relies on these SP1-based zero-knowledge proofs to cryptographically verify TEE attestation documents. There is no long-running infrastructure needed to generate these proofs; they are typically generated ad-hoc on temporary, expensive, high-CPU/GPU machines, which are then promptly shut down to save costs. These proofs only need to be generated once when spinning up a new Synd-Enclave, and only need to be regenerated if the enclave crashes and the old proof was not retained.

Although the infra requirements for these proofs are temporary, we have included the proof-generation process in the guide below since it is integral to the rest of the withdrawals configuration.

Withdrawals Infra DIY Guide

In this section we will guide you through configuring and running your own withdrawals infrastructure for your own Syndicate Appchain.

This guide assumes that your Syndicate Appchain already exists, you have a working RPC URL for it, and you have all of the Appchain configurations on-hand. If that is not the case, you may need to start with our 'GET STARTED' docs first to get your Appchain up and running, or contact Syndicate for support.

This guide assumes that the reader:

  • is familiar with containerization and Docker
  • is comfortable using ssh to execute commands on a remote machine
  • is proficient in running and maintaining infrastructure using a cloud provider
  • has an AWS account and project available for running Withdrawals infra
  • is familiar with AWS Trusted Execution Environments
  • has access to GPU machines for one-time proof generation
  • is up-to-date on Syndicate's latest withdrawals docs: https://github.com/SyndicateProtocol/syndicate-appchains/tree/main/synd-withdrawals
  • has a working RPC URL for this Appchain and all of the Appchain config at hand
  • has settlement chain funds or a way to obtain settlement chain funds
  • has access to an 'owner' wallet for this Appchain

Run Sequencing Chain (L2) RPC Node

In this section we will walk through running a readonly, non-archival, synd-enabled Syndicate Chain RPC node in Docker. This custom sequencing chain node must be healthy and synced in order to run the rest of the withdrawals infra below.

To run the node in Docker, use the following command:

docker run \
  -v /Path/to/mount/nitro/data:/home/user/.arbitrum \
  -v /Path/to/node-config.json:/home/user/.arbitrum/node-config.json \
  -p 0.0.0.0:8547:8547 \
  ghcr.io/syndicateprotocol/nitro/nitro:eigenda-v3.7.6 \
  --conf.file /home/user/.arbitrum/node-config.json

Note that you will need a few things set up beforehand:

  • /Path/to/mount/nitro/data must be mounted with an empty disk that has at least 500Gi of available storage
  • /Path/to/node-config.json must have the content from the node-config file below
  • Replace the beacon-url, connection.url, and init.url values in the node-config file below
  • This container needs at least 4 cores CPU and 16Gi memory to run

Syndicate Chain node-config.json:

{
  "chain": {
    "info-json": "[{\"chain-id\":510,\"parent-chain-id\":1,\"parent-chain-is-arbitrum\":false,\"chain-name\":\"synd-mainnet\",\"chain-config\":{\"homesteadBlock\":0,\"daoForkBlock\":null,\"daoForkSupport\":true,\"eip150Block\":0,\"eip150Hash\":\"0x0000000000000000000000000000000000000000000000000000000000000000\",\"eip155Block\":0,\"eip158Block\":0,\"byzantiumBlock\":0,\"constantinopleBlock\":0,\"petersburgBlock\":0,\"istanbulBlock\":0,\"muirGlacierBlock\":0,\"berlinBlock\":0,\"londonBlock\":0,\"clique\":{\"period\":0,\"epoch\":0},\"arbitrum\":{\"EnableArbOS\":true,\"AllowDebugPrecompiles\":false,\"DataAvailabilityCommittee\":false,\"InitialArbOSVersion\":32,\"GenesisBlockNum\":0,\"MaxCodeSize\":24576,\"MaxInitCodeSize\":49152,\"InitialChainOwner\":\"0x9a64b61bdfb2375d43a90e7bbf09efe18bd6cadc\",\"EigenDA\":false},\"chainId\":510},\"rollup\":{\"bridge\":\"0x3C8cF0ae6E89AC0796f29B3a58e7dEa1cD072277\",\"inbox\":\"0x5EA55Fd41D42Eb307D281bdE78E4e7572A35ea13\",\"sequencer-inbox\":\"0x12ad349e5d72B582856290736e0f13FE5fA57Aa4\",\"rollup\":\"0x451bD7813909B899DA6EbEC55E8fF823c057e14A\",\"validator-utils\":\"0xAa1EaB2ea108FDbCABd760a37E0B06f6e1dA8cC0\",\"validator-wallet-creator\":\"0x87f3f4620B4e8Dd170c7cCe84d1F8bEAD9f79b92\",\"deployed-at\":23026187}}]",
    "name": "synd-mainnet"
  },
  "parent-chain": {
    "blob-client": {
      "beacon-url": "https://some-eth-mainnet-beacon-url"
    },
    "connection": {
      "url": "https://some-eth-mainnet-rpc-url"
    }
  },
  "init": {
    "url": "https://some-syndicate-chain-snapshot-url.tar",
    "validate-checksum": false
  },
  "http": {
    "addr": "0.0.0.0",
    "port": 8547,
    "vhosts": "*",
    "corsdomain": "*",
    "api": [
      "arb",
      "eth",
      "net",
      "web3",
      "txpool",
      "debug",
      "synd"
    ],
    "server-timeouts": {
      "write-timeout": "10m"
    }
  },
  "node": {
    "eigen-da": {
      "enable": false
    },
    "staker": {
      "enable": false
    },
    "data-availability": {
      "enable": false
    },
    "dangerous": {
      "disable-blob-reader": false
    },
    "feed": {
      "input": {
        "url": "wss://synd-mainnet-relay.rollups.alchemy.com"
      }
    }
  },
  "execution": {
    "forwarding-target": null,
    "caching": {
      "archive": false
    }
  },
  "ensure-rollup-deployment": false
}

Refer to the Nitro documentation for node troubleshooting or reach out to Syndicate support.

This Syndicate node must be healthy, synced, and accessible to all of the services you will create in the following steps.

Run Synd-Enclave in AWS TEE

In this section we will walk through deploying Synd-Enclave and its proxy-server to a secure TEE (Nitro Enclave) in AWS.

Create Nitro Enclave VM

Create a new VM in AWS with the following spec:

  • Instance type: m5.2xlarge
  • AMI: ami-0c803b171269e2d72 or some other AMI with Nitro Enclave support in your desired region
  • Enable Nitro enclaves: true

Configure and Run the Nitro Enclave VM

Connect to the running VM and run the following steps:

Install dependencies

  yum update -y
  yum install -y git
  curl -L https://foundry.paradigm.xyz | bash
  source ~/.bash_profile
  foundryup

Create application directory and user

  mkdir -p /opt/syndicate
  useradd -r -s /bin/false -d /opt/syndicate syndicate

Pull Go Server and Enclave binaries from GitHub releases

  GITHUB_REPO="SyndicateProtocol/syndicate-appchains"
  RELEASE_VERSION="v1.0.7-rc.1"
  BASE_URL="https://github.com/$GITHUB_REPO/releases/download/$RELEASE_VERSION"
  sudo curl -fsSL -o /opt/syndicate/synd-withdrawals-server "$BASE_URL/server-$RELEASE_VERSION.bin"
  sudo curl -fsSL -o /opt/syndicate/eif.bin "$BASE_URL/eif-$RELEASE_VERSION.bin"

Set binary permissions

  chown syndicate:syndicate /opt/syndicate/synd-withdrawals-server
  chmod +x /opt/syndicate/synd-withdrawals-server

Install and configure AWS Nitro Enclaves CLI

  sudo dnf install aws-nitro-enclaves-cli -y
  sudo printf '---\nmemory_mib: 20480\ncpu_count: 6\n' > /etc/nitro_enclaves/allocator.yaml
  sudo systemctl enable --now nitro-enclaves-allocator.service

Run the withdrawals verifier with nitro-cli

  ENCLAVE_CID=16
  EIF_PATH=/opt/syndicate/eif.bin
  nitro-cli run-enclave --cpu-count 6 --memory 20480 --enclave-cid $ENCLAVE_CID --eif-path $EIF_PATH

Run the go-server

  BIND_ADDRESS=0.0.0.0:7333
  /opt/syndicate/synd-withdrawals-server

Obtain config values from the Enclave (save these outputs for the next section)

nitro-cli describe-enclaves
nitro-cli describe-eif  --eif-path /opt/syndicate/eif.bin
cast rpc --rpc-url localhost:7333 enclave_signerAttestation

Now you should have the go-server and verifier both running on this VM, but the verifier runs in a nitro enclave so you can only interact with it via the nitro-cli CLI. All requests to the go-server on port 7333 will get forwarded to the verifier over their vsock connection.

This Synd-Enclave VM must be running and accessible from wherever you plan on running Synd-Proposer.

Generate TEE Attestation Proof

In this section we will walk through generating a TEE attestation proof using SP1 tooling on an Nvidia virtual machine with a GPU.

Start by cloning and running this provisioning script on the GPU VM.

Once the provisioning is complete, you can run the the proof submission script

  • Replace all dummy/placeholder values in the script
  • PCR values and attestation document come from the Enclave step above
  • Wallet must be the Appchain owner wallet and have settlement chain funds
  • For the --deploy_new_contract_with_sp1_verifier flag, pass the target SP1 verifier contract (pick groth or plonk depending on the proof you’ll generate, default is groth16).

Depending on your GPU machine, this script could take up to an hour to complete. On success you should see that the AttestationDocVerifier and TeeKeyManager contracts have been deployed on the settlement chain. Save those addresses for the next section.

On success you can spin down this VM. Generating and submitting the proof is only needed once.

Run Synd-Proposer in Docker

In this section we will walk through running Synd-Proposer for your appchain.

To run the proposer in Docker, use the following command:

  docker run \
    --env ETHEREUM_RPC_URL=<ethereum_rpc_url> \
    --env SETTLEMENT_RPC_URL=<settlement_rpc_url> \
    --env SETTLEMENT_CHAIN_ID=<settlement_chain_id> \
    --env SEQUENCING_RPC_URL=<sequencing_rpc_url> \
    --env APPCHAIN_RPC_URL=<appchain_rpc_url> \
    --env ENCLAVE_RPC_URL=<enclave_rpc_url> \
    --env TEE_MODULE_CONTRACT_ADDRESS=<tee_module_contract_address> \
    --env APPCHAIN_BRIDGE_ADDRESS=<appchain_bridge_address> \
    --env PRIVATE_KEY=<private_key> \
    --env SEQUENCING_CONTRACT_ADDRESS=<sequencing_contract_address> \
    --env SEQUENCING_BRIDGE_ADDRESS=<sequencing_bridge_address> \
    --env MTLS_ENABLED_ENCLAVE=false \
    --publish 9292:9292 \
    ghcr.io/syndicateprotocol/syndicate-appchains/synd-proposer:v1.0.11

Note that all of those placeholder values need to be replaced. They come from a mix of sources including your appchain config, RPC providers, your Syndicate sequencing chain node, and your Synd-Enclave node and deployments from earlier. Also, you must create a dedicated wallet for this proposer, add 0.01Eth of settlement chain funds to it, and use that key for the PRIVATE_KEY var above.

Test

Now that you are operating the full Syndicate Appchain withdrawals system, test that everything is hooked up correctly.

Verify TEE Module

The simplest way to confirm the whole withdrawal flow is to check the TEE module directly on-chain. Find your TEEModule on the settlement chain (TEE_MODULE_CONTRACT_ADDRESS above) and check the recent transactions to verify that assertions are being posted and the challenge window is closing on an interval.

Test a Real Withdrawal

In order to perform a real withdrawal from your appchain, you need a wallet with appchain funds. Once you have that, you can connect that wallet to the Arbitrum Bridge tool and initiate a withdrawal from your appchain to the settlement chain.

You will need to add you appchain's bridge config to the Arbitrum Bridge UI as a custom chain (in testnet mode) in order to see it in the bridge dropdown. Reach out to Arbitrum support for help with this step.

From the Arbitrum Bridge UI you can initiate a withdrawal, then wait 2 hours (Syndicate fast-withdrawals challenge window time), then the UI will give you the option to "claim" the withdrawal in the Pending Transactions tab.

Troubleshooting

Here are some observability and troubleshooting tips to keep your Syndicate Appchain withdrawals system healthy:

  • Synd-Proposer exposes prometheus metrics at /metrics. Those values track the proposer wallet's balance and error responses from Synd-Enclave
  • If withdrawals are lagging, check your appchain's teeModule contract to confirm that assertion-posting is not halted. That could indicate an issue with the proposer or verifier
  • All of these services have debug-level logging that can be enabled if needed
  • Run nitro-cli describe-enclaves on the Synd-Enclave VM and check the go-server logs if you suspect an issue with the verifier

For details on how the withdrawals system works, refer to the official project README.

For technical support with your Appchain withdrawals infrastructure, join our Discord.

To report a bug, you can create a new issue in https://github.com/SyndicateProtocol/syndicate-appchains/issues.

Run an Appchain Node on GCP

Deploy and manage a Syndicate Appchain node on Google Cloud Platform

Sequencing Modules

Modular smart contracts for customizable transaction ordering, permissions, auctions, and cross-chain coordination

On this page

Withdrawals Overview
Synd-Enclave [docs]
Synd-Proposer [docs]
Source chains
TEE Attestation Proofs [docs]
Withdrawals Infra DIY Guide
Run Sequencing Chain (L2) RPC Node
Run Synd-Enclave in AWS TEE
Create Nitro Enclave VM
Configure and Run the Nitro Enclave VM
Generate TEE Attestation Proof
Run Synd-Proposer in Docker
Test
Verify TEE Module
Test a Real Withdrawal
Troubleshooting