What I learned building, auditing, and operating a reward distribution system that bridges from an AI-evaluated source chain into two very different Layer 2s, Base (OP Stack) and zkSync Era. Custom LayerZero V2 messaging, merkle-root bridging to dodge payload limits, and the gotchas only zkSync teaches you.
Most "cross-chain" tutorials assume your destination chains are interchangeable EVM clones. They're not. Once you actually deploy the same protocol to Base (OP Stack optimistic rollup) and zkSync Era (zkRollup), you discover that "EVM-equivalent" is a marketing term, and that the same Solidity file can build, deploy, and revert in three different ways depending on which chain you point it at.
This post walks through how I built Rally's cross-chain reward distribution system: an architecture that originates reward decisions on an AI-evaluated chain (GenLayer) and bridges them into two heterogeneous L2s using LayerZero V2, with the design choices, audit findings, and L2-specific gotchas that only emerge once you actually ship.
The contracts went through a full security audit by Halborn (public report). Where relevant I'll reference what came out of it.
The product: a campaign platform where contributors are scored by AI-driven evaluation on a source chain, and rewards land on the user's L2 of choice (Base or zkSync Era) where they claim trustlessly.
Three constraints framed the architecture:
zksolc), its own gas model, and stricter contract-size limits. Same source, different destinations.LayerZero V2 was the right transport for this: chain-agnostic generic messaging, configurable DVNs (so you control the security stack rather than inheriting a multisig), and a clean V2 endpoint API. But how you use it matters more than the choice itself.
LayerZero ships two reference patterns most projects start from:
_lzReceive, you get a generic message bus.I deliberately used neither.
OFT was wrong because we weren't bridging a token's supply; we were bridging a distribution decision. The reward token already exists on each destination L2; what crosses the wire is a statement about who gets what.
OApp was tempting, but the inheritance chain pulls in opinionated peer/options handling that didn't fit the model where the same forwarder serves many destination contracts on the destination L2, not a single bonded peer. Instead, the bridge contracts implement ILayerZeroReceiver directly and talk to endpoint.send() / endpoint.quote() against the V2 endpoint.
The shape ends up being three pieces:
GenLayer (source) Bridge service Forwarder chain → Destination L2
───────────────── ────────────── ──────────────────────────────────
BridgeSender (IC) ──poll──▶ Node.js + HSM signer ──tx──▶ BridgeForwarder ──LZ──▶ BridgeReceiver ──forward──▶ Campaign
(stores msg hash) (validates, quotes) (endpoint.send) (endpoint.lzReceive) (settles distribution)
The intelligent contract on GenLayer doesn't talk to LayerZero. It writes a message hash into its state tree. An off-chain service polls, decodes, runs safety checks, quotes the LayerZero fee, and submits to a BridgeForwarder deployed on a "forwarder chain" (in our case, zkSync Era). The forwarder calls endpoint.send and LayerZero relays to a BridgeReceiver on Base or zkSync Era, which calls into the actual Campaign contract.
That extra hop (GenLayer → off-chain service → forwarder) exists because GenLayer's intelligent contracts can't directly send LayerZero messages (they execute in a different VM). The bridge service is the trusted transport that bolts the two worlds together.
LayerZero V2 can technically carry arbitrary-size payloads, but the economics punish you for it: every byte you send is metered on the source chain's gas, paid for a second time as a DVN fee, then a third time as executor gas on the destination. A naive distribution payload (array of handles, array of points, period index) for a non-trivial campaign turns a routine cross-chain message into something you'd rather not write into a busy bridge.
The fix is the kind of design choice that makes auditors happy and gas-conscious users happier: don't bridge the distribution, bridge the commitment.
Each period's distribution is built into a merkle tree off-chain. The cross-chain message carries only:
struct BridgedDistribution {
uint256 magic; // version + protocol marker
uint256 periodIndex;
bytes32 merkleRoot;
uint256 sumShares;
uint256 leafCount;
}
That's a tight, fixed-size payload. On the destination L2, the Campaign contract stores periodMerkleRoot[periodIndex] = merkleRoot and exposes claim(periodIndex, handle, amount, proof[]). Each user pays gas only for their own claim, and the bridge never has to fit a thousand recipients into a single LayerZero message.
This is the section I wish someone had written for me before we started.
Base (OP Stack) is close enough to mainnet EVM that you can deploy with standard Hardhat or Foundry, contracts behave the way you expect, and your gas estimates from a fork test are within a few percent of reality. EIP-170 still caps runtime bytecode at 24,576 bytes, same as mainnet, but in practice a well-factored monolith fits comfortably.
zkSync Era is a different chain that happens to speak Solidity. The differences that mattered:
zkSync compiles Solidity through zksolc, which produces different bytecode than solc for the same source. You need @matterlabs/hardhat-zksync to compile, deploy, and verify. Artifacts live in a separate path. Your typechain-types are still valid, but the artifact hashes won't match your Base build, meaning you cannot reuse Base bytecode on zkSync, even if the source is identical.
Deploy ceilings on zkSync don't come from EIP-170 (EraVM technically allows much larger bytecode); they come from bootloader and pubdata constraints that punish big contracts in ways the Solidity compiler can't predict. The practical effect is that a Campaign contract that deployed fine on Base ran into deploy-time failures on zkSync that only resolved when I factored PriceOracle out into an external library linked at deploy time. Same source, same logical size, different chain reality.
That linking is more fragile than it looks: hardhat-zksync resolves library bytecode references via the artifact's linkReferences. If your deploy script doesn't supply the address (or passes it before the library is deployed), the resulting bytecode contains literal zero bytes where the library address should live, and your contract is silently broken; calls into the library will revert with no useful error. Always verify with provider.getCode() after deploy and grep for the deployed library address in the bytecode hex.
Never hardcode gas on zkSync. Estimates are not in the same units; the chain accounts for prover cost on top of execution. A gas: 2_000_000n override that worked fine on Base will silently revert a proxy deployment on zkSync. Let the RPC estimate, and if it complains, fix the call site rather than padding the limit.
On zkSync, a failed transaction can still return a contractAddress in the receipt. If your deploy script trusts receipt.contractAddress without checking receipt.status === 1, you'll end up with a database row pointing at an address that has no code. We learned this the hard way; the deploy checklist now mandates checking status and re-reading bytecode before considering a deploy successful.
LayerZero V2's quote() returns a MessagingFee { nativeFee, lzTokenFee }. The fee is denominated in the forwarder chain's native gas token, and it varies with:
_options blob, which is where you tell LayerZero how much gas the executor should spend running your lzReceive on the destinationOptions are encoded with OAppOptionsType3, a tagged byte array where you specify executor gas and msg.value. We hardcode 200k gas per delivery, which is comfortable headroom for a Campaign merkle-root update (the receiver's logic is intentionally minimal: validate envelope, dispatch to local contract). For payloads that do more work on receipt, you'd want to budget higher and quote dynamically.
The forwarder exposes a view:
function quoteCallRemoteArbitrary(
uint32 dstEid,
bytes calldata data,
bytes calldata options
) external view returns (uint256 nativeFee, uint256 lzTokenFee);
The bridge service calls this immediately before submitting the relay tx, and uses the returned nativeFee as msg.value. This is the only sane approach. Fees fluctuate, and pre-quoting in batch jobs guarantees stale fee data.
One edge case the Halborn audit caught: when the source and destination are the same chain, quote() returns zero, the message is delivered "locally" without going through LayerZero, and naive code happily passes a non-zero msg.value that the contract then has no way to refund. The fix is a one-liner (require(msg.value == 0, ...) on the local-delivery branch), but you only think to write it after someone with a fresh perspective reads the code. That's what audits are for.
lzReceive revertsLayerZero V2 delivers a message by calling lzReceive on the destination. If your code reverts inside that call, two things can happen depending on your endpoint configuration: ordered mode bricks the channel until the bad message is fixed; unordered mode skips, but the message is lost.
Neither is what you want for a financial system. The pattern we landed on:
function lzReceive(
Origin calldata origin,
bytes32 guid,
bytes calldata message,
address /* executor */,
bytes calldata /* extraData */
) external override onlyEndpoint {
(uint32 srcChainId, address srcSender, address local, bytes memory inner)
= abi.decode(message, (uint32, address, address, bytes));
try IBridgeTarget(local).processBridgeMessage(srcChainId, srcSender, inner) {
emit ForwardCallSuccess(guid, local);
} catch (bytes memory reason) {
failedMessages[messageHashOf(guid)] = FailedMessage({...});
emit ForwardCallFailed(guid, local, reason);
}
}
The receiver catches downstream reverts, stores the failed message in state, and emits a structured event. From the LayerZero protocol's point of view, the message was delivered successfully; the channel keeps moving. From the operator's point of view, the failed message is parked and can be retried manually with retryFailedMessage(messageHash) once the destination contract's state is unblocked.
The audit (finding B-01) flagged this exact pattern as carrying operator risk: a permanently broken downstream contract can accumulate stuck messages, and recovery is owner-only. We accepted this trade-off explicitly. The alternative (auto-retry, queueing, etc.) is a lot more code to audit and a lot more failure modes. Manual retry is fine for a beta where ops humans are watching.
The off-chain service has a complementary safety net: before relaying a message it calls simulateContract() against the forwarder and skips messages that would revert mid-flight. Anything it can't simulate goes onto a deferredHashes set and is retried on the next sync cycle. Combined with idempotency on the Campaign side (per-periodIndex distributions can only be set once), this gives us at-least-once delivery without double-spend risk.
Cross-chain code is famously hard to test. Three patterns that worked:
1. A custom mock endpoint. Rather than depending on EndpointMock from @layerzerolabs/test-devtools, we wrote a minimal MockEndpointRelay that simulates the V2 send/receive flow against two in-memory endpoints. The mock exposes a simulateLzDelivery() helper that constructs the Origin struct and invokes lzReceive on the destination, so a single Hardhat test can exercise sender → endpoint → receiver → target end-to-end. The mock is intentionally small (it doesn't model DVNs, executors, or fees), but it covers the contract-level invariants.
2. Fork tests against real endpoints. The mock is fast but not honest. A separate suite forks Base Sepolia, points at the real LayerZero endpoint deployment, and validates things the mock can't: payload size enforcement, executor option parsing, actual fee quotes. These tests run slow and are gated behind a flag, but they catch the things "but it worked locally" never does.
3. A local bridge mode. The off-chain service has a LocalBridgeReceiver flag that points it at a single Anvil instance instead of two live RPCs. Useful for running the full GenLayer → service → forwarder → receiver flow on a developer laptop in seconds.
lzReceive, store them, retry manually. The LayerZero channel must not get stuck on a single bad message, and humans should sign off on each recovery.The Halborn engagement is documented in the public report. Two findings I keep coming back to:
msg.value lock on the local-delivery branch: the kind of small, embarrassing bug that auditors find precisely because they don't share your mental model of "this branch is rarely hit."The most valuable part of the engagement wasn't the findings themselves. It was the structured pressure to re-read every branch from the perspective of "what assumption am I making that isn't enforced in code?" That's a perspective that's hard to manufacture when you've been the only person writing the thing.
After a recent rsETH-related LayerZero incident underscored the risk of running on a single DVN, we hardened the configuration on the forwarder. We moved from the default DVN stack to a 3-DVN setup with explicit consensus thresholds, using three independent providers: LayerZero Labs, Canary, and Horizen. All three DVN addresses are fetched at deploy time from LayerZero's metadata API (https://metadata.layerzero-api.com/v1/metadata/dvns) and verified as actually deployed on each destination chain before any setConfig call.
The deployed configuration:
zkSync Era (eid 30165), addresses ordered ascending:
| # | Provider | Address |
|---|---|---|
| 1 | Canary | 0x05db3a229293c09f639a16526bb2481704716df0 |
| 2 | Horizen | 0x1253e268bc04bb43cb96d2f7ee858b8a1433cf6d |
| 3 | LayerZero Labs | 0x620A9DF73D2F1015EA75AeA1067227f9013f5C51 |
Base (eid 30184), addresses ordered ascending:
| # | Provider | Address |
|---|---|---|
| 1 | Canary | 0x554833698ae0fb22ecc90b01222903fd62ca4b47 |
| 2 | LayerZero Labs | 0x9e059a54699a285714207b43b055483e78faac25 |
| 3 | Horizen | 0xa7b5189bca84cd304d8553977c7c614329750d99 |
Final UlnConfig on both sides: confirmations=1, requiredDVNCount=3, optionalDVNCount=0.
One non-obvious detail that bit us during testing: the DVN array passed to setConfig must be sorted ascending by address on each chain, or the endpoint reverts with no useful error. The metadata API returns DVNs by provider name, not by address, so naive map-into-config code silently produces unordered arrays. The deploy script has an assertAscending() guard that aborts before sending anything if the array regresses out of order, so a misordered config can't even make it to a transaction. Notice the provider ordering ends up different across chains (Horizen sits at index 2 on zkSync but at index 3 on Base) because the underlying contract addresses sort differently. This is intentional and correct.
The change costs meaningfully more per message in DVN fees, but it eliminates the single point of trust that the original setup carried and lines up with how serious cross-chain protocols configure their security stacks.
The lesson generalizes: cross-chain messaging security is a configuration problem, not a protocol problem. The default options ship safely enough for testnets and small flows, but production deployments should pick their DVN set deliberately as soon as real value is at stake.
The architecture is good enough for the volume we're running. The remaining direction worth chasing:
Cross-chain is one of those areas where the second time you build it is dramatically cheaper than the first, because you've finally internalized that "EVM-equivalent" is a vibe, not a spec. I'll take that lesson into whatever the next bridge looks like.