Skip to content

Methods of Running Tests

EEST has two commands, consume and execute, that run test cases against EL clients:

  1. consume runs JSON test fixtures against a client - the client is said to "consume" the test case fixture.
  2. execute runs test cases from Python source against a client - the test case is "executed" against the client.

Top-Level Comparison

Both consume and execute provide sub-commands which correspond to different methods of testing EL clients using EEST test cases:

Command Description Components tested Environment Scope
consume direct Client consume tests via a statetest interface EVM None Module test
consume direct Client consume tests via a blocktest interface EVM, block processing None Module test,
Integration test
consume engine Client imports blocks via Engine API EngineNewPayload in Hive EVM, block processing, Engine API Staging, Hive System test
consume enginex Client imports blocks via Engine API in Hive, optimized by client reuse EVM, block processing, Engine API Staging, Hive System test
consume sync Client syncs from another client using Engine API in Hive EVM, block processing, Engine API, P2P sync Staging, Hive System test
consume rlp Client imports RLP-encoded blocks upon start-up in Hive EVM, block processing, RLP import (sync*) Staging, Hive System test
execute hive Tests executed against a client via JSON RPC eth_sendRawTransaction in Hive EVM, JSON RPC, mempool Staging, Hive System test
execute remote Tests executed against a client via JSON RPC eth_sendRawTransaction on a live network EVM, JSON RPC, mempool, EL-EL/EL-CL interaction (indirectly) Production System Test

*sync: Depending on code paths used in the client implementation, see the RLP vs Engine Simulator section below.

The following sections describe the different methods in more detail.

./hive --sim=eels/consume-engine vs consume engine

The execution-specs simulators can be ran either standalone using the ./hive command or via a uv/Python-based command against a ./hive --dev backend, more details are provided below.

Direct

Nomenclature
Command consume direct
Simulator None
Fixture Formats state_test,
blockchain_test

The direct method provides the fastest way to test EVM functionality by executing tests directly through a client's dedicated test interface (e.g. statetest or blocktest). This method requires clients to implement a custom interface to read tests and pass their inputs through appropriate code paths; implementation guides available for state tests and blockchain tests.

The EEST consume direct command is a small wrapper around client direct interfaces that allows fast and easy selection of test subsets to execute via test ID regex match (thanks to an index file). See Consume Direct and the Cache and Fixture Inputs and Useful Pytest Options pages for help with options.

Rapid EVM development

The direct method with the StateTest format should be used for the fastest EVM development feedback loop. Additionally, EVM traces can be readily generated and compared to other implementations.

Engine

Nomenclature
Command consume engine
Simulator eels/consume-engine
Fixture format blockchain_test_engine

The consume engine method tests execution clients via the Engine API by sending block payloads and verifying the response (post-merge forks only). This method provides the most realistic testing environment for production Ethereum client behavior, covering consensus integration, payload validation, and state synchronization.

The consume engine command:

  1. Initializes the execution client with genesis state.
  2. Connects via Engine API (port 8551), primitively mocking a consensus client.
  3. Sends a forkchoice update to establish the chain head.
  4. Submits payloads using engine_newPayload calls.
  5. Validates responses against expected results.
  6. Tests error conditions and exception handling.

EngineX

Nomenclature
Command consume enginex
Simulator eels/consume-enginex
Fixture format blockchain_test_engine_x

The EngineX method is a faster alternative to consume engine that executes multiple tests against a single client instance. This is achieved via the Blockchain Engine X Test fixture format which groups tests that share the same fork and EVM Environment together and contains a larger, shared pre-allocation state that all tests in the group use. This allows the EngineX simulator to execute multiple tests against the same client instance, whereas the Engine Simulator starts a fresh client for each test.

The consume enginex command, for each pre-allocation group:

  1. Initializes the execution client with the group's shared genesis state.
  2. Connects via Engine API (port 8551).

  3. Executes all tests in the group against the same client:

    • Submits payloads from each test using engine_newPayload calls.
    • Validates responses against expected results.
    • Tests error conditions and exception handling.

  4. Stops the client when all tests in the group complete.

Engine vs EngineX

consume engine consume enginex
Fixture format blockchain_test_engine blockchain_test_engine_x
Client lifecycle New client per test Client reused across tests with same pre-alloc
Fork choice update FCU called for genesis and final payload FCU for genesis and final payload skipped
Execution speed Slower (client startup overhead) Faster (amortized startup cost)
Test isolation Full isolation Shared genesis state within group

EngineX achieves faster execution by:

  1. Grouping tests by their pre-allocation state (genesis configuration).
  2. Reusing clients across all tests in a group, avoiding repeated client startup.
  3. Skipping redundant initialization since the client is already at the expected genesis state.

When to use EngineX vs Engine

Use consume enginex for faster test runs when full per-test isolation is not required. Use consume engine when you need complete isolation between tests or when debugging issues triggered by a single test case.

RLP

Nomenclature
Command consume rlp
Simulator eels/consume-rlp
Fixture format blockchain_test

The RLP consumption method tests execution clients by providing them with RLP-encoded blocks to load upon startup, similar to the block import process during historical synchronization. This method tests the client's core block processing logic without the overhead of network protocols.

The consume rlp command:

  1. Reads blockchain test fixtures from the specified input source.
  2. Extracts RLP-encoded blocks from the fixture files.
  3. Copies blocks to the client's container via files in the /blocks/ directory.
  4. Starts the client with the genesis state and block files.
  5. Validates the client's final blockHash via JSON RPC against the test's expectations.

This method simulates how clients import blocks during historical sync, testing the complete block validation and state transition pipeline, see below for more details and a comparison to consumption via the Engine API.

Sync

Nomenclature
Command consume sync
Simulator None
Fixture format blockchain_test_sync

The consume sync method tests execution client synchronization capabilities by having one client sync from another via the Engine API and P2P networking. This method validates that clients can correctly synchronize state and blocks from peers, testing both the Engine API, sync triggering, and P2P block propagation mechanisms.

The consume sync command:

  1. Initializes the client under test with genesis state and executes all test payloads.
  2. Spins up a sync client with the same genesis state.
  3. Establishes P2P connection between the two clients, utilizing admin_addPeer with enode url.
  4. Triggers synchronization by sending the target block to the sync client via engine_newPayload followed by engine_forkchoiceUpdated requests.
  5. Monitors sync progress and validates that the sync client reaches the same state.
  6. Verifies final state matches between both clients.

Engine vs RLP Simulator

The RLP Simulator (eels/consume-rlp) and the Engine Simulator (eels/consume-engine) should be seen as complimentary to one another. Although they execute the same underlying EVM test cases, the block validation logic is executed via different client code paths (using different fixture formats). Therefore, ideally, both simulators should be executed for full coverage.

Code Path Choices

Clients consume fixtures in the eels/consume-engine simulator via the Engine API's EngineNewPayloadv* endpoint; a natural way to validate, respectively invalidate, block payloads. In this case, there is no flexibility in the choice of code path - it directly harnesses mainnet client functionality. The eels/consume-rlp Simulator, however, allows clients more freedom, as the rlp-encoded blocks are imported upon client startup. Clients are recommended to try and hook the block import into the code path used for historical syncing.

Differences

eels/consume-rlp eels/consume-engine
Fixture Format Used BlockchainTest BlockchainTestEngine
Fork support All forks (including pre-merge) Post-merge forks only (Paris+)
Client code path Historical sync / block import pipeline Engine API / consensus integration
Real-world analogy Blocks received during sync Blocks received from consensus client
Interface Block import upon start-up via RLP files Engine API calls (newPayload, forkchoiceUpdated)
Exception testing Basic exception handling Advanced exception verification with client-specific mappers

Running both simulators adds some redundancy that can assist test debugging

If Engine tests fail but RLP tests pass, the issue is likely in your Engine API implementation rather than core EVM logic.

Execute

See Execute Command.

Two Methods to Run EELS Simulators

Many of the methods use the Hive Testing Environment to interact with clients and run tests against them. These methods are also called Hive simulators. While Hive is always necessary to run simulators, they can be called in two different ways. Both of these commands execute the same simulator code, but in different environments, we take the example of the eels/consume-engine simulator:

  1. ./hive --sim=eels/consume-engine is a standalone command that installs and configures execution-specs and its consume command in a dockerized container managed by Hive. This is the standard method to execute EEST fixture releases against clients in CI environments and is the method to generate the results at hive.ethpandaops.io. See Hive and its Common Options for help with this method.
  2. uv run consume engine requires the user to clone and configure execution-specs and start a Hive server in development mode. In this case, the simulator runs on the native system and communicate to the client via the Hive API. This is particularly useful during test development as fixtures on the local disk can be specified via --input=fixtures/. As the simulator runs natively, it is easy to drop into a debugger and inspect the simulator or client container state. See Hive Developer Mode for help with this method.