Skip to content

Getting started#

ZIO-Maelstrom makes it easier to solve Gossip Glomers challenges in Scala using ZIO

Gossip Glomers is a series of distributed systems challenges by Fly.io in collaboration with Kyle Kingsbury, author of Jepsen. It's a great way to learn distributed systems by writing your own.

The challenges are built on top of a platform called Maelstrom. Maelstrom simulates network traffic using stdin and stdout of node processes.

“

ZIO-Maelstrom is a high level Scala driver for Maelstrom which abstracts away the low level details of the platform and let's you focus on solving distributed systems challenges

Alternatives

Maelstrom has an official Go library called maelstrom-go

There is also a java open-source library called maelstrom-java

Prerequisites#

1. Java#

Since Maelstrom is Closure program, you will need JVM installed on your machine. This templates makes use of graalvm native image to create OS native executable files for improved node performance. I have used sdkman to install java.

sdk install java 21.0.2-graal

2. SBT#

SBT is a build tool for Scala projects.

brew install sbt

3. Maelstrom#

It can be downloaded from here

Important

Ensure that maelstrom is added in your PATH variable so it can be run using maelstrom command from anywhere

You will also need graphviz & gnuplot. These are maelstrom dependencies for visualizing the simulation results.

brew install graphviz gnuplot

4. Coursier#

Coursier is the Scala application and artifact manager. You can download it from here

brew install coursier/formulas/coursier

Verify that it is available in your PATH variable and you can running it using coursier command

Get started using template#

The easiest way to get started is to create a new project using zio-maelstrom github template

View template

The template contains

  • sbt project with zio-maelstrom dependency
  • empty directory structure for all the solutions
  • scripts to compile solutions using graalvm native image
  • scripts to run solutions with expected workload

Create a new project using the template repository or clone the repository

Important

If you have not installed graalvm using sdkman, please update "nativeImageGraalHome" in build.sbt to point to your graalvm installation directory. You may have to install native-image component using gu install native-image command if it is not already installed

Echo challenge for example, has echo/src/main/scala/gossipGlomers/Main.scala as shown below

package gossipGlomers

import zio.ZIOAppDefault

object Main extends ZIOAppDefault {

  def run = ???

}

package gossipGlomers

import zio.json.{JsonEncoder, JsonDecoder}
import zio.{ZIOAppDefault, ZIO}
import com.bilalfazlani.zioMaelstrom.*

case class Echo(echo: String, msg_id: MessageId) extends NeedsReply derives JsonDecoder

case class EchoOk(echo: String, in_reply_to: MessageId, `type`: String = "echo_ok") 
    extends Sendable, Reply
    derives JsonEncoder

object Main extends ZIOAppDefault {
    val echoHandler: ZIO[MaelstromRuntime, Nothing, Unit] =
        receive[Echo](msg => reply(EchoOk(echo = msg.echo, in_reply_to = msg.msg_id)))

    val run = echoHandler.provide(MaelstromRuntime.live)
}

After writing the solution, we can run the solution using either using a JVM process or a native image

Running the simulation#

Create and run a fat JAR#

# This command will compile and create a fat jar and a runner script
sbt "echo/bootstrap"

# This command will run maelstraom with appropriate work load arguments
./echo/target/testjar.sh

jepsen.core {:perf {:latency-graph {:valid? true},
        :rate-graph {:valid? true},
        :valid? true},
:timeline {:valid? true},
:exceptions {:valid? true},
:stats {:valid? true,
        :count 45,
        :ok-count 45,
        :fail-count 0,
        :info-count 0,
        :by-f {:echo {:valid? true,
                      :count 45,
                      :ok-count 45,
                      :fail-count 0,
                      :info-count 0}}},
:availability {:valid? true, :ok-fraction 1.0},
:net {:all {:send-count 92,
            :recv-count 92,
            :msg-count 92,
            :msgs-per-op 2.0444446},
      :clients {:send-count 92, :recv-count 92, :msg-count 92},
      :servers {:send-count 0,
                :recv-count 0,
                :msg-count 0,
                :msgs-per-op 0.0},
      :valid? true},
:workload {:valid? true, :errors ()},
:valid? true}


Everything looks good! ヽ(‘ー`)ノ

Create and run a graalvm native image#

# This command will run a reduced workload to generate graalvm native reflection configurations
# Those configurations will be stored in resources/META-INF/native-image dir of the project
sbt "echo/maelstromRunAgent"

# This command will compile and create a native image and a runner script
# It will use the reflection configurations generated in previous step
sbt "echo/nativePackage"

# This command will run maelstraom with appropriate work load arguments
./echo/target/test.sh

jepsen.core {:perf {:latency-graph {:valid? true},
        :rate-graph {:valid? true},
        :valid? true},
:timeline {:valid? true},
:exceptions {:valid? true},
:stats {:valid? true,
        :count 44,
        :ok-count 44,
        :fail-count 0,
        :info-count 0,
        :by-f {:echo {:valid? true,
                      :count 44,
                      :ok-count 44,
                      :fail-count 0,
                      :info-count 0}}},
:availability {:valid? true, :ok-fraction 1.0},
:net {:all {:send-count 90,
            :recv-count 90,
            :msg-count 90,
            :msgs-per-op 2.0454545},
      :clients {:send-count 90, :recv-count 90, :msg-count 90},
      :servers {:send-count 0,
                :recv-count 0,
                :msg-count 0,
                :msgs-per-op 0.0},
      :valid? true},
:workload {:valid? true, :errors ()},
:valid? true}


Everything looks good! ヽ(‘ー`)ノ

Maelstrom reports#

To view run results, run

# This command starts a web server on port 8080 where you can view the results
maelstrom serve