Working With GRPC, Kotlin and Gradle

Mon, Oct 9, 2017
7-minute read

Edit: The source code for this post is located on GitHub

Sometimes when I travel I end up trying to learn something completely new. For a while I was playing with Rust, Capn Proto, Scala, or I’d start a throwaway project at an airport and just tinker.

My passion is and has always been databases. I’ve maintained this blog for roughly a decade, starting with MySQL for the first part of my career but moving to Apache Cassandra several years ago, and am now a committer and member of the PMC.

Regarding languages, I’ve been all over the map. While I really enjoyed Rust, I found the ecosystem to be a bit immature, making it difficult to work on toy projects. There are some fantastic libraries, such as Rust-PEG, but when you’re just trying to kill some time at an airport it’s a lot more fun to already have a selection of RPC libraries and database drivers.

When it comes to mature libraries, there’s really no competing with Java. That said, Java is an awkward, bloated, overly verbose language. Fortunately Java and the JVM are two separate beasts, and there are many alternatives to Java on the JVM.

Enter Kotlin. Kotlin is a very pleasant language targeting the JVM. We get the benefits of a mature ecosystem combined with a nice language design that’s not trying to be overly academic. I’m looking at you, Scala.

In this post I’m going to walk you through creating a dead simple Kotlin based key value store that supports two operations, get and set. We’ll leverage the gRPC library to run the server, and Gradle to build everything. I’m going to be referencing this branch of a simple key-value store to show how to get up and running. The project is deliberately simple. I want to avoid getting into the details of testing, design patterns, etc, and focus on getting up and running.

Disclaimer: As I’m relatively new to all three topics, it should be obvious I’m not an expert! If I manage to spent a few hundred hours working on any of this I’ll do a follow up post.

First off, I’ve created a new project in IDEA, picking gradle for my build tool. I’ve given it an appropriate name: awfuldb.

Next, we’re going to configure our Gradle build configuration. This is the build.gradle file in the top level of your project.

We’ll need the Java plugin to compile the gRPC libraries. The application plugin will let us use ./gradlew run, whichi is convenient.

group 'com.rustyrazorblade'
version '1.0-SNAPSHOT'

apply plugin: 'java'
apply plugin: 'kotlin'
apply plugin: 'application'
apply plugin: 'com.google.protobuf'
apply plugin: 'idea'

Here we set the main class name for gradle to know what to run, we’ll come back to this later.

mainClassName = "com.rustyrazorblade.awfuldb.server.MainKt"

Here we configure out build. We’re pulling a bunch of stuff out of Maven central, so we’ll need it in our repositories list. The buildscript lets us add plugins to assist us in our build process. In this case, we’re using the protobuf Gradle plugin to generate our gRPC and protobuf libraries for us automatically whenever we compile.

repositories {
    mavenCentral()
}

buildscript {
    ext.kotlin_version = '1.1.4-3'

    repositories {
        mavenCentral()
    }
    dependencies {
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
        classpath 'com.google.protobuf:protobuf-gradle-plugin:0.8.3'
    }
}

Besides compiling our gRPC and protobuf spec, we need to include the protobuf library itself. We’ll use junit for a dead simple couple of tests.


def grpcVersion = '1.6.1' // CURRENT_GRPC_VERSION

dependencies {
    compile "org.jetbrains.kotlin:kotlin-stdlib-jre8:$kotlin_version"

    compile "com.google.api.grpc:proto-google-common-protos:0.1.9"

    compile "io.grpc:grpc-netty:${grpcVersion}"
    compile "io.grpc:grpc-protobuf:${grpcVersion}"
    compile "io.grpc:grpc-stub:${grpcVersion}"

    testCompile group: 'junit', name: 'junit', version:'4.12'
}

compileKotlin {
    kotlinOptions.jvmTarget = "1.8"
}
compileTestKotlin {
    kotlinOptions.jvmTarget = "1.8"
}

Next we’ll tell Gradle how to compile the protobuf.

protobuf {
    protoc {
        artifact = 'com.google.protobuf:protoc:3.4.0'
    }
    plugins {
        grpc {
            artifact = "io.grpc:protoc-gen-grpc-java:${grpcVersion}"
        }
    }
    generateProtoTasks {
        all()*.plugins {
            grpc {
                // To generate deprecated interfaces and static bindService method,
                // turn the enable_deprecated option to true below:
                option 'enable_deprecated=false'
            }
        }
    }
}

IDEA needs to know about the generated code, so we’ll tell it about some new source directories to look for our generated Java code. This will give us the delicious auto complete we love.

idea {
    module {
        // Not using generatedSourceDirs because of
        // https://discuss.gradle.org/t/support-for-intellij-2016/15294/8
        sourceDirs += file("${projectDir}/build/generated/source/proto/main/java");
        sourceDirs += file("${projectDir}/build/generated/source/proto/main/grpc");
    }
}

compileKotlin {
    kotlinOptions.jvmTarget = "1.8"
}
compileTestKotlin {
    kotlinOptions.jvmTarget = "1.8"
}

This next bit configures gradle to know where to look for the generated code.

// auto build protobufs
compileKotlin.dependsOn ':generateProto'

sourceSets.main.java.srcDirs += 'build/generated/source/proto/main/grpc'
sourceSets.main.java.srcDirs += 'build/generated/source/proto/main/java'

src/main/proto/AwfulServer.proto

We’ll tell the protobuf generator to use the same group and package

syntax = "proto2";

package com.rustyrazorblade.awfuldb;
option java_outer_classname = "AwfulServer";

service AwfulService {
    rpc PutValue(PutRequest) returns (PutResponse);
    rpc GetValue(GetRequest) returns (GetResponse);
}

message PutRequest {
    required bytes key = 1;
    required bytes value = 2;
}

message PutResponse {
    required bool success = 1;
    optional string error = 2;
}

message GetRequest {
    required bytes key = 1;
}

message GetResponse {
    required bytes value = 1;
}

In com/rustyrazorblade/awfuldb/server/DB.kt you’ll find a really dead simple key value DB. No validation, nothing really. Just using a ConcurrentHashMap.

package com.rustyrazorblade.awfuldb.server

import com.google.protobuf.ByteString
import java.util.concurrent.ConcurrentHashMap
import java.util.Optional

class DB {
    private var data : MutableMap<ByteString, ByteString> = ConcurrentHashMap()

    init {
        println("Creating a new DB")
    }

    fun put(key: ByteString, value: ByteString) {
        data.put(key, value)
    }

    fun get(key: ByteString) : Optional<ByteString> {
        var tmp = data[key]
        if (tmp != null) {
            return Optional.of(tmp)
        } else {
            return Optional.empty()
        }
    }

}

An interesting note regarding the above code, I had originally used a ByteArray instead of the ByteString, but it turns out ByteArray does object equality comparisons, making it unsuitable to be the key in a MutableMap. So for instance, this doesn’t work:

val key1 = "test".toByteArray()
val key2 = "test".toByteArray()

var map = mutableMapOf<ByteArray, String>()

map.set(key1, "my value")
val result = map.get(key2) // returns null, sad face

Even though key1 and key1 are both "test", they’re different objects, so you will never be able to fetch the values out of the Map when doing RPC calls. ByteString has the value comparison behavior we need, so it will work for our needs.

In com/rustyrazorblade/awfuldb/server/Server.kt, we’ll implement the server generated by gRPC.

package com.rustyrazorblade.awfuldb.server

import com.rustyrazorblade.awfuldb.AwfulServer
import com.rustyrazorblade.awfuldb.AwfulServer.PutResponse
import com.rustyrazorblade.awfuldb.AwfulServiceGrpc
import io.grpc.stub.StreamObserver

class Server : AwfulServiceGrpc.AwfulServiceImplBase() {
    private var db = DB()

    override fun putValue(request: AwfulServer.PutRequest?, responseObserver: StreamObserver<PutResponse>?) {

        var success = false

        if (request?.key != null && request.value != null) {
            db.put(request.key, request.value)
            success = true
        }

        val response = PutResponse.newBuilder().setSuccess(success).build()
        responseObserver?.onNext(response)
        responseObserver?.onCompleted()
    }

     override fun getValue(request: AwfulServer.GetRequest?, responseObserver: StreamObserver<AwfulServer.GetResponse>?) {
        val response = AwfulServer.GetResponse.newBuilder()

        val key = request?.key

        if(key != null) {
            val value = db.get(key)

            if(value.isPresent ) {
                println("data found")
                response.value = value.get()
            } else {
            }
            responseObserver?.onNext(response.build())
        }
        responseObserver?.onCompleted()
    }
}

Let’s write our main(). One of the things I really like about Kotlin is you don’t need to wrap a main in a class, it’s a little less boiler plate.

com/rustyrazorblade/awfuldb/server/Main.kt

package com.rustyrazorblade.awfuldb.server

import io.grpc.ServerBuilder

fun main(args: Array<String>) {
    println("Starting this awful piece of garbage.")

    val port = 5000

    val server = Server()
    val s = ServerBuilder.forPort(port).addService(server).build()
    var tmp = s.start()

    println("Server started, listening on " + port)
    tmp.awaitTermination()
}

Let’s write a simple client to test out the database. In real life, we’d want unit tests to ensure the underlying DB functions properly, and use the client library for integration / smoke tests.

com/rustyrazorblade/awfuldb/client/Client.kt

package com.rustyrazorblade.awfuldb.client

import com.google.protobuf.ByteString
import com.rustyrazorblade.awfuldb.AwfulServer
import com.rustyrazorblade.awfuldb.AwfulServiceGrpc
import io.grpc.ManagedChannelBuilder
import java.nio.charset.Charset

fun main(args: Array<String>) {
    println("Running the test client.")

    var channel = ManagedChannelBuilder.forAddress("localhost", 5000).usePlaintext(true).build()

    var stub = AwfulServiceGrpc.newBlockingStub(channel)

    val key = ByteString.copyFromUtf8("test")
    val value = ByteString.copyFromUtf8("value")

    val put = AwfulServer.PutRequest.newBuilder().setKey(key).setValue(value).build()

    println("Putting $key, $value")
    val response = stub.putValue(put)

    println("Done putting, $response")

    val get = AwfulServer.GetRequest.newBuilder().setKey(key).build()
    val response2 = stub.getValue(get)

    println("Get($key): $response2")
    if(response2.value == value) {
        println("Got expected value")
    } else {
        println("Something fishy...")
    }
}

Alright, enough code. We’ve got a working key value store and client that can access it.

What’s next from here? Well, there’s a lot of options. More complex data types, the ability to run in a cluster, maybe add some metadata to make it ready for RAMP? The world is my oyster, or something.

If you found this post helpful, please consider sharing to your network. I'm also available to help you be successful with your distributed systems! Please reach out if you're interested in working with me, and I'll be happy to schedule a free one-hour consultation.

kotlin protobuf grpc gradle