↓Skip to main content
  1. Posts/

Experience Report - NYTimes Marvin for building Microservices on Appengine

·2 mins

NYTimes recently open-sourced Marvin, a library for building Microservices on Google AppEngine. The project is based off their earlier library Gizmo. I was looking for a similar project to implement some of my own microservices, so I thought that it was an opportune moment to give this toolkit a shot.

By way of documentation, the repository has a sample application, readinglist, and its documentation is lacking. The README page has bit description about the files. It took me a while to understand the machinations because I had not worked on Gizmo earlier.

Marvin is a toolkit to implement Microservices which use protocol buffers or JSON as their communication format. First off, Marvin expects an OpenAPI specification file. You use the spec file to generate a .proto file. openapi2proto is another open source project from NYTimes which comes handy here. You can then use the .proto file to generate Go code for services. Marvin’s MixService interface gives you endpoints to implement protocol buffer and JSON Web Services.Here’s a run-down of the workflow:

  • Write the openapi spec in a file, say service.yaml

  • Use openapi2proto to generate the service specification in Protobuf. The protobuf file has domain specific models from the openapi definitions.

    openapi2proto -spec service.yaml > service.proto

  • Generate Go code for the service using the protobuf compiler

    protoc --go _out= . service.proto

  • Implement the service method, usually it is the marvin.MixService which is an interface for implementing both json and protobuf services. I discovered impl, a utility to generate stub implementation for any interface. You can generate the stub implementation like so:

    impl 's Service' github.com/NYTimes/marvin.MixService

    You can then add custom middlewares, routing and Endpoints by implementing the methods from the interface.

  • Write the usual app.yaml file for deploying the project on GAE. In the main function, bootstrap the service like so:

    1
2
3
4

    	func main() {
		marvin.Init(api.NewService())
		appengine.Main()
}

  • Use the local setup to test your microservice

    dev_appserver.py server/app.yml

It took me a while to figure out the workflow. I liked the openapi-first approach for developing Microservice APIs. While openapi2proto generates the domain objects from the definitions file, it discards the endpoints information, which you have to implement separately. It’d be nice to generate stub implementation from the paths section of the spec file. To sum up:

  • You can generate protobuf and JSON webservices is a huge plus.
  • You can generate neat documentation from swagger-gen tool for your APIs.
  • It’d be great if the tool also generated stub implementation for the endpoints using the paths information from the openapi spec file.