Comparing Performance of gRPC, Web API and WCF Services

I started paying attention to gRPC about a year ago. Almost every gRPC advocate and evangelist I came across touted its performance capabilities as a significant value-add over other available service technologies.

The release of .NET Core 3.0 came with a new gRPC project template so I dove right in. During my explorations I found myself continually wondering how its performance stacked up against other service types available in .NET. A couple months ago I decided to circle back and satisfy this curiosity.

Methodology

After some consideration I decided to quantify performance characteristics for gPRC, Web API and WCF service implementations built with the latest (2020) .NET technologies. The best way to compare these services would have been to implement each targeting the same version of .NET (Core 3.1). Unfortunately, WCF has never been a well supported technology in .NET Core. Because of this I chose to create the WCF service implementation in .NET Framework 4.8. To help normalize service comparisons I decided to create separate .NET Core and Framework Web API implementations.

In order to compare services I defined a set of operations to be exposed by each and consumed by a common client. Completion times for each of these operations across all implementations could then be analyzed. Operations selected were as follows.

  • Get - Get a fixed collection of objects from the service
  • First - Get a single object from the service (the first in the fixed collection)
  • Send - Send a fixed collection of objects to the service
  • Send One - Send a single object to the service

Additionally, I decided to examine streaming operations available only with gRPC.

  • Get Streaming - Stream a fixed collection of objects from a service
  • Send Streaming - Stream a fixed collection of objects to a service

Setup

I started by creating a the service-compare branch of my Stupid Todo exploratory application. I added projects as needed for a total of four service implementations and a single client.

  • StupidTodo.Grpc - gRPC service project, .NET Core 3.1
  • StupidTodo.WebApi - ASP.NET Web API project, .NET Core 3.1
  • StupidTodo.Framework.WebApi - ASP.NET Web API project, .NET Framework 4.8
  • StupidTodo.Framework.Wcf - ASP.NET WCF project, .NET Framework 4.8
  • StupidTodo.AdminConsole - Console application project used to gather data from each service implementation, .NET Core 3.1

Each service implementation exposes the previously discussed operations and shares the data provider GenFuDataProvider. That provider statically loads 5000 pre-generated Todo objects from a source shared by all implementations.

Each data set was gathered using an instance of the client backed by an instance of a specific service implementation. Both service and client instances were hosted side-by-side on the same physical machine.

Each data gathering cycle began in the client with a warm-up calling each available operation once. Afterwards, operations were executed one at time in the following order 1000 times each.

  • Get
  • Send
  • First
  • Send One
  • Get Streaming (gRPC only)
  • Send Steaming (gRPC only)

Execution times for each operation were recorded in the client.

During my initial research into gRPC performance I came across the blog post REST vs gRPC | Why Milliseconds Matter. In it the author (unnamed) found that the performance difference between gRPC and REST technologies became even more pronounced as the number of clients accessing the services increased. After reading this I decided to perform additional test runs where each set of operations was run on ten separate client instances simultaneously.

Results

Analysis

If we first compare the two primary technologies .NET Core and .NET Framework we find that .NET Core based services generally showed better performance. There were anomalies to this trend in the First and SendOne operations but I think these can be attributed to the extremely short response times. For those operations the uncertainty in measurements was large and thus the accuracy and utility of the data is questionable.

The results of the Get operation were largely as expected. gRPC performed the best followed by .NET Core Web API, then .NET Framework Web API and finally WCF. Additionally, these trends held for both the single client and ten simultaneous clients cases. The outlier was gRPC's service streaming. This operation was considerably slower than other Get variants and service streaming to ten simultaneous clients was especially slow. This is consistent with the fact that gRPC server streaming isn't designed as a high-performance data transfer mechanism. It's primary value proposition can be thought of as providing a long lived connection to a client.

The results of the Send operations were somewhat surprising. Here performance rankings were gRPC, .NET Core Web API, WCF and then .NET Framework Web API. The .NET Core based services behaved as might be expected but the fact that the WCF service outperformed the .NET Framework Web API is fairly shocking. I have no good explanation for the data but I replicated the experiment several times and consistently found similar results. Also, note that data for gRPC client streaming was absent. During testing upwards of 10% of gRPC Send attempts using client streaming resulted in some form of exception so these results were omitted. The common case found the server reporting that the connection had been closed before the client had completed sending all data.

Results from the First operations were again consistent with expectations showing service order by performance was gRPC, .NET Core Web API, .NET Framework Web API and WCF. Note how significant the variance is for the ten simultaneous clients tests in all cases. One interesting feature is that the WCF service shows very small performance differences between the single and ten simultaneous client cases. More testing would be needed here but results would seem to indicate that WCF handles increasing numbers of client connections quite efficiently for small get operations.

Results of the SendOne operations showed a number of unexpected results. Performance rankings were gRPC, WCF, .NET Core Web API and finally .NET Framework Web API. At first glance it seems very strange that WCF would out perform not only .NET Framework Web API (which was also the case in the Send operation) but also .NET Core Web API. However, examination of the deviation shows the variance in the WCF data to be on the order of the mean values. This is a clear indicator that the mean values alone do not give an accurate picture of results. An additional oddity is that the ten simultaneous WCF clients case showed better performance than the single client case. Here again note the significant variance especially in the single client data. If we refrain from drawing conclusions about the WCF data in this operation the remaining data shows trends consistent with expectations.

Conclusions

Through all testing gRPC performance was found to be superior to that of the other examined technologies. Most of the results confirmed the expectations that performance ranking would be gRPC, .NET Core Web API, .NET Framework Web API and finally WCF. The WCF data did exhibit a few unexpected and interesting results but nothing which convinced me that it wasn't the least performant overall.

Despite it's impressive performance gRPC in .NET Core (Grpc.Net) does have some limitations.

First, due to its heavy reliance upon HTTP/2 it is currently (2020) unsuitable as a service layer for most browser based applications. This limitation can be largely mitigated by the use of gRPC-Web but this project is currently "experimental". Though I haven't tested it yet, I suspect there's also a performance cost associated with its use.

To me the most significant limitation of Grpc.Net is that it's entirely reliant upon code generation based on your protobuf (Protocol Buffer) service definitions. Generated code is almost never sufficient as a domain model so you'll likely need to create and maintain a mapping layer between you domain model and the models generated by Grpc.Net. Such mapping code is often significant in size and can be brittle. The project protobuf-net.Grpc attempts to address this issue by allowing you to generate protobuf definitions from C#. It relies upon the use of data annotations in the C# code and supports both its own data annotations and many of those found in the System.ComponentModel.DataAnnotations namespace (same annotations used in WCF). I'd like to test the performance of this framework but my early explorations have been somewhat less than fruitful.

Grpc.Net definitely has potential. Today its use cases often involve service-to-service communication such as you might find in a micro-services architecture. For such cases the superior performance can be of significant benefit. Over time I expect the technology to continue improving and its use cases to broaden. I'll be watching with interest.

References