Rust, second impression

Six months ago I published “Rust, first impression”, an article in which I give my feeling about this language which is like no other.

After almost a year of use, and quite a few lines of code written, some readers asked me for a sequel, insisting on the challenges I had faced, or the tips I could give.

What did I achieve with Rust?

I wrote mainly small tools (CLI) for my own use or for clients, I touched Web development (which I practiced a lot more with Go), and I have been working for several months on a software of a certain size, a PACS (a kind of ECM / Network Proxy / Image processing system / Application Router used in the medical imaging world).

It is clearly this last project that gave me the most trouble.

The Rust ecosystem maturity

When developing a serious enterprise application, you are often faced with two challenges:

1. To have the libraries allowing you to interact with other components: database, web service, (in my case) DICOM network protocol, distributed storage system (e.g. S3), authentication and authorization system (IAM), and a set of various services provided by your Cloud provider.
2. Have the right software engineering tools (IDE, debugger, CI-CD chain, static or dynamic code analysis system, APM, etc.)

Let’s start by the end and speak about tooling.

The right tool for the right use : IDE / Debugger

The first classical question for a professional developer when they start to work with a new language is : “is it supported by my IDE ?”

The IDE (Integrated Development Environment) is usually (for 99.9% of humans) a productivity booster in the daily work of a developer.

They provide auto-completion, code inspection, refactoring automation… and a lot of useful functions that allow to work faster.

I use mainly IntelliJ from Jetbrain with appropriate plugins for different languages I work with (Java, Scala, Kotlin, Go, JS, PHP, yaml for Kubernetes, Protobuf, CSS, HTML, …).

So it was clear for I just had to add the Rust plugin a start working, and that’s what I did, at first.

The operating mode quickly turned out to be much too slow (even with my overpowered Macbook M1).

As soon as my projects reached a respectable size, the real-time code inspection system became unusable because it was much too slow.

The various tuning attempts I made did not help, the Macros extension time was the biggest problem.

So I had to resign myself to study the available alternatives.

Quickly VSCode (which I had already used a little) presented itself with two usable plugins:

• The official plugin which is simply called “Rust”, but which suffers from the same performance issues as the Jetbrains one
• An alternative plugin called “rust-analyzer” from Ferrous Systems which has the triple advantage of being much faster, much more functional and updated much more frequently.

So I opted for the VSCode + rust-analyzer couple, with some additional useful plugins :

• “lldb” to allow native debug
• “crates” to manage crates auto-completion in your Cargo.toml
• “Git Graph” to improve VSCode git management

VSCode is clearly a step down on the scale of evolution and cleverness of IDEs compared to IntelliJ, but it does the job.

Its integrated debug mode will rely on lldb plugin for Rust and globally, this work fine.

CI/CD

For the CI/CD you have the choice today.

Most tools are language agnostic, whether they are on premisses or SaaS.

I tend to favor on premisses for everything that is related to my professional intellectual property, and when its implementation does not involve significant overhead.

So I use the excellent tool drone.io , a real Swiss army knife of build and integration, open-source moreover.

It installs in 2 minutes on any Kubernetes cluster and works like a charm.

There is almost no configuration to do in the tool, the whole description of the build, test and deployment steps are done in a .drone.yaml file located in your project’s Git repo.

Drone will then instantiate in a kubernetes ephemeral namespace the pods needed for each step, including a complete integration test environment (with databases and other services) if needed.

These namespaces are automatically destroyed after the build, leaving your infrastructure clean.

Here is a simple example of .drone.yaml file allowing to build a small Rust program to two targets : Linux/x86_64 and Windows/x86_64 :

You can add as many steps as you want, to run unit tests, integration tests, benchmarks (to detect performance deviations) and of course, automatic packaging or/and deployment steps.

TIP : you can build your own rust:latest image with your usual target and/or additional tools to save some build time.

Code analysis tools

I usually use SonarQube on the other languages I use for static code analysis. It is then just an extra step in my .drone.yaml.

However, the Rust compiler is so powerful that, for the moment at least, I don’t see the point of adding this step for this language.

Potential errors and security risks are not only detected at compile time but also in a blocking way.

A static analysis tool would not have much more added value, that’s maybe why SonarQube has not added Rust to the list of supported languages.

So, code analysis tools are not a priority or a “must have” for Rust, even in a professional environment.

APM / Tracing

Over the past five years, Application Performance Monitoring and Application Tracing have evolved considerably.

These technologies and services have become indispensable when an application or platform is to be deployed in production.

Indeed, with the generalization of micro-service architectures, performance monitoring and incident analysis has quickly become a headache.

It is necessary to collect logs and traces from multiple components, to classify them chronologically, to integrate the dependency tree between them and finally to be able to reconstitute the chain of calls and responses that have allowed to obtain a given result for a given call.

Standards and software dedicated to this issue have been developed.

Examples include:

• Open Telemetry
• Open Tracing
• Jaeger
• Prometheus
• New Relic (Commercial / SaaS)
• DataDog (Commercial / SaaS)

The implementation of some of these tools requires a modification of your code (in order to declare the operations to be measured/monitored, to add the logic of transmission of the collected data, etc.), for others the installation of a simple agent on the server (or in the Pod, or in the JVM) can be sufficient.

This will depend not only on the tool used but also on the language with which your component was developed.

For languages requiring an interpreter or a VM (PHP, Ruby, Java, …) a well placed agent will be able to integrate with the interpretation engine and capture all the necessary information during execution.

But for Rust, there is neither interpreter nor VM (this is one of its invaluable qualities).

Therefore there is no other choice than to instrument your code yourself, like in the following example :

Although there are libraries available for most of the tools/platforms mentioned above, they are often less complete and less well documented (for the moment) than for other languages.

Again, I think this is simply a maturity issue that should be resolved with time.

Available libraries

So, what about available libraries for your daily challenges ?

I have always found what I needed on crates.io, however I have been frustrated in some cases.

In the Rust ecosystem, the available open-source libraries are gathered in a central repository named crates.io , which is directly used by Cargo, the build and dependency management tool of Rust.

At the time of writing, it has 67,581 libraries, each of which usually has several versions.

This may seem weak compared to other languages (Go, Java, JS), and it is.

Despite these ten years (the same age as Go), Rust lags behind in terms of available libraries.

This is mainly due to the fact that it remains (unfairly) a niche language (26th in the TIOBE 2021 ranking although first in the most loved languages in the StackOverflow ranking) and that, as mentioned in my first article, it is less accessible to junior developers.

Despite this, there is a good chance that you will find what you are looking for in crates.io, which offers libraries that you will not find in any other language (very low level tools to talk directly with hardware without going through any OS (embedded), high performance 3D rendering libraries, crypto-currency or blockchain management libraries, …).

You will often simply have less choice, do to the 30 or 100 alternatives that Java can offer you for a given need, you will have the choice between 2 or 3 alternatives in Rust.

But the quantity does not prejudge the quality, and it makes the choice easier.

That being said, here are the three areas that surprised me the most.

ORM (Object-Relational Mapping)

These tools are frequently used in enterprise applications because they simplify the management of structured data persistence.

They ensure the conversion between your representation of the business domain data within the program and their storage in a database.

Although they must be used with care (beware of performance impacts), they save a lot of time and are sometimes even essential to keep your application code readable and controllable.

I tested several ORMs in Rust :

• Diesel.rs (the most known and maintained),
• Rbatis,
• ORMX,
• SQLX (not really an ORM but rather a query generator).

Finally and for my use case (a rather complex business domain with nested data structures at several levels), the only one that met the needs is Diesel.rs.

However, it is not yet perfect, although it is relatively powerful, it is not easy to use compared to its Go counterpart (e.g. GORM).

The migration management is a bit surprising (but powerful when you finally understand the principle), the association management is very limited (officially to avoid N+1 query bugs) and its DSL relies massively on macros which does not facilitate the analysis by your IDE and the auto-completion functions.

Let’s focus on the association management.

On many ORM, you can define some nested structures and let the ORM manage the classical INSERT / UPDATE / SELECT / DELETE operations automatically, for example in Go with GORM :

With Diesel.rs you can’t do that.

You have to define your structures independently, INSERT them one by one (in a transaction to avoid incomplete state errors), and when you SELECT them, you have to recreate associations yourself.

Exemple for INSERT :

And for SELECT :

It’s a little frustrating at first.

Another annoying limitation I have encountered is the lack of support for composite foreign key based relationships, this forces to artificially create unique primary keys computed from the values of other columns.

Nothing really bad, but once again, we lose flexibility and simplicity of use.

In short, ORM is an area where progress is still possible in Rust, being a tool massively used in enterprise applications, I do not doubt that things do not continue to evolve in the right direction.

I specify that there may be other ORMs, more recent, less known but more advanced. For example, I have not yet tested SeaORM.

gRPC

Another type of technology often used in microservice architectures is gRPC. gRPC is a more efficient, secure and high-performance alternative to REST calls, and is increasingly used in backend exchanges between microservices.

The principle of use for most languages is simple.

After defining your data structures and calls, you use a compiler (protoc) which itself will rely on plugins (one per target language) to generate the client and server code that you would then just have to include in your project.

This is true for all languages except Rust :)

There is no (to my knowledge) protoc plugin for Rust. The best gRPC framework I found is Tonic.

Tonic will indeed generate your client and server code, but not via protoc.

Instead it relies on the advanced multi-stage build functionality offered by Cargo.

You have to define a pre-build step (via a build.rs file) that will use the tonic lib to compile the proto and generate the code that will be added to your project for the final build.

Exemple of build.rs file :

This work perfectly, but it doesn’t simplify your CI/CD pipeline especially when your .proto files are used for multiple languages in the company.

Async / Green Thread / Actor model

As mentioned in my previous article, the management of asynchronous programming or green threads has suffered from some hesitation in their implementation within Rust.

As a result, these features are not directly implemented in the standard library but in external libraries like the excellent Tokio.rs or the no less deserving Actix (two examples among many others).

So we have the advantage of choice, of fast innovation, and of specialization according to the needs, but at the price…. of a choice to be made, and a new API to learn each time you select a new implementation.

Once again, Rust favors the choice of implementation rather than trying to impose one by default (as Go does with Go-routines and channels), in this sense it is more aimed at experienced developers, able to understand the qualities of such or such implementation according to their needs and their target architecture (OS, CPU, embedded, etc.).

The good surprises

Enough about the frustrations, let’s talk about the good surprises.

Error management

Rust has an error handling mode essentially based on enum, an enum in Rust does not only enumerate values, it can also be used to describe a set of finite types.

For example, if you want to express the result of a function whose execution can fail, the most idomatic is to call the Result enum.

A “Result” can therefore take two forms:

• Ok(of any type, the T in definition )
• Err( of any type, the E in definition )

So you can make sure that all execution paths are handled (including error cases) easily, example (intentionally rough) :

Now imagine that you want to record the results of a set of calls (which could be launched in parallel) in a list.

You then want to retrieve the first error in case the result list contains at least one, or all the Ok(values) in the opposite case:

Simple and powerful, right?

Collections

Rust, like Scala, has a very powerful set of collection management features, most of which are grouped in the crate itertools.

For example, you want to clean a list of structures of its duplicates?

Have a look at the itertools documentation, there are many other very useful functions, and as most of the time, applicable to generic types, so to yours.

Automatic type conversion

As explained in my first article, Rust has a type inference system.
But that’s not all, if your type (I.E. structure) implements the From trait then it can be automatically converted into the expected type.

In the example below, db_pool.get()? returns (the ? return from function in case of error) an error of type r2d2::Error (a pool manager error), but it will automatically be converted to the PersistError format, as this is the format expected by the function, simply because the From feature has been implemented for this type.

It is with this kind of small automation that you can write elegant, readable and safe code.

There are still many others, but to use an expression of Pierre de Fermat, I don’t have enough space to present them all.

Conclusion

Using Rust on a daily basis is often a mixture of wonder at its ability to elegantly handle complex (if not impossible) problems that can be solved in other languages, and frustration at the occasional complexity of subjects whose complexity we have become used to ignoring because it is so hidden by these same languages (e.g. ORM).

But generally speaking, it keeps an invaluable quality in my eyes: a program that compiles is a program that works (no crashes at runtime), and that works fast while consuming a minimum of resources.

In this respect, it is simply unbeatable (at least in the open source world) and just for that, in a professional environment, the small inconveniences encountered are not expensive.

I ended my previous article with two questions and two answers:

• Will I continue to use it? Certainly.
• Will I replace Go with Rust? Certainly not.

But I must admit that during these last three months, I didn’t develop any new program in Go. I even converted some of them to Rust… Maybe I’ll review this in 6 or 12 months

For more information on the impact of Rust on the professional environment, see: Rust impact on the engineering team.