At Fieldwire, we use Rust for one of our API servers. As is common when it comes to Rust, we use the excellent Serde framework for our serialization & deserialization needs. The only format we support is JSON & for that we use serde-json.

While for the most part this works well, there is one significant issue which is explained best with an example. Consider that we have an incoming json payload looking like this:

{
    "name": "John Doe",
    "age": null
}

The following will deserialize as defined in the test method:

#[derive(serde::Deserialize)]
pub struct Person {
    pub name: String, 
    pub age: Option<u16>,
    pub address: Option<String>,
}

impl Person {
    pub fn test() {
        let json = r#"{
            "name": "John Doe",
            "age": null
        }"#;

        let person = serde_json::from_str::<Person>(json).unwrap();
        assert_eq!(person.name, "John Doe");
        assert!(person.age.is_none());
        assert!(person.address.is_none());
    }
}

The keen eyed among you might have noticied that both age & address deserialize to None whereas one of those fields were null in the incoming payload & the other one was altogether missing from it. The reason behind this is that Option is a 2 state enum while the incoming value could be in one of 3 states: present, null or missing from the payload. This is a known issue & was already discussed about in this Github issue. David Tolnay, the main author of Serde came up with a nifty solution of using Option<Option<T>> to handle having more than 2 states which was even added to the serde_with crate for easier consumption by others.

While this works, the developer experience of using nested Options isn’t great and it is also very easy to forget adding those extra annotations everytime a value could be in one of the 3 states. To work around this, we introduced a 3-state enum to our codebase:

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum Value<T> {
    Missing,
    Null,
    Present(T),
}

In addition to this, we also came up with a new macro json_deserialize which does the following:

  • derives serde_json::Deserialize for structs this is annotated on
  • raises compilation error if Option is used (enforces using Value instead)
  • tags fields using Value to use Value::default when deserializing if field not in payload (so it becomes Value::Missing)

Putting the new enum & macro to use, the example we started with now becomes:

#[json_deserialize]
pub struct Person {
    pub name: String, 
    pub age: Value<u16>,
    pub address: Value<String>,
}

impl Person {
    pub fn test() {
        let json = r#"{
            "name": "John Doe",
            "age": null
        }"#;

        let person = serde_json::from_str::<Person>(json).unwrap();
        assert_eq!(person.name, "John Doe");
        assert!(person.age.is_null());
        assert!(person.address.is_missing())
    }
}

With this change we can now easily tell apart for which field the clients had sent null & which field is not even in the payload. This is absolutely required for the way we handle our HTTP PATCH calls:

  • passing in null says that the client wants to nullify the value in DB for that field
  • missing field in the payload says that the client wants to let the value in DB be as is