On meta-programming with Rust
This story describes the writing of a Rust macro that mechanically implements wrapper strategies round SQL statements to cut back the boilerplate code necessities when utilizing SQL databases. The macro is accessible on crates.io, however is a work-in-progress at this stage with quite a few limitations, together with solely supporting the rusqlite wrapper. The supply code for the procedural macro is accessible at https://github.com/juliendecharentenay/derive-sql underneath an MIT open-source license.
I wrote this code and story as I wished to refine my understanding of procedural macros. I selected to develop one thing round SQL, as I’m anticipating I’ll want it for my e-mail masking aspect mission — https://1-ml.com — to retailer and retrieve utilization statistics.
The views/opinions expressed on this story are my very own. This story pertains to my private expertise and selections and is supplied with info within the hope that it is going to be helpful however with none guarantee.
My studying began from the part ‘Learn how to write a {custom} derive macro’ of the Rust Programming Language — https://doc.rust-lang.org/ebook/ch19-06-macros.html#how-to-write-a-custom-derive-macro and the coding journey began with the creation of a library crate known as derive-sql, with the next proc-macro=true in Cargo.toml to declare it as a procedural macro….
The target of the procedural macro is to facilitate the storage and retrieval of a struct in an SQL database. This story makes use of, for example, the next knowledge construction to retailer a contact:
The DeriveSql macro is to implement the next strategies for Contact:
I wish to be aware a few points:
I selected to have the insert , replace , and delete technique eat the article when known as. There aren’t any particular causes for that alternative other than permitting technique chaining — which can or might not be related.
The implementation is predicated on the rusqlite crate that wraps round SQLite. In the long run, I hope to try to make it agnostic from the SQL engine in order that it’s appropriate with the rust-postgres crate (and presumably as a stretch goal, so as to add compatibility with IndexedDB API for internet shopper purposes).
For this mission, I selected to make use of test-driven improvement utilizing documentation assessments. With this method, I’ve to put in writing documentation and assessments on the similar time. That’s one side of Rust that I like — it makes it straightforward to use (and even drives me in direction of) good follow.
The documentation assessments tackle the next type — the required //! main feedback have been eliminated for readability:
I began with the implementation of the create_table technique first and progressively added the opposite strategies and their related assessments. At this stage, not one of the testing succeeds as nothing has been carried out but.
Troubleshooting
Earlier than speaking in regards to the implementation itself, this paragraph discusses what could be performed when issues don’t go to plan.
Fixing points with procedural macros isn’t easy. Writing a procedural macro is an train in ‘pondering like a compiler.’ One writes code that then generates the code to be compiled. This results in compilation errors in code that aren’t ‘seen’:
Troubleshooting such issues requires both a wonderful developer — which I’m not — a variety of trial and error, or making the ‘invisible’ code seen. The latter is simple to realize utilizing both the cargo-expand crate — see right here — or a compiler command (that cargo-expand wraps). The compiler command, proven under, is barely accessible in nightly — i.e., one wants to alter the Rust toolchain from steady to nightly utilizing rustup default nightly :
cargo rustc --profile=verify -- -Zunpretty=expanded
I choose to make use of the compiler command straight because it saves including one more dependency. Utilizing it within the above state of affairs led me to the next code extract the place my error is clear — I’m trying to make use of r.get(0)? (strains 8 and 9) nevertheless it will get expanded right into a string instead of code:
It’s typically useful to output the expanded code to a file. The generated code, after a bit little bit of tweaking to keep away from compilation errors as proven under, could be edited to research modifications to be performed and see what success seems like.
Implementation construction
I structured the implementation of the DeriveSql macro into (a) a lib.rs that accommodates the documentation and the entry operate derive_sql and (b) a struct ImplDerive — within the file implderive.rs — that takes care of the code era. The code era is completed within the technique generate proven under. The strategy scaffold on particular person strategies, that every correspond to a technique to be carried out — i.e., one technique known as impl_create_table that generates the code for the tactic create_table.
The E book of Rust in all probability explains how procedural macro works higher than I might. However let me try and phrase it utilizing my very own phrases and understanding.
The procedural macro writes code that’s compiled primarily based on the code on which the procedural macro is utilized. Within the case of a derive macro, the procedural macro is utilized to a struct , enum or union. The code learn by the procedural macro is offered as a TokenStream — nominally a stream of tokens. This TokenStream is parsed utilizing the syn library — see docs — to provide, within the derive macro state of affairs, a useable knowledge construction DeriveInput — see the documentation — representing the struct (or enum or union ) on which the procedural macro is utilized, i.e., the struct preceded by #[derive(...)] .
Utilizing the data accessible within the DeriveInput knowledge construction, the procedural macro is to generate a stream of tokens representing the code generated. For this function, the quote crate — see the documentation — is used.
My problem has been (and nonetheless is) to grasp and use the DeriveInput knowledge construction and quote::quote! macro appropriately.
In my implementation, the ImplDerive struct shops a reference to the DeriveInput object created from the enter TokenStream as proven under:
On this story, I’m discussing the implementation of two strategies: create_table, as it’s a comparatively easy, and insert, as it’s barely extra advanced. The implementation of the opposite strategies could be seen at https://github.com/juliendecharentenay/derive-sql/blob/fundamental/src/implderive.rs.
Implementation of `create_table`
For our instance utilizing Contact , an implementation of the tactic create_table would look as follows:
The next code exhibits the procedural macro producing the implementation of the tactic create_table. For ease of rationalization, two technique calls self.identify() and self.get_fields_named() have been expanded.
The implementation is split into two sections: (a) preparation of knowledge and statements — line 6 and 37 — and (b) era of the token stream utilizing the quote! macro — strains 39 to 45.
Understanding how token streams are generated is, for my part, key. The quote! interprets Rust code, utilizing token stream and therefore token identifiers. Within the above code line 49, #identify refers back to the content material of the variable identify outlined in line 7, which is the identify of the struct on which the derive macro is utilized, particularly Contact within the instance.
Studying the above one, I could assume (wrongly) that Contact could be modified to MyContact by writing impl My#identify. Sadly, this isn’t the way it works. To make such a change, one must outline a brand new Ident utilizing one thing like syn::Ident::new(format!("My{}", identify), identify.span()).
The problem within the era of the create_table technique lays in retrieving the members of the struct on which the derive macro is utilized. These members are used to generate the SQL assertion that creates the desk. That is achieved by (a) checking that the macro is utilized to a knowledge struct (strains 10–14) and that the struct fields are named fields (strains 15–19), and (b) constructing the SQL question assertion primarily based on the listing of the named fields utilizing their identify and changing their sort to SQL varieties (strains 24–25).
Implementation of `choose`
For our instance, utilizing Contact, an implementation of the tactic choose would look as follows:
The next code exhibits the procedural macro producing the implementation of the choose technique:
The principle distinction between choose and create_table operate lays in line 39. Line 39 generates a Contact object from the fields returned by the SQL assertion. The #( #fields: #fields_assignment ), * ends in the growth of the 2 vectors: fields and fields_assignement, with a , character separation — ensuing within the assertion identify: r.get(0)?, phone_number: r.get(1)? . That is defined within the Interpolation part of the quote crate doc that’s worthwhile studying intimately.
The vector fields is a vector containing the named discipline identifier, particularly identify , phone_number , and so on., as Ident.
The vector fields_assignment accommodates the project aspect. Going again to the troubleshooting part, my preliminary method was to make use of a vector of string. However as talked about within the earlier part, the quote! macro operates on a stream of tokens and therefore every string in a vector of strings is handled as a string token.
At line 32, the above code exhibits the chosen method whereby the vector fields_assignment is a vector of token streams — every generated utilizing the quote! macro. It took a while to grasp, however I acquired there ultimately.
