On meta-programming with Rust
This story describes the writing of a Rust macro that mechanically implements wrapper strategies round SQL statements to cut back the boilerplate code necessities when utilizing SQL databases. The macro is accessible on crates.io, however is a work-in-progress at this stage with quite a few limitations, together with solely supporting the rusqlite
wrapper. The supply code for the procedural macro is accessible at https://github.com/juliendecharentenay/derive-sql underneath an MIT open-source license.
I wrote this code and story as I wished to refine my understanding of procedural macros. I selected to develop one thing round SQL, as I’m anticipating I’ll want it for my e-mail masking aspect mission — https://1-ml.com — to retailer and retrieve utilization statistics.
The views/opinions expressed on this story are my very own. This story pertains to my private expertise and selections and is supplied with info within the hope that it is going to be helpful however with none guarantee.
My studying began from the part ‘Learn how to write a {custom} derive macro’ of the Rust Programming Language — https://doc.rust-lang.org/ebook/ch19-06-macros.html#how-to-write-a-custom-derive-macro and the coding journey began with the creation of a library crate known as derive-sql
, with the next proc-macro=true
in Cargo.toml
to declare it as a procedural macro….
The target of the procedural macro is to facilitate the storage and retrieval of a struct
in an SQL database. This story makes use of, for example, the next knowledge construction to retailer a contact:
The DeriveSql
macro is to implement the next strategies for Contact
:
I wish to be aware a few points:
I selected to have the insert
, replace
, and delete
technique eat the article when known as. There aren’t any particular causes for that alternative other than permitting technique chaining — which can or might not be related.
The implementation is predicated on the rusqlite
crate that wraps round SQLite. In the long run, I hope to try to make it agnostic from the SQL engine in order that it’s appropriate with the rust-postgres
crate (and presumably as a stretch goal, so as to add compatibility with IndexedDB API for internet shopper purposes).
For this mission, I selected to make use of test-driven improvement utilizing documentation assessments. With this method, I’ve to put in writing documentation and assessments on the similar time. That’s one side of Rust that I like — it makes it straightforward to use (and even drives me in direction of) good follow.
The documentation assessments tackle the next type — the required //!
main feedback have been eliminated for readability:
I began with the implementation of the create_table
technique first and progressively added the opposite strategies and their related assessments. At this stage, not one of the testing succeeds as nothing has been carried out but.
Troubleshooting
Earlier than speaking in regards to the implementation itself, this paragraph discusses what could be performed when issues don’t go to plan.
Fixing points with procedural macros isn’t easy. Writing a procedural macro is an train in ‘pondering like a compiler.’ One writes code that then generates the code to be compiled. This results in compilation errors in code that aren’t ‘seen’:
Troubleshooting such issues requires both a wonderful developer — which I’m not — a variety of trial and error, or making the ‘invisible’ code seen. The latter is simple to realize utilizing both the cargo-expand
crate — see right here — or a compiler command (that cargo-expand
wraps). The compiler command, proven under, is barely accessible in nightly
— i.e., one wants to alter the Rust toolchain from steady
to nightly
utilizing rustup default nightly
:
cargo rustc --profile=verify -- -Zunpretty=expanded
I choose to make use of the compiler command straight because it saves including one more dependency. Utilizing it within the above state of affairs led me to the next code extract the place my error is clear — I’m trying to make use of r.get(0)?
(strains 8 and 9) nevertheless it will get expanded right into a string instead of code:
It’s typically useful to output the expanded code to a file. The generated code, after a bit little bit of tweaking to keep away from compilation errors as proven under, could be edited to research modifications to be performed and see what success seems like.
Implementation construction
I structured the implementation of the DeriveSql
macro into (a) a lib.rs
that accommodates the documentation and the entry operate derive_sql
and (b) a struct
ImplDerive
— within the file implderive.rs
— that takes care of the code era. The code era is completed within the technique generate
proven under. The strategy scaffold on particular person strategies, that every correspond to a technique to be carried out — i.e., one technique known as impl_create_table
that generates the code for the tactic create_table
.
The E book of Rust in all probability explains how procedural macro works higher than I might. However let me try and phrase it utilizing my very own phrases and understanding.
The procedural macro writes code that’s compiled primarily based on the code on which the procedural macro is utilized. Within the case of a derive macro, the procedural macro is utilized to a struct
, enum
or union
. The code learn by the procedural macro is offered as a TokenStream
— nominally a stream of tokens. This TokenStream
is parsed utilizing the syn
library — see docs — to provide, within the derive macro state of affairs, a useable knowledge construction DeriveInput
— see the documentation — representing the struct
(or enum
or union
) on which the procedural macro is utilized, i.e., the struct
preceded by #[derive(...)]
.
Utilizing the data accessible within the DeriveInput
knowledge construction, the procedural macro is to generate a stream of tokens representing the code generated. For this function, the quote
crate — see the documentation — is used.
My problem has been (and nonetheless is) to grasp and use the DeriveInput
knowledge construction and quote::quote!
macro appropriately.
In my implementation, the ImplDerive
struct
shops a reference to the DeriveInput
object created from the enter TokenStream
as proven under:
On this story, I’m discussing the implementation of two strategies: create_table
, as it’s a comparatively easy, and insert
, as it’s barely extra advanced. The implementation of the opposite strategies could be seen at https://github.com/juliendecharentenay/derive-sql/blob/fundamental/src/implderive.rs.
Implementation of `create_table`
For our instance utilizing Contact
, an implementation of the tactic create_table
would look as follows:
The next code exhibits the procedural macro producing the implementation of the tactic create_table
. For ease of rationalization, two technique calls self.identify()
and self.get_fields_named()
have been expanded.
The implementation is split into two sections: (a) preparation of knowledge and statements — line 6 and 37 — and (b) era of the token stream utilizing the quote!
macro — strains 39 to 45.
Understanding how token streams are generated is, for my part, key. The quote!
interprets Rust code, utilizing token stream and therefore token identifiers. Within the above code line 49, #identify
refers back to the content material of the variable identify
outlined in line 7, which is the identify of the struct
on which the derive macro is utilized, particularly Contact
within the instance.
Studying the above one, I could assume (wrongly) that Contact
could be modified to MyContact
by writing impl My#identify
. Sadly, this isn’t the way it works. To make such a change, one must outline a brand new Ident
utilizing one thing like syn::Ident::new(format!("My{}", identify), identify.span())
.
The problem within the era of the create_table
technique lays in retrieving the members of the struct
on which the derive macro is utilized. These members are used to generate the SQL assertion that creates the desk. That is achieved by (a) checking that the macro is utilized to a knowledge struct
(strains 10–14) and that the struct
fields are named fields (strains 15–19), and (b) constructing the SQL question assertion primarily based on the listing of the named fields utilizing their identify and changing their sort to SQL varieties (strains 24–25).
Implementation of `
choose`
For our instance, utilizing Contact
, an implementation of the tactic choose
would look as follows:
The next code exhibits the procedural macro producing the implementation of the choose
technique:
The principle distinction between choose
and create_table
operate lays in line 39. Line 39 generates a Contact
object from the fields returned by the SQL assertion. The #( #fields: #fields_assignment ), *
ends in the growth of the 2 vectors: fields
and fields_assignement
, with a ,
character separation — ensuing within the assertion identify: r.get(0)?, phone_number: r.get(1)?
. That is defined within the Interpolation part of the quote
crate doc that’s worthwhile studying intimately.
The vector fields
is a vector containing the named discipline identifier, particularly identify
, phone_number
, and so on., as Ident
.
The vector fields_assignment
accommodates the project aspect. Going again to the troubleshooting part, my preliminary method was to make use of a vector of string. However as talked about within the earlier part, the quote!
macro operates on a stream of tokens and therefore every string in a vector of strings is handled as a string token.
At line 32, the above code exhibits the chosen method whereby the vector fields_assignment
is a vector of token streams — every generated utilizing the quote!
macro. It took a while to grasp, however I acquired there ultimately.