This matter is a extremely pleasurable one for me. It is fairly widespread in lots of net functions to just accept consumer enter and save a single file to your database. However what about when your customers (otherwise you) wish to carry out a number of inserts in a single command?
On this article, we’ll show methods to create a CSV template and a type to add the CSV file, and methods to parse the CSV right into a Mongoose Mannequin that shall be saved to a MongoDB database.
This text assumes that you’ve got a fundamental understanding of Mongoose and the way it interacts with MongoDB. When you do not, I might recommend studying my Introduction to Mongoose for MongoDB and Node.js article first. This text describes how Mongoose interacts with MongoDB by creating strongly-typed Schemas which a Mannequin is created from. If you have already got understanding of Mongoose, then let’s proceed.
Getting Began
To start, you’d instantiate a brand new Node.js software. In a command immediate, navigate to the place you wish to host your Node.js functions and carry out the next instructions:
mkdir csvimport cd csvimport npm init -y
Utilizing the -y flag, the undertaking get initialised with the default values in place, so the applying will begin with index.js. Earlier than creating and parsing CSV recordsdata, some preliminary setup must be achieved first. We wish to make this an internet software; to try this, you’ll use the Categorical bundle to deal with all the nitty-gritty server setup. In your command immediate, set up Categorical by working the next command:
npm set up specific --save
Since this net software will settle for recordsdata through an internet type, we’re additionally going to make use of the Categorical sub-package Categorical File Add. Let’s set up that now as properly:
npm set up express-fileupload --save
We’ve achieved sufficient preliminary configuration to arrange the net software and create a fundamental net web page that can create a file add type.
Within the root listing, create an index.js file and add the next code snippets. This file units up the net server.
var app = require('specific')();
var fileUpload = require('express-fileupload');
var server = require('http').Server(app);
app.use(fileUpload());
app.get('/', perform (req, res) {
res.sendFile(__dirname + '/index.html');
});
server.hear(8080, () => {
console.log('Server began on port 8080')
})
This file imports Categorical and the Categorical File Add libraries, configures the net software to make use of the File Add, and listens on port 8080. It additionally creates a route utilizing Categorical at “/” which would be the default touchdown web page for the net software. This route returns an index.html file that comprises the net type that can permit a consumer to add a CSV file.
You can begin your net server by working the command in your terminal:
node index.js
It is best to get the message in your terminal Server began on port 8080. Which means you’ve got efficiently linked to the net server.
Within the root listing, create an index.html file. This file creates the shape for importing a CSV file.
<!DOCTYPE html>
<html lang="en">
<head>
<title>Add Authors</title>
</head>
<physique>
<p>Use the shape beneath to add a listing of authors.
Click on <a href="https://code.tutsplus.com/template">right here</a> for an instance template.</p>
<type motion="/" technique="POST" encType="multipart/form-data">
<enter sort="file" identify="file" settle for="*.csv" />
<br/><br/>
<enter sort="submit" worth="Add Authors" />
</type>
</physique>
</html>
This HTML file comprises two vital issues:
- A hyperlink to /template which, when clicked, will obtain a CSV template that may be populated with the knowledge to be imported.
- A type with the
encTypeset asmultipart/form-dataand an enter subject with a kind offilethat accepts recordsdata with a .csv extension. Themultipart/form-datais a technique of encoding utilized in HTML when the shape comprises a file add.
Whenever you go to http://localhost:8080/ you will note the shape created earlier.
As you could have seen, the HTML makes reference to an Creator template. When you learn the Introduction to Mongoose article, an Creator Schema was created. On this article, you’ll recreate this Schema and permit the consumer to bulk import a set of authors into the MongoDB database. Let’s check out the Creator Schema. Earlier than we do this, although, you in all probability guessed it—we have to set up the Mongoose bundle:
npm set up mongoose --save
Creating the Schema and Mannequin
With Mongoose put in, let’s create a brand new writer.js file that can outline the writer schema and mannequin:
var mongoose = require('mongoose');
var authorSchema = mongoose.Schema({
_id: mongoose.Schema.Sorts.ObjectId,
identify: {
firstName: {
sort: String,
required: true
},
lastName: String
},
biography: String,
twitter: {
sort: String,
validate: {
validator: perform(textual content) {
if (textual content !== null && textual content.size > 0)
return textual content.indexOf('https://twitter.com/') === 0;
return true;
},
message: 'Twitter deal with should begin with https://twitter.com/'
}
},
fb: {
sort: String,
validate: {
validator: perform(textual content) {
if (textual content !== null && textual content.size > 0)
return textual content.indexOf('https://www.fb.com/') === 0;
return true;
},
message: 'Fb Web page should begin with https://www.fb.com/'
}
},
linkedin: {
sort: String,
validate: {
validator: perform(textual content) {
if (textual content !== null && textual content.size > 0)
return textual content.indexOf('https://www.linkedin.com/') === 0;
return true;
},
message: 'LinkedIn should begin with https://www.linkedin.com/'
}
},
profilePicture: Buffer,
created: {
sort: Date,
default: Date.now
}
});
var Creator = mongoose.mannequin('Creator', authorSchema);
module.exports = Creator;
With the writer schema and mannequin created, allow us to change gears and give attention to creating the CSV template that may be downloaded by clicking on the template hyperlink. To assist with the CSV template technology, we’ll use the json2csv bundle. This npm bundle converts JSON into CSV with column titles and correct line endings.
npm set up json2csv --save
Now you’ll replace the beforehand created index.js file to incorporate a brand new route for /template. You’ll append this new code for the template path to the beforehand created index.js file
var template = require('./template.js');
app.get("https://code.tutsplus.com/template", template.get);
The very first thing this code does is embody a brand new template.js file (to be created subsequent) and create a route for /template. This route will name a get perform within the template.js file.
With the Categorical server up to date to incorporate the brand new route, let’s create the brand new template.js file:
var json2csv = require('json2csv').parse;
exports.get = perform(req, res) {
var fields = [
'name.firstName',
'name.lastName',
'biography',
'twitter',
'facebook',
'linkedin'
];
var csv = json2csv({ knowledge: '', fields: fields });
res.set("Content material-Disposition", "attachment;filename=authors.csv");
res.set("Content material-Kind", "software/octet-stream");
res.ship(csv);
};
This file first contains the put in json2csv bundle. It then creates and export a get perform. This perform accepts the request and response objects from the Categorical server.
Contained in the perform, you create an array of the fields that you just wish to embody within the CSV template. This may be achieved considered one of two methods. The primary means (that’s achieved on this tutorial) is to create a static listing of the fields to be included within the template. The second means is to dynamically create the listing of fields by extracting the properties from the writer schema.
The second means may very well be achieved with the next code:
var fields = Object.keys(Creator.schema.obj);
It might have been a pleasant thought to make use of the dynamic technique, but it surely turns into a bit sophisticated when you do not wish to embody a number of properties from the Schema to the CSV template. On this case, our CSV template would not embody the _id and created properties as a result of these shall be populated through code. Nonetheless, if you do not have fields you want to exclude, the dynamic technique will work as properly.
Creating the CSV Template
With the array of fields outlined, you’ll use the json2csv bundle to create the CSV template out of your JavaScript object. This csv object would be the outcomes of this route.
And at last, utilizing the res property from the Categorical server, two header properties shall be set that can pressure the downloading of an authors.csv file.
At this level, if you happen to have been to run your Node software and navigate to http://localhost:8080/ in your net browser, the net type can be displayed with a hyperlink to obtain the template. Clicking the hyperlink to obtain the template will assist you to obtain the authors.csv file. This file shall be populated earlier than it’s uploaded.
Opening the template file in Microsoft Excel, the CSV file ought to be populated as such:
Right here is an instance of the populated CSV file
identify.firstName,identify.lastName,biography,twitter,fb,linkedin Mary,Doe,frontend developer,https://twitter.com/mary,https://www.fb.com/mary,https://www.linkedin.com/mary Michael,Doe,backend developer,https://twitter.com/michael,https://www.fb.com/michael,https://www.linkedin.com/michael John,Doe,app developer,https://twitter.com/john,https://www.fb.com/john,https://www.linkedin.com/john
This instance, when uploaded, will create three authors.
The puzzle items are beginning to come collectively and type an image. Let’s get to the meat and potatoes of this instance and parse that CSV file. The index.js file requires some updating to hook up with MongoDB and create a brand new POST route that can settle for the file add:
var app = require('specific')();
var fileUpload = require('express-fileupload');
var server = require('http').Server(app);
var template = require('./template.js');
var add = require('./add.js');
app.use(fileUpload());
app.get('/', perform (req, res) {
res.sendFile(__dirname + '/index.html');
});
app.get("https://code.tutsplus.com/template", template.get);
app.submit('/', add.submit);
mongoose.join('mongodb://localhost/csvimport');
server.hear(8080, () => {
console.log('Server began on port 8080')
})
With a database connection and a brand new POST route configured, it is time to parse the CSV file. Fortunately there are a number of nice libraries that help with this job. For this tutorial, we’ll use the fast-csv bundle that may be put in with the next command:
npm set up fast-csv --save
The POST route was created equally to the template route which invokes a submit perform from the add.js file. It isn’t vital to position these capabilities in separate recordsdata; nonetheless, it’s best to create separate recordsdata for these routes because it helps preserve the code good and arranged.
Submitting Information
And at last, let’s create the add.js file that comprises the submit perform that is named when the beforehand created type is submitted:
var csv = require('fast-csv');
var mongoose = require('mongoose');
var Creator = require('./writer');
exports.submit = perform (req, res) {
if (!req.recordsdata)
return res.standing(400).ship('No recordsdata have been uploaded.');
var authorFile = req.recordsdata.file;
var authors = [];
csv
.fromString(authorFile.knowledge.toString(), {
headers: true,
ignoreEmpty: true
})
.on("knowledge", perform(knowledge){
knowledge['_id'] = new mongoose.Sorts.ObjectId();
authors.push(knowledge);
})
.on("finish", perform(){
Creator.create(authors, perform(err, paperwork) {
if (err) throw err;
});
res.ship(authors.size + ' authors have been efficiently uploaded.');
});
};
Fairly a bit is occurring on this file. The primary three strains embody the mandatory packages that shall be required to parse and save the CSV knowledge.
Subsequent, the submit perform is outlined and exported to be used by the index.js file. Inside this perform is the place the magic takes place.
The perform first checks that there’s a file contained within the request physique. If there’s not, an error is returned indicating {that a} file have to be uploaded.
When a file has been uploaded, a reference to the file is saved to a variable referred to as authorFile. That is achieved by accessing the recordsdata array and the file property within the array. The file property matches the identify of my file enter identify that has been first outlined within the index.html instance.
An empty authors array that shall be populated because the CSV file is parsed is created. This array shall be used to avoid wasting the info to the database.
The fast-csv library is now referred to as by leveraging the fromString perform. This perform accepts the CSV file as a string. The string is extracted from the authorFile.knowledge property. The knowledge property comprises the contents of the uploaded CSV file.
The following line contains two choices to the fast-csv perform: headers and ignoreEmpty. These are each set to true. This tells the library that the primary line of the CSV file will comprise the headers and that vacant rows ought to be ignored.
With the choices configured, we arrange two listener capabilities which might be referred to as when the knowledge occasion and the finish occasion are triggered. The knowledge occasion is named as soon as for each row of the CSV file. This occasion comprises a JavaScript object of the parsed knowledge.
The article is up to date to incorporate the _id property of the writer schema with a brand new ObjectId. This object is then added to the authors array.
When the CSV file has been totally parsed, the finish occasion is triggered. Contained in the occasion callback perform, we’ll name the create perform on the writer mannequin, passing the array of authors to it.
If an error happens attempting to avoid wasting the array, an exception is thrown; in any other case, successful message is exhibited to the consumer indicating what number of authors have been uploaded and saved to the database.
The database schema ought to be just like this:
If you want to see the complete supply code, you may take a look at the GitHub repository with the code.
Conclusion
On this tutorial, now we have solely uploaded a few information. In case your use case requires the power to add hundreds of information, it is perhaps a good suggestion to avoid wasting the information in smaller chunks.
This may be achieved a number of methods. If I have been to implement it, I might recommend updating the knowledge callback perform to test the size of the authors array. When the array exceeds your outlined size, e.g. 100, name the Creator.create perform on the array, after which reset the array to empty. This can then save the information in chunks of 100. Remember to depart the ultimate create name within the finish callback perform to avoid wasting the ultimate information.
Take pleasure in!
This submit has been up to date with contributions from Mary Okosun. Mary is a software program developer based mostly in Lagos, Nigeria, with experience in Node.js, JavaScript, MySQL, and NoSQL applied sciences.
Submit thumbnail generated with OpenAI DALL-E.
