Introduction
Comma separated values, also known as CSVs, are one of the most common and basic formats for storing and exchanging tabular datasets for statistical analysis tools and spreadsheet applications. Due to its popularity, it is not uncommon for governing bodies and other important organizations to share official datasets in CSV format.
Despite its simplicity, popularity, and widespread use, it is common for CSV files created using one application to display incorrectly in another. This is because there is no official universal CSV specification to which applications must adhere. As a result, several implementations exist with slight variations.
Most modern operating systems and programming languages, including JavaScript and the Node.js runtime environment, have applications and packages for reading, writing, and parsing CSV files.
In this article, we will learn how to manage CSV files in Node. We shall also highlight the slight variations in the different CSV implementations. Some popular packages we will look at include csv-parser, Papa Parse, and Fast-CSV.
We will also go above and beyond to highlight the unofficial RFC 4180 technical standard, which attempts to document the format used by most CSV implementations.
What is a CSV file?
CSV files are ordinary text files comprised of data arranged in rectangular form. When you save a tabular data set in CSV format, a new line character will separate successive rows while a comma will separate consecutive entries in a row. The image below shows a tabular data set and its corresponding CSV format.
In the above CSV data, the first row is made up of field names, though that may not always be the case. Therefore, it is necessary to investigate your data set before you start to read and parse it.
Although the name Comma Separated Values seems to suggest that a comma should always separate subsequent entries in each record, some applications generate CSV files that use a semicolon as delimiter instead of a comma.
As already mentioned, there is no official universal standard to which CSV implementations should adhere, even though the unofficial RFC 4180 technical standard does exist. However, this standard came into existence many years after the CSV format gained popularity.
How to manage CSV files in Node.js
In the previous section, we had a brief introduction to CSV files. In this section, you will learn how to read, write, and parse CSV files in Node using both built-in and third-party packages.
Using the fs module
The fs module is the de facto module for working with files in Node. The code below uses the readFile function of the fs module to read from a data.csv file:
const fs = require("fs");
fs.readFile("data.csv", "utf-8", (err, data) => {
if (err) console.log(err);
else console.log(data);
});
The corresponding example below writes to a CSV file using the writeFile function of the fs module:
const fs = require("fs");
const data = `
id,name,age
1,Johny,45
2,Mary,20
`;
fs.writeFile("data.csv", data, "utf-8", (err) => {
if (err) console.log(err);
else console.log("Data saved");
});
If you are not familiar with reading and writing files in Node, check out my complete tutorial to reading and writing JSON files in Node.
If you use fs.readFile, fs.writeFile, or its synchronous counterpart like in the above examples, Node will read the entire file into memory before processing it. With the createReadStream and createWriteStream functions of the fs module, you can use streams instead to reduce memory footprint and data processing time.
The example below uses the createReadStream function to read from a data.csv file:
const fs = require("fs");
fs.createReadStream("data.csv", { encoding: "utf-8" })
.on("data", (chunk) => {
console.log(chunk);
})
.on("error", (error) => {
console.log(error);
});
Additionally, most of the third-party packages we shall look at in the following subsections also use streams.
Using the csv-parser package
This is a relatively tiny third-party package you can install from the npm package registry. It is capable of parsing and converting CSV files to JSON.
The code below illustrates how to read data from a CSV file and convert it to JSON using csv-parser. We are creating a readable stream using the createReadStream method of the fs module and piping it to the return value of csv-parser():
const fs = require("fs");
const csvParser = require("csv-parser");
const result = [];
fs.createReadStream("./data.csv")
.pipe(csvParser())
.on("data", (data) => {
result.push(data);
})
.on("end", () => {
console.log(result);
});
There is an optional configuration object that you can pass to csv-parser. By default, csv-parser treats the first row of your data set as field names(headers).
If your dataset doesn’t have headers, or successive data points are not comma-delimited, you can pass the information using the optional configuration object. The object has additional configuration keys you can read about in the documentation.
In the example above, we read the CSV data from a file. You can also fetch the data from a server using an HTTP client like Axios or Needle. The code below illustrates how to go about it:
const csvParser = require("csv-parser");
const needle = require("needle");
const result = [];
const url = "https://people.sc.fsu.edu/~jburkardt/data/csv/deniro.csv";
needle
.get(url)
.pipe(csvParser())
.on("data", (data) => {
result.push(data);
})
.on("done", (err) => {
if (err) console.log("An error has occurred");
else console.log(result);
});
You need to first install Needle before executing the code above. The get request method returns a stream that you can pipe to csv-parser(). You can also use another package if Needle isn’t for you.
The above examples highlight just a tiny fraction of what csv-parser can do. As already mentioned, the implementation of one CSV document may be different from another. Csv-parser has built-in functionalities for handling some of these differences.
Though csv-parser was created to work with Node, you can use it in the browser with tools such as Browserify.
Using the Papa Parse package
Papa Parse is another package for parsing CSV files in Node. Unlike csv-parser, which works out of the box with Node, Papa Parse was created for the browser. Therefore, it has limited functionalities if you intend to use it in Node.
We illustrate how to use Papa Parse to parse CSV files in the example below. As before, we have to use the createReadStream method of the fs module to create a read stream, which we then pipe to the return value of papaparse.parse().
The papaparse.parse function you use for parsing takes an optional second argument. In the example below, we pass the second argument with the header property. If the value of the header property is true, papaparse will treat the first row in our CSV file as column(field) names.
The object has other fields that you can look up in the documentation. Unfortunately, some properties are still limited to the browser and not yet available in Node.
const fs = require("fs");
const Papa = require("papaparse");
const results = [];
const options = { header: true };
fs.createReadStream("data.csv")
.pipe(Papa.parse(Papa.NODE_STREAM_INPUT, options))
.on("data", (data) => {
results.push(data);
})
.on("end", () => {
console.log(results);
});
Similarly, you can also fetch the CSV dataset as readable streams from a remote server using an HTTP client like Axios or Needle and pipe it to the return value of papa-parse.parse() like before.
In the example below, I illustrate how to use Needle to fetch data from a server. It is worth noting that making a network request with one of the HTTP methods like needle.get returns a readable stream:
const needle = require("needle");
const Papa = require("papaparse");
const results = [];
const options = { header: true };
const csvDatasetUrl = "https://people.sc.fsu.edu/~jburkardt/data/csv/deniro.csv";
needle
.get(csvDatasetUrl)
.pipe(Papa.parse(Papa.NODE_STREAM_INPUT, options))
.on("data", (data) => {
results.push(data);
})
.on("end", () => {
console.log(results);
});
Using the Fast-CSV package
This is a flexible third-party package for parsing and formatting CSV data sets that combines @fast-csv/format and @fast-csv/parse packages into a single package. You can use @fast-csv/format and @fast-csv/parse for formatting and parsing CSV datasets, respectively.
The example below illustrates how to a read CSV file and parse it to JSON using Fast-CSV:
const fs = require("fs");
const fastCsv = require("fast-csv");
const options = {
objectMode: true,
delimiter: ";",
quote: null,
headers: true,
renameHeaders: false,
};
const data = [];
fs.createReadStream("data.csv")
.pipe(fastCsv.parse(options))
.on("error", (error) => {
console.log(error);
})
.on("data", (row) => {
data.push(row);
})
.on("end", (rowCount) => {
console.log(rowCount);
console.log(data);
});
Above, we are passing the optional argument to the fast-csv.parse function. The options object is primarily for handling the variations between CSV files. If you don’t pass it, csv-parser will use the default values. For this illustration, I am using the default values for most options.
In most CSV datasets, the first row contains the column headers. By default, Fast-CSV considers the first row to be a data record. You need to set the headers option to true, like in the above example, if the first row in your data set contains the column headers.
Similarly, as we mentioned in the opening section, some CSV files may not be comma-delimited. You can change the default delimiter using the delimiter option as we did in the above example.
Instead of piping the readable stream as we did in the previous example, we can also pass it as an argument to the parseStream function as in the example below:
const fs = require("fs");
const fastCsv = require("fast-csv");
const options = {
objectMode: true,
delimiter: ",",
quote: null,
headers: true,
renameHeaders: false,
};
const data = [];
const readableStream = fs.createReadStream("data.csv");
fastCsv
.parseStream(readableStream, options)
.on("error", (error) => {
console.log(error);
})
.on("data", (row) => {
data.push(row);
})
.on("end", (rowCount) => {
console.log(rowCount);
console.log(data);
});
The functions above are the primary functions you can use for parsing CSV files with Fast-CSV. You can also use the parseFile and parseString functions, but we won’t cover them here. For more about them, you should to read the documentation.
Conclusion
The Comma Separated Values format is one of the most popular formats for data exchange. CSV datasets consist of simple text files readable to both humans and machines. Despite its popularity, there is no universal standard.
The unofficial RFC 4180 technical standard attempts to standardize the format, but some subtle differences exist among the different CSV implementations. These differences exist because the CSV format started before the RFC 4180 technical standard came into existence. Therefore, it is common for CSV datasets generated by one application to display incorrectly in another application.
You can use the built-in functionalities or third-party packages for reading, writing, and parsing simple CSV datasets in Node. Most of the CSV packages we looked at are flexible enough to handle the subtle differences resulting from the different CSV implementations.
The post A complete guide to CSV files in Node.js appeared first on LogRocket Blog.
from LogRocket Blog https://ift.tt/B7izI85
via Read more
