This is part of my series of posts about the PolyAnno project – more here
Basics
Eventually I decided it would be easier to separate out development of the longer term storage i.e. the Node/Express/MongoDB/Mongoose framework such that it can be installed separately and understood more simply in isolation.
The Github repo can currently be found at https://github.com/BluePigeons/polyanno_storage .
The NPM page can be found here: https://www.npmjs.com/package/polyanno_storage.
The aim of this page is to elaborate in more detail about the development of the package than the Github and NPM documentation provides for quick usage, as part of the series for the whole PolyAnno framework.
Setup
To use it you need to install it via NPM (the package.json file should ensure that all the other dependencies are then installed with it).
Then you need to start your MongoDB database somewhere. You need to open your copy of the “routes_setup.js” file in the “lib” folder and change the “exports.databaseport” variable to your MongoDB URL.
exports.databaseport = "mongodb://localhost:27017/testMongoDB"
Then similarly, you need to change the “thisport” variable and “hostname” variable in the routes_setup.js file to the URL port and host name you will be using for this website. I know that this whole editing the file thing isn’t ideal and is on the to-do list for future development.
var hostname = "http://localhost:"; var thisport = 8080;
Then you need to include a line in your Polyanno package (if there isn’t one already) importing the functions from polyanno_storage into your backend MongoDB to use as you wish!
var polyanno_storage = require("polyanno_storage");
Please note that I have already setup PolyAnno and Polyglot to import this already as it is their defaults to use polyanno_storage.
Structure
All the files sit inside the “lib” folder but the routing gives the files a tree structure within that.
Firstly, at the top of that tree is the polyanno_storage.js which simply provides all the other files as exports of that one so as to make the package cohesive.
exports.setup = require('./routes_setup');
exports.annotations = require('./annotations');
exports.vectors = require('./vectors');
exports.transcriptions = require('./transcriptions');
exports.translations = require('./translations');
exports.users = require('./usersroute');
Within each of those, several of the others are imported in, creating horizontal connections between files, but I still felt it was important to separate into separate files for clarity.
Then several ‘key’ or generic pieces of information or functions that I found myself frequently reusing throughout development of the backend have been set in the routes_setup.js file.
Routes_Setup: Setup Information
The first part of the file needs to be adjusted according to the user – ideally this would be improved so as to allow easier updating for development, but for now require a manual adjustment.
var hostname = "http://localhost:"; var thisport = 8080; exports.applicationport = thisport; exports.databaseport = "mongodb://localhost:27017/testMongoDB";
The second part defines the structures of the URLs used for the routes as defined and expected in the Polyanno package itself.
var website_address = hostname + thisport.toString();
exports.vectorURL = website_address.concat("/api/vectors/");
exports.transcriptionURL = website_address.concat("/api/transcriptions/");
exports.translationURL = website_address.concat("/api/translations/");
exports.annotationURL = website_address.concat("/api/annotations/");
exports.userURL = website_address.concat("/user/");
Routes_Setup: Functions
The last part is generic functions that were frequently used in the checking of the different JSON annotations.
isUseless
var rejectionOptions = new Set(["false",'""' , null , false , 'undefined']);
exports.isUseless = function(something) {
if (rejectionOptions.has(something) || rejectionOptions.has(typeof something)) { return true; }
else { return false; };
};
var isUseless = function(something) {
if (rejectionOptions.has(something) || rejectionOptions.has(typeof something)) { return true; }
else { return false; };
};
asyncPush
exports.asyncPush = function(addArray, oldArray) {
var theArray = oldArray;
var mergedArray = function() {
addArray.forEach(function(addDoc){
theArray.push(addDoc);
});
if (theArray.length = (oldArray.length + addArray.length)) {
return theArray;
};
};
return mergedArray();
};
fieldMatching
exports.fieldMatching = function(searchArray, field, fieldValue) {
if (isUseless(searchArray) || isUseless(field) || isUseless(fieldValue)) { return false }
else {
var theMatch = false;
searchArray.forEach(function(childDoc){
if (childDoc[field] == fieldValue) {
theMatch = childDoc;
};
});
return theMatch;
};
};
jsonFieldEqual
exports.jsonFieldEqual = function(docField, bodyDoc, bodyField) {
if (isUseless(bodyDoc[bodyField]) == false ) { return bodyDoc[bodyField]; }
else { return docField; };
This compares the values of one JSON with another JSON for agiven field. This ensures that if the field is blank or invalid for the new JSON then the original JSON’s field value is returned, but if a new value exists in the second JSON then it is that returned.
Mongoose Schema
The Mongoose Schema for the different annotation types are defined in the following files:
- newAnno.js
- newTranscription.js
- newTranslation.js
- newUser.js
- newVector.js
Annotation Route Files
Then the individual files to handle the functions for the different annotation types are defined in each of the following files:
- annotations.js
- transcriptions.js
- translations.js
- usersroute.js
- vectors.js
Functions
The table below shows which functions are used by which type of data, and what REST function is expected for usage. To use you would therefore send the relevant REST function to the data type + function name e.g. POST transcriptions.addNew.
The purpose and structure expected for each function should be fairly standard for Mongoose with the exceptions of ‘.voting’ which is explained in more detail in the section on Verification, and ‘.updateOne’ which is explained further below.
.updateOne
The code handling updating for a transcription example, without the section handling the parent/children voting structure fields (see Verification for more details) is as follows:
exports.updateOne = function(req, res) {
var updateDoc = setup.transcription_model.findById(req.params.transcription_id);
updateDoc.exec(function(err, transcription) {
if (err) {res.send(err)};
var jsonFieldPush = function(bodyDoc, theField) {
if (!setup.isUseless(bodyDoc[theField])) {
bodyDoc[theField].forEach(function(subdoc){
transcription[theField].addToSet(subdoc);
});
};
};
transcription.text = setup.jsonFieldEqual(transcription.text, req.body, "text");
transcription.language = setup.jsonFieldEqual(transcription.language, req.body, "language");
/////"parent" field comparison function here
transcription.translation = setup.jsonFieldEqual(transcription.translation, req.body, "translation");
jsonFieldPush(req.body, "metadata");
jsonFieldPush(req.body, "target");
///// functions handling "children" fields are here
transcription.save(function(err) {
if (err) {res.send(err)}
else {res.json(transcription)};
});
});
};
This can be broken into sub-parts, the first of which is simply the standard Node REST format:
exports.updateOne = function(req, res) {
};
Then within that is the use of Mongoose/MongoDB to search by the “id” field for the relevant transcription (in this example), with error catching:
var updateDoc = setup.transcription_model.findById(req.params.transcription_id);
updateDoc.exec(function(err, transcription) {
if (err) {res.send(err)};
transcription.save(function(err) {
if (err) {res.send(err)}
else {res.json(transcription)};
});
});
Then we have use of the generic function “jsonFieldEqual” from the routes_setup file discussed earlier to compare the values of the req JSON and the stored transcription JSON for each given field. This ensures that if the req JSON is blank or invalid for these fields then nothing is changed, but if a new value has been given then they are returned and then saved into database JSON.
transcription.text = setup.jsonFieldEqual(transcription.text, req.body, "text"); transcription.language = setup.jsonFieldEqual(transcription.language, req.body, "language"); transcription.translation = setup.jsonFieldEqual(transcription.translation, req.body, "translation");
Similarly, a new function “jsonFieldPush” is defined and used, that checks if the input has valid values for the given fields, and if so it pushes them to the field arrays (as opposed to replacing them in like in “jsonFieldEqual“).
var jsonFieldPush = function(bodyDoc, theField) {
if (!setup.isUseless(bodyDoc[theField])) {
bodyDoc[theField].forEach(function(subdoc){
transcription[theField].addToSet(subdoc);
});
};
};
jsonFieldPush(req.body, "metadata");
jsonFieldPush(req.body, "target");
This allows the most simplistic updating methodology for developers, simply submitting one JSON with updates of multiple fields in a single REST API request, as opposed to necessitating one request per field.
Next: Homepages, Queues, and Others
This is part of my series of posts about the PolyAnno project – more here
Blue Pigeons 