Important: This documentation covers Yarn 1 (Classic).
For Yarn 2+ docs and migration guide, see yarnpkg.com.

Package detail

gstore-node

sebelga2.4kMIT9.1.0TypeScript support: included

Google Datastore Entities Modeling for Node.js. Validate the Entity properties and type before saving to the Datastore. Advanced cache to speed up entities fetching.

google datastore, gcloud node, google app engine, node.js

readme

gstore-node Tweet

Entities modeling for Google's Datastore

NPM Version Build Status coveralls-image Commitizen friendly

:tada: NEWS: new maintainer!

A few weeks ago I announced that gstore-node was deprecated as I currently don't have bandwidth to work on it. A few days later, Hendrik Schalekamp stepped in and offered to be a maintainer of the project. I was thrilled! :blush: Hendrik will be the lead of the project and I will be around to provide any necessary guidance. Thanks Hendrik!

Documentation | Example | Demo Application | Support | Changelog

gstore-node is a Google Datastore entities modeling library for Node.js inspired by Mongoose and built on top of the @google-cloud/datastore client.
It is not a replacement of @google-cloud/datastore but a layer on top of it to help modeling your entities through Schemas and to help validating the data saved in the Datastore.

Highlight

  • explicit Schema declaration for entities
  • properties type validation
  • properties value validation
  • shortcuts queries
  • pre & post middleware (hooks)
  • custom methods on entity instances
  • Joi schema definition and validation
  • Advanced cache layer
  • Typescript support
  • populate() support to fetch reference entities and do cross Entity Type "joins" when querying one or multiple entities (since v5.0.0)

Please don’t forget to star this repo if you find it useful :)

Installation

npm install gstore-node --save
# or
yarn add gstore-node

Important: gstore-node requires Node version 8+

Getting started

Import gstore-node and @google-cloud/datastore and configure your project.
For the information on how to configure @google-cloud/datastore read the docs here.

const { Gstore } = require('gstore-node');
const { Datastore } = require('@google-cloud/datastore');

const gstore = new Gstore();
const datastore = new Datastore({
    projectId: 'my-google-project-id',
});

// Then connect gstore to the datastore instance
gstore.connect(datastore);

After connecting gstore to the datastore, gstore has 2 aliases set up

  • gstore.ds
    The @google/datastore instance. This means that you can access all the API of the Google library when needed.

  • gstore.transaction. Alias of the same google-cloud/datastore method

Documentation

The complete documentation of gstore-node is in gitbook.
If you find any mistake in the docs or would like to improve it, feel free to open a PR.

Example

Initialize gstore-node in your server file

// server.js

const { Gstore, instances } = require('gstore-node');
const { Datastore } = require('@google-cloud/datastore');

const gstore = new Gstore();
const datastore = new Datastore({
    projectId: 'my-google-project-id',
});

gstore.connect(datastore);

// Save the gstore instance
instances.set('unique-id', gstore);

Create your Model

// user.model.js

const { instances } = require('gstore-node');
const bscrypt = require('bcrypt-nodejs');

// Retrieve the gstore instance
const gstore = instances.get('unique-id');
const { Schema } = gstore;

/**
 * A custom validation function for an embedded entity
 */
const validateAccessList = (value, validator) => {
    if (!Array.isArray(value)) {
        return false;
    }

    return value.some((item) => {
        const isValidIp = !validator.isEmpty(item.ip) && validator.isIP(item.ip, 4);
        const isValidHostname = !validator.isEmpty(item.hostname);

        return isValidHostname && isValidIp;
    });
}

/**
 * Create the schema for the User Model
*/
const userSchema = new Schema({
    firstname: { type: String, required: true },
    lastname: { type: String, optional: true  },
    email: { type: String, validate: 'isEmail', required: true },
    password: { type: String, read: false, required: true },
    createdOn: { type: String, default: gstore.defaultValues.NOW, write: false, read: false },
    address: { type: Schema.Types.Key, ref: 'Address' }, // Entity reference
    dateOfBirth: { type: Date },
    bio: { type: String, excludeFromIndexes: true },
    website: { validate: 'isURL', optional: true },
    ip: {
        validate: {
            rule: 'isIP',
            args: [4],
        }
    },
    accessList: {
        validate: {
            rule: validateAccessList,
        }
    },
});

// Or with **Joi** schema definition
// You need to have joi as a dependency of your project ("npm install joi --save")
const userSchema = new Schema({
    firstname: { joi: Joi.string().required() },
    email: { joi: Joi.string().email() },
    password: { joi: Joi.string() },
    ...
}, {
    joi: {
        extra: {
            // validates that when "email" is present, "password" must be too
            when: ['email', 'password'],
        },
    }
});

/**
 * List entities query shortcut
 */
const listSettings = {
    limit: 15,
    order: { property: 'lastname' }
};
userSchema.queries('list', listSettings);

/**
 * Pre "save" middleware
 * Each time the entity is saved or updated, if there is a password passed, it will be hashed
*/
function hashPassword() {
    // scope *this* is the entity instance
    const _this = this;
    const password = this.password;

    if (!password) {
        return Promise.resolve();
    }

    return new Promise((resolve, reject) => {
        bcrypt.genSalt(5, function onSalt(err, salt) {
            if (err) {
                return reject(err);
            };

            bcrypt.hash(password, salt, null, function onHash(err, hash) {
                if (err) {
                    // reject will *not* save the entity
                    return reject(err);
                };

                _this.password = hash;

                // resolve to go to next middleware or save method
                return resolve();
            });
        });
    });
}

// add the "pre" middleware to the save method
userSchema.pre('save', hashPassword);

/**
 * Export the User Model
 * It will generate "User" entity kind in the Datastore
*/
module.exports = gstore.model('User', userSchema);

Use it in your Controller

// user.constroller.js

const User = require('./user.model');

const getUsers = (req ,res) => {
    const pageCursor = req.query.cursor;

    // List users with the Query settings defined on Schema
    User.list({ start: pageCursor })
        .then((entities) => {
            res.json(entities);
        })
        .catch(err => res.status(400).json(err));
};

const getUser = (req, res) => {
    const userId = +req.params.id;
    User.get(userId)
        .populate('address') // Retrieve the reference entity
        .then((entity) => {
            res.json(entity.plain());
        })
        .catch(err => res.status(400).json(err));
};

const createUser = (req, res) => {
    const entityData = User.sanitize(req.body);
    const user = new User(entityData);

    user.save()
        .then((entity) => {
            res.json(entity.plain());
        })
        .catch((err) => {
            // If there are any validation error on the schema
            // they will be in this error object
            res.status(400).json(err);
        })
};

const updateUser = (req, res) => {
    const userId = +req.params.id;
    const entityData = User.sanitize(req.body); // { email: 'john@snow.com' }

    /**
     * This will fetch the entity, merge the data and save it back to the Datastore
    */
    User.update(userId, entityData)
        .then((entity) => {
            res.json(entity.plain());
        })
        .catch((err) => {
            // If there are any validation error on the schema
            // they will be in this error object
            res.status(400).json(err);
        });
};

const deleteUser = (req, res) => {
    const userId = +req.params.id;
    User.delete(userId)
        .then((response) => {
            res.json(response);
        })
        .catch(err => res.status(400).json(err));
};

module.exports = {
    getUsers,
    getUser,
    createUser,
    updateUser,
    deleteUser
};

Demo application

If you want to see an example on how to use gstore-node in your Node.js app, check the demo blog application repository.

Development

  1. Install npm dependencies
yarn install
  1. Run the unit tests
yarn test:unit
  1. Run the integration tests

Prerequise:
In order to run the integration tests you need to have the Google Datastore Emulator installed as well as Redis.

  • Launch the Redis server
# From the folder where you've installed the Redis SDK run: 
cd src && ./redis-server
  • Launch the Datastore Emulator (separate terminal window)
# From the root of the project
yarn local-datastore
  • Execute the integration tests (separate terminal window)
# From the root of the project
yarn test:integration

Meta

Sébastien Loix – @sebloix
Hendrik Schalekamp - @carnun

Distributed under the MIT license. See LICENSE for more information.

Contributing

  1. Fork it (https://github.com/sebelga/gstore-node/fork)
  2. Create your feature branch (git checkout -b feature/fooBar)
  3. Generate a conventional commit message (npm run commit)
  4. Push to the branch (git push origin feature/fooBar)
  5. Rebase your feature branch and squash (git rebase -i master)
  6. Create a new Pull Request

Credits

I have been heavily inspired by Mongoose to write gstore. Credits to them for the Schema, Model and Entity definitions, as well as 'hooks', custom methods and other similarities found here. Not much could neither have been done without the great work of the guys at googleapis.

changelog

Changelog

All notable changes to this project will be documented in this file. See standard-version for commit guidelines.

9.1.0 (2024-03-04)

9.0.1 (2024-02-21)

Bug Fixes

  • query.ts: allow 'key' in selected property list (11cc921)

9.0.0 (2024-02-21)

⚠ BREAKING CHANGES

  • YES
  • Entity keys return ids as Number

Features

Bug Fixes

  • src/serializers/datastore.ts: load cached entities with correct Datastore Key type (#265) (b06641b), closes #243

8.0.0 (2022-04-07)

⚠ BREAKING CHANGES

  • datastore.ts: Entity keys return ids as Number

⚠ KNOWN ISSUES

  • Caching - when not using in memory caching (e.g. Redis cache) Date objects data is not being correctly deserialized to Date objects

    • Workaround - cast known Date data to a Date object using the new Date() JavaScript constructor

Bug Fixes

  • datastore.ts: ensure entities with id based keys are correcly loaded from cache (a3a5b33), closes #243

7.2.8 (2022-04-07)

This release fixes incorrectly published code from v7.2.7

Bug Fixes

  • src/serializers/datastore.ts: load cached entities with correct Datastore Key type (#265) (b06641b), closes #243

7.2.7 (2022-04-07)

Bug Fixes

  • fix(src/serializers/datastore.ts): load cached entities with correct Datastore Key type (#265) b06641b
  • test(integration tests): add integration tests for fineOne and check entityKey construction (#246) b87a275

Dependency upgrades

  • chore(deps): bump validator from 13.0.0 to 13.7.0 (#257) d35f66a
  • chore(deps): bump tmpl from 1.0.4 to 1.0.5 (#256) fef76a2
  • chore(deps): bump path-parse from 1.0.6 to 1.0.7 (#255) 707943e
  • chore(deps): bump glob-parent from 5.0.0 to 5.1.2 (#253) 8782798
  • chore(deps): bump hosted-git-info from 2.7.1 to 2.8.9 (#251) 4012b03
  • chore(deps): bump handlebars from 4.7.6 to 4.7.7 (#250) b75752e
  • chore(deps): bump redis from 3.0.2 to 3.1.2 (#249) 3cb6c45
  • chore(deps): bump y18n from 4.0.0 to 4.0.1 (#248) 766310d
  • chore(deps): bump elliptic from 6.5.3 to 6.5.4 (#247) 5d083f7

Documentation

  • docs(github issue templates): add github issue templates (#245) 605a848

https://github.com/sebelga/gstore-node/compare/v7.2.6...v7.2.7

7.2.6 (2020-12-15)

7.2.5 (2020-05-11)

Chore

  • Dependencies Update all dependencies.
    gstore-node has been tested and works with @google-cloud/datastore version 5+.

Bug Fixes

  • typings: Fix passing of generic parameters to Entity (#221) (dc3fba2)

7.2.4 (2019-12-10)

Bug Fixes

  • gstore.save(): update "modifiedOn" property on entities (#209) (94b74d8), closes #202
  • schema.date: allow valid string Date in validation (#210) (268c22e), closes #206

7.2.3 (2019-10-26)

Bug Fixes

7.2.2 (2019-10-01)

Bug Fixes

  • typings: add Methods generic to Model<T, M> (7c3e5b0)
  • Typings: Add generic types to Entity, Model & Schema (#195) (c939de5), closes #194

7.2.1 (2019-09-26)

7.2.0 (2019-09-26)

Features

  • Typescript: Improve typings support (43d9dc2)

7.1.0 (2019-09-13)

Features

  • auto-unindex: add Schema option to auto unindex large properties (#183) (dbf9861)

7.0.0 (2019-09-11)

⚠ BREAKING CHANGES

  • entitykey: The "keyType" Schema option has been removed as it is no longer needed. Also, as gstore does not parse the id anymore, running your project against the Datastore emulator locally might break as the emulator treats differently User.get(123) than User.get("123"). Auto-allocated ids are integers and need to be provided as integers for the Emulator.
  • Dependencies: Node version 8.x not supported anymore. Upgrade to v10.x or superior.
  • Dependencies: The @google-cloud/datastore package is not defined anymore as a dependency of gstore-node. You will need to manually install it in your project.

  • Dependencies: Set google-cloud datastore as peerDependency (#177) (ac52ffb)

  • Dependencies: Update lib dependencies (#178) (7fa94b1)

Bug Fixes

  • call execPostHooks on internalTransaction (#161) (7b132cf)
  • entity: add "id" property to entity after it has been saved (#180) (15a713a), closes #172
  • entitykey: remove convertion of string number to integer for entity key id (#179) (75dc869), closes #168
  • excludefromindexes: update logic to add all properties of Array embedded entities (#182) (c9da35b), closes #132
  • model: throw NOT_FOUND error when trying to update a Model that does not exist (#181) (cc11e02), closes #164
  • Types: Schema methods() signature (#171) (4a144ce)

6.0.2 (2019-04-26)

Bug Fixes

  • added default options when Joi is enabled (528da24)

6.0.1 (2019-03-26)

Bug Fixes

  • Typescript: Add missing export types (f91fc39)

6.0.0 (2019-03-07)

Bug Fixes

  • Model.get(): Consistently return an Array when providing an array of ids (#155) (45e68fc), closes #134

BREAKING CHANGES

  • Model.get(): When an Array of ids is provided to Model.get(), gstore will now consistently return an Array. In earlier versions, if an array of one id was provided, gstore would return a single entity instead of an array containing the entity.

5.0.2 (2019-03-07)

Bug Fixes

  • Entity: Allow saving an entity that has been populated (a24c75a)
  • model.get(): handle null when entity does not exist (#152) (21d258f)
  • Model.update(): fix onUpdateError throwing empty object instead of error (#153) (b1929c7)

5.0.1 (2019-02-05)

5.0.0 (2019-02-04)

Bug Fixes

  • Model.update(): Fix bug in Model.update() inside a transaction (#148) (e4cfaa6), closes #144

Code Refactoring

  • Change gstore instantiation to be consistent with es modules (#149) (3f27d4c)

Features

  • Populate: Fetch entities references in Model.get() and queries (72fff67)

BREAKING CHANGES

  • The new way to create gstore instances is with "new Gstore(<config>)". Refer to the documentation.
  • Populate: Callback (hell) are not supported anymore as the last argument of gstore methods. Only Promises are returned.
  • Populate: Node runtime must be version 8 or superior
  • Populate: The old Schema property types "datetime" and "int" have been removed. Date and Number types should be used instead.

4.3.3 (2018-12-29)

Bug Fixes

  • Sanitize: Remove non writable property on Joi schema (#140) (4ba1ce6), closes #139

4.3.2 (2018-12-21)

Bug Fixes

  • Fixes 'exludeFromRead' for nested paths where object does not exist (f7c336c), closes #128
  • excludeFromRead: Make sure segment exist before trying to access it when deserializing (03bcf53)
  • Model: Update validate() to not sanitize prop where write is set to false (#138) (e86a875)

4.3.1 (2018-11-13)

Bug Fixes

  • Model: Modify validate() method to update entityData on validation (98dab4b)
  • Model: Preserve gstore KEY on entityData when validating Joi Schema (f86dbcb)

4.3.0 (2018-09-07)

Bug Fixes

  • entity: Remove Array wrapping of datastoreEntity() response from cache (00254d0)

Features

  • Add global gstore config to return null on entity not found (6b73631), closes #123
  • Support "read"-type config for embedded objects (e3e554f), closes #122

4.2.6 (2018-08-26)

Bug Fixes

  • typescript definitions file (e99125e)
  • typescript definitions file (399087c)
  • cache: Throw error when Model.get() returns undefined from cache (b46758a), closes #119

4.2.5 (2018-08-18)

Bug Fixes

  • cache: Support nsql-cache 1.1.3 (31d9767)
  • tests: Fix integration tests with Redis cache (0a0838d)

4.2.4 (2018-07-30)

Bug Fixes

  • transaction-post-hooks: Add missing scope to post hooks inside a transaction (3fe059d), closes #115

4.2.3 (2018-07-22)

Bug Fixes

  • delete-hooks: Wrong argument mapping in getScopeForDeleteHooks() (5c91046)

4.2.2 (2018-07-10)

Bug Fixes

  • dataloader: Add maxBatchSize option to limit entities to 1000 (a7c43e9), closes #114

4.2.1 (2018-05-08)

4.2.0 (2018-05-08)

Features

  • global-save: Add option to validate entity before saving in gstore.save() method (39ccb9c), closes #103
  • gstore-save: Add save method to global save (9908d7c), closes #105
  • virtual properties: Access and update virtuals properties directly on the entity instance (b079f7e), closes #102

4.1.1 (2018-04-11)

Bug Fixes

  • typings: Add missing generics to Schema in Model creation (f3cc4b4)

4.1.0 (2018-04-11)

Bug Fixes

  • queries: Allow namespace to be set in "list" queries options (ea5326e)
  • queries: Forward options object to Datastore Query (2eb0f3f)

Features

  • hooks: Model.delete() "post" hooks callback have now their scope on the entity instance delete (4d9b4dd)
  • Schema: Set new types for Schema definition (ad51508)
  • Typescript: Add Typescript support (351538b)

4.0.0 (2018-03-19)

Code Refactoring

  • error handling: Set error code when entity not found in entity.datastoreEntity() method (03cfd7b)

Features

  • cache: Add cache layer to entity.datastoreEntity() method (63780e4)

BREAKING CHANGES

  • error handling: The error code when the entity is not found has been changed from "404" to the "gstore.errors.code.ERR_ENTITY_NOT_FOUND" code