Apify Global Store

THIS PACKAGE IS DEPRECATED! It was build for old versions of Crawlee and the Apify SDK.

What's new in 1.1.3?

• Addition of addMethod and useMethod (experimental)
• pushPathToDataset method name shortened to pushPath
• Support for array formatted paths in setPath, deletePath, and pushPath
• Bug fixes for pushPath, summon, and summonAll
• Improve debug logs
• Improve JSDoc

Installation

npm install apify-global-store

About

This is a simple and lightweight API which allows you to easily create and manage multiple global stores local to an actor's run. By using apify-global-store, you completely remove the need to pass large and complex data through requests. Simplify the process of storing incomplete data that isn't yet ready to be pushed to the dataset with GlobalStore!

State persistence, actor migrations, and race conditions are automatically handled under the hood, so the global state will not be lost on migrations or graceful aborts.

Table of Contents:

• Importing
• Example
• General Usage
  • await GlobalStore.init()
  • store.state
  • store.info
• Mutating State
  • store.set()
  • store.setPath()
  • store.deletePath()
  • store.addReducer()
  • store.setWithReducer()
• Data Management
  • await store.pushPath()
  • store.dump()
  • await store.backup()
  • await store.forceSave()
• Static Methods
  • GlobalStore.summon()
  • GlobalStore.summonAll()
  • GlobalStore.dumpAll()
• Experimental Methods
  • GlobalStore.addMethod()
  • GlobalStore.useMethod()
• Best Practices
  • Best practices with store management
  • Best practices when using a reducer
• Storage
• Available Types
• License

Importing

ES6+

import { GlobalStore } from 'apify-global-store'

ES5-

const { GlobalStore } = require('apify-global-store')

Example

const Apify = require('apify');
const { GlobalStore } = require('apify-global-store');

Apify.main(async () => {
    const store = await GlobalStore.init();

    console.log(store.state); // -> {}

    store.set((prev) => {
        return {
            ...prev,
            message: 'hello'
        }
    })

    console.log(store.state); // -> { message: 'hello' }

    store.set((prev) => {
        return {
            ...prev,
            message2: 'world'
        }
    })

    console.log(store.state); // -> { message: 'hello', message2: 'world' }

    await store.pushPath('message')
});

General Usage

await GlobalStore.init()

(initializeOptions?: InitializeOptions => Promise<GlobalStore>

Argument	Type	Required	Description
initializeOptions	InitializeOptions	false	Configure the store's name, its initial state, and where it is located.

interface InitializeOptions {
    name?: string; // The name of the store to use
    initialState?: Record<string, unknown>; // The initial state to start with
    cloud?: boolean; // Whether or not to keep the store on the cloud, or locally to the actor's run
    debug?: boolean; // Whether or not to display helpful debug messages
}

GlobalStore doesn't have a public constructor. Its init() method runs necessary asynchronous tasks prior to calling on its private constructor.

Usage:

const store = await GlobalStore.init({
    name: 'clothing-store',
    initialState: {
        websites: [],
        errors: [],
    },
    cloud: true,
    debug: true
});

If the store already has a previous state stored within the Key-Value store (in the situation that a migration/abort occurred), then the initialState option will have no effect.

You can only pass letters and "-" within the string, otherwise an error will be thrown (to follow naming conventions). The provided store name will always be forced into uppercase.

Note: A name for a store can only be used once. That includes the default name of GLOBAL-STORE. If the cloud option is set to true, then the global store will be stored in the cloud on your Apify account under "Storages" within a named Key-Value store called CLOUD-GLOBAL-STORES.

store.state

Usage:

const { myValue } = store.state;

A getter method that returns the current state object of the store.

store.info

const { sizeInBytes, lastModified } = store.info;

An object containing information about the contents of the store. Example:

{
    sizeInBytes: 17,
    lastModified: '2022-03-18T15:39:45.041Z',
    globalStoreVersion: '1.0.9',
    type: 'LOCAL'
}

Mutating State

store.set()

(setStateParam: SetStateFunctionCallBack) => void

Argument	Type	Required	Description
setStateParam	SetStateFunctionCallBack	true	A callback function in which the current state is passed and can be modified. Return value must be an object which will be set as the new state.

A synchronous function which sets the current state. The previous state is passed into the callback function as a parameter. It is recommended to refer to the previous state through this parameter instead of using store.state within the callback.

Usage:

store.set((prev) => {
    return {
        ...prev, // spread out the previous state
        newValue: 'hello world'
    }
});

Note: It is always best practice to spread out the previous state at the top level of the returned object, then to add/override any values below.

store.setPath()

(path: string | string[], value: unknown) => void

Argument	Type	Required	Description
path	string	true	A string version of the path within the state you'd like to set/replace.
value	unknown	true	The value you'd like to add for the specified path.

Usage:

store.setPath(`products.${productId}.reviews`, reviewsData);

If the key you are using has dots in it (eg. "https://google.com"), you should use bracket notation instead of dot notation.

const link = 'https://google.com';
const data = { hello: 'world' }

// Will throw an error saying the path "websites -> https://google -> com" doesn't exist, which is true.
store.setPath(`websites.${link}`, data);

// Using bracket notation instead; therefore, the ".com" is not evaluated as dot notation.
store.setPath(`websites[${link}]`, data);

// Array format can also be used
store.setPath(['websites', link], data);

store.deletePath()

(path: string | string[]) => void

Argument	Type	Required	Description
path	string	true	A string version of the path within the state you'd like to delete.

Usage:

store.setPath(`products.${productId}.reviews`);

Note: This method works similar to store.pushPath(), except it does not push the data to the dataset prior to deleting it from the state.

store.addReducer()

(reducerFn: ReducerFunction) => void

Argument	Type	Required	Description
reducerFn	ReducerFunction	true	A custom reducer function which takes (state, action) as its arguments.

When you find yourself writing the same code over and over again using the store.set() method, or you are dealing with modifying deeply nested values, it is best practice to migrate to using a reducer function.

Usage:

// the "action" parameter is expected to have a "type" property
const reducer = (state, action) => {
    switch (action.type) {
        default:
            return state;
        case 'ADD-PRODUCT-REVIEW':
            return {
                ...state,
                products: {
                    ...state.products,
                    [action.productId]: { ...state[action.productId], reviews: [...state[action.productId.reviews], ...action.payload] },
                },
            };
    }
};


store.addReducer(reducer);

Note: Only one reducer can be added to a store instance. Trying to add a second one will result in an error.

store.setWithReducer()

(action: ReducerAction) => void

Argument	Type	Required	Description
action	ReducerAction	true	Use your reducer function to modify the state.

Usage:

const action = { type: 'ADD-PRODUCT-REVIEW', productId: 1234, payload: { title: 'review 1', author: 'super cool product!', stars: 4 } };

store.setWithReducer(action);

Data Management

await store.pushPath()

(path: string, dataset?: Dataset => Promise<void>

Argument	Type	Required	Description
path	string	true	A string version of the path within the state you'd like to push to the dataset.
dataset	Dataset	false	The dataset to push to. If not provided a dataset, the default one will be used.

Push some data from the store into the specified dataset (or into the default one), then automatically delete it from the store after it's been pushed.

Usage:

await store.pushPath(`products.${productId}.reviews.${reviewId}`);

Note: When using this method to push to a dataset, the path is deleted within the state. If you don't want the data to be deleted from the global store after being pushed to the dataset, use regular Apify.pushData() instead.

store.dump()

() => void

Completely empty the entire contents of the store.

await store.backup()

() => Promise<void>

Back the store up to the cloud. The backup will be stored in the "Storages" section of your Apify account under a Key-Value store named CLOUD-GLOBAL-STORES. If you'd prefer to exclusively keep the store on the cloud, set the cloud option to true in InitializeOptions.

Note: If you're using cloud: true in InitializeOptions, you don't need to use this method. Just use store.forceSave() instead

await store.forceSave()

() => Promise<void>

The store's data is saved to the Key-Value Store every single time the "persistState" event is fired. It can also be forced to be saved instantly with this method.

Static Methods

GlobalStore.summon()

(storeName: string) => GlobalStore

Argument	Type	Required	Description
storeName	string	false	The name of the store to summon.

A static method which returns the instance of the store attached to the name provided. A useful and simple aternative to passing a store instance around a parameter.

You are not required to provide the uppercase version of the store name in the storeName parameter. It is automatically done for you.

If you do not provide a store name, then the default store GLOBAL-STORE will be summoned.

Usage:

const store = await GlobalStore.init();

const summoned = GlobalStore.summon('GLOBAL-STORE');

console.log(summoned.state); // => {}

GlobalStore.summonAll()

() => StoreInstances

Similar to summon, but returns the entire storeInstances object, which is a map of all created instances of GlobalStore.

GlobalStore.dumpAll()

() => void

Dump all instances of GlobalStore at once.

Experimental Methods

GlobalStore.addMethod()

(addMethodOptions: AddMethodOptions) => void

Argument	Type	Required	Description
addMethodOptions	AddMethodOptions	true	An object containing the name and method to add. Within this method, the return value of GlobalStore.summonAll() is already passed in as the first parameter.

Add a custom method to the entire GlobalStore class, which is accessible through GlobalStore.useMethod(). This method can be used to modify any store instance.

This is valuable for convenience sake, or when you are making complex modifications to multiple stores.

Usage:

Here is an example of using addMethod to add a complex method to GlobalStore, which modifies the state of three different instances.

// Import GlobalStore and the CustomMethod type
import { GlobalStore, CustomMethod } from 'apify-global-store';

import { methodNames, storeNames } from './consts';

// Destructure our constants
const { ADD_PRODUCT_AND_REVIEWS } = methodNames;
const { PRODUCTS, REVIEWS, RUN_DATA } = storeNames;

// Create a function which ALWAYS takes StoreInstances as the first argument
// Our custom arguments come after the first one
const method: CustomMethod = async (instances, product, website) => {
    // Grab the instances of each store we need
    const productStore = instances[PRODUCTS];
    const reviewsStore = instances[REVIEWS];
    const runDataStore = instances[RUN_DATA]

    // First, add our product to the PRODUCTS GlobalStore instance
    productStore.set((prev) => {
        // Grab the product's data without its reviews
        const { reviews, ...productData } = product;

        return {
            // Spread the previous state
            ...prev,
            [website]: {
                products: {
                    // Spread the previous data for [website].products
                    ...prev[website].products,
                    // Add our new product
                    [productData.id]: productData
                }
            }
        }
    })

    // Add our product's reviews to the REVIEWS GlobalStore instance
    reviewsStore.setPath([website, product.id, 'latestReviews'], product.reviews)

    // If the products for this specific website has reached our threshold,
    // push the data to the dataset and delete it from the REVIEWS store
    if (reviewsStore.state[website].length >= 100) {
        await reviewsStore.pushPath(website);
    };

    // Update our RUN-DATA store
    runDataStore.set((prev) => {
        return {
            ...prev,
            [website]: {
                ...prev[website],
                runs: prev[website].runs + 1
            }
        }
    })
};

GlobalStore.addMethod({ name: ADD_PRODUCT_AND_REVIEWS, method });

Note: It's not recommended to use reducers inside of these custom methods. That can make things complicated very quickly. addMethod can serve as an alternative to addReducer.

GlobalStore.useMethod()

(name: string, ...rest) => void | Promise<void>

Argument	Type	Required	Description
name	string	true	The name of the method used when adding it to GlobalStore.
...rest	any[]	false	Any custom parameters the method needs to run properly

Usage:

import { methodNames } from './consts';

const { ADD_PRODUCT_AND_REVIEWS } = methodNames;

const product = parseProduct($);

await GlobalStore.useMethod(ADD_PRODUCT_AND_REVIEWS, product, new URL(request.url).hostname);

Best Practices

These are not necessarily gospel, but they can help you modularize and scale your project while also making it more readable.

Store Management

When using more than one instance of GlobalStore, it is best to use custom store names, and to put them into a constant:

consts.ts:

export enum storeNames { PRODUCTS = 'PRODUCTS', HOTELS = 'HOTELS' };

main.ts:

import { storeNames } from './consts';

const productStore = await GlobalStore.init({ name: storeNames.PRODUCTS });

const hotelStore = await GlobalStore.init({name: storeNames.HOTELS});

const summoned = GlobalStore.summon(storeNames.HOTELS);

Using a reducer

1. It is always best practice to define your action types as constants, then to import them when defining your reducer.
2. Always define your reducer outside of the store.addReducer() method. Ideally, this should be done in a separate file, then imported.

Example:

consts.ts:

export enum actions {
    GENERAL = 'GENERAL',
    ADD_PRODUCT = 'ADD_PRODUCT',
}

reducer.ts:

import { ReducerFunction } from 'apify-global-store';
import { actions } from './consts';

const reducer: ReducerFunction = (state, action) => {
    switch (action.type) {
        default:
            return state;
        case actions.GENERAL:
            return {
                ...state,
                ...action.payload,
            };
        case actions.ADD_PRODUCT:
            return {
                ...state,
                products: { ...state.products, ...action.payload }
            }
    }
};

export default reducer;

main.ts:

import { GlobalStore } from 'apify-global-store';
import { actions } from './consts'
import reducer from './reducer'

const store = await GlobalStore.init();
store.addReducer(reducer);

store.setWithReducer({
        type: actions.GENERAL,
        payload: { hello: 'world' },
    });

Storage

In the Key-Value store, each store will be represented by an object looking like this:

{
    "store": {
        "hello": "world"
    },
    "data": {
        "sizeInBytes": 17,
        "lastModified": "2022-03-18T15:39:45.041Z",
        "globalStoreVersion": "1.0.9",
        "type": "LOCAL"
    }
}

If the cloud option was not set to true upon the store's initialization, the store will be local to the actor's run under a key in the default Key-Value store.

If cloud was set to true, the global store will be stored in the cloud on your Apify account, under a named Key-Value store called CLOUD-GLOBAL-STORES.

Please: DO NOT manually modify these objects using Apify.setValue(). GlobalStore uses the actor's default Key-Value store under the hood.

Available Types

• GlobalStore

The GlobalStore class.

• InitializeOptions

The options to initialize an instance of GlobalStore.

interface InitializeOptions {
    name?: string;
    initialState?: Record<string, unknown>;
    cloud?: boolean;
    debug?: boolean;
}

• StoreState

An object representing the state returned from store.state.

type StoreState = Record<string, unknown>;

• SetStateFunctionCallback

The callback function to be passed into store.set()

type SetStateFunctionCallBack = (previous: StoreState) => StoreState;

• ReducerParam

The action object of your reducer function. Must include a type key.

interface ReducerType {
    type: string;
}

type ReducerParam<T> = ReducerType & Record<string, T>;

• ReducerFunction

A function which should be passed into store.addReducer()

type ReducerFunction = <T>(state: StoreState, action: ReducerParam<T>) => StoreState;

• StoreData

An object representing the data which is returned from store.info

interface StoreData {
    sizeInBytes: number;
    lastModified: string;
    globalStoreVersion: string;
    type: 'LOCAL' | 'CLOUD';
}

• StoreInstances

An object which hold all instances of GlobalStore, accessible through GlobalStore.summonAll()

type StoreInstances = Record<DefaultStoreName | string, GlobalStore>;

• AddMethodOptions

export type CustomMethod = (storeInstances: StoreInstances, ...rest: any[]) => unknown | Promise<unknown>;

interface AddMethodOptions {
    name: string;
    method: CustomMethod;
}

Credits

Matt Stephens

• LinkedIn
• Github

Kristýna Lhoťanová

• Github

License

Copyright (c) 2022-present Matt Stephens

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.