Fetch Firestore collection data with the Firebase NodeJS SDK

You have to love the power of Firestore when it comes to storing complex data structures. But with great power, comes great responsibility. It may be easy to store this data, but can sometimes be a tedious task to get the data out to be consumed by a client application.

In this blog post, we are going to cover how you can fetch and filter collection data and also how to flatten the structure when dealing with reference fields. There are several reasons why you would want to get the data in this format, one of them being reporting and data visualization.

This code is adapted from the official Firestore integration in Chartbrew.

Data and tools used in this example

In this blog post, we are going to use a small dataset as an example, but the code we are going to write here has been used by companies with tens of thousands of documents.

To connect to Firestore, we are going to use the Firebase Admin Node.js SDK which you can find here: https://github.com/firebase/firebase-admin-node

Authenticate with Firestore from Node

To get started, install the Firebase SDK in your project:

npm install --save firebase-admin

Now let's create a Javascript class to handle all the interactions with Firestore, including the authentication. We will add the authentication in the constructor and then use methods to fetch data.

const firebase = require("firebase-admin");

const serviceAccount = require("./path/to/serviceAccountKey.json");

class FirestoreConnection {
  constructor(uuid = "test") {
    let admin;
    const firebaseAppName = `firestore-${uuid}`;

    // first check if there is a firebase app already created for this uuid
    firebase.apps.forEach((firebaseApp) => {
      if (firebaseApp.name === firebaseAppName) {
        admin = firebaseApp;
      }
    });

    if (!admin) {
      admin = firebase.initializeApp({
        credential: firebase.credential.cert(serviceAccount),
      }, firebaseAppName);
    }

    this.db = admin.firestore();
  }
}

module.exports = FirestoreConnection;

To authenticate, we need a service account key that can be generated from your firebase dashboard. Here is how you can get yours:

Save the key into a JSON file and add the correct path in the Javascript code above. When authenticating with Firebase, the SDK will create a new app so if you want to keep the connections separate, you can pass a unique string to the constructor. The code will make sure that no connection will be made for an existing app because this results in an error.

The reference to the Firestore database is now stored in this.db and we can use this to fetch data.

Listing the existing collections

To get an overview of your collections, you can do so with a simple call to the listCollections() method. To extend the functionality of your class, you can add this new method:

listCollections() {
	return this.db.listCollections();
}

The use this with the class, you can do the following:

const FirestoreConnection = require("./path/to/FirestoreConnection");

module.exports = async () => {
	const firestoreConnection = new FirestoreConnection();
	const collections = await firestoreConnection.listCollections(); 

	console.log("collections", collections);
};

Getting collection data

To get collection data from Firestore, we need to know the name of the collection, which can be found using the list method, or directly from the Firebase dashboard. Extend the class with the following method to get collection data:

async getCollectionData(collection) {
    const docsRef = await this.db.collection(collection);
    const mainDocs = [];

    const docs = await docsRef.get();
    docs.forEach(async (doc) => {
      mainDocs.push({ ...doc.data(), _id: doc.id });
    });

    return mainDocs;
}

If we want data from the connections collection, the method call would look like this:

const data = await firestoreConnection.getCollectionData("connections");

console.log("data", data);

Filtering collection data

Filtering is a powerful feature available in the Firestore SDK. There are quite a few operations available so we can write a helper function to filter the data before sending it to the client.

function filter(docsRef, condition) {
  let newRef = docsRef;
  switch (condition.operator) {
    case "==":
      newRef = docsRef.where(condition.field, "==", condition.value);
      break;
    case "!=":
      newRef = docsRef.where(condition.field, "!=", condition.value);
      break;
    case "isNull":
      newRef = docsRef.where(condition.field, "==", null);
      break;
    case "isNotNull":
      newRef = docsRef.where(condition.field, "!=", null);
      break;
    case ">":
      newRef = docsRef.where(condition.field, ">", condition.value);
      break;
    case ">=":
      newRef = docsRef.where(condition.field, ">=", condition.value);
      break;
    case "<":
      newRef = docsRef.where(condition.field, "<", condition.value);
      break;
    case "<=":
      newRef = docsRef.where(condition.field, "<=", condition.value);
      break;
    case "array-contains":
      newRef = docsRef.where(condition.field, "array-contains", condition.value);
      break;
    case "array-contains-any":
      newRef = docsRef.where(condition.field, "array-contains-any", condition.value);
      break;
    case "in":
      newRef = docsRef.where(condition.field, "in", condition.value);
      break;
    case "not-in":
      newRef = docsRef.where(condition.field, "not-in", condition.value);
      break;
    default:
      break;
  }

  return newRef;
}

This function takes 3 parameters:

• docsRef - This is an array containing the references to all the documents in the collection
• condition - an object containing the field to apply the filter on, the operator, and values to filter based on

To use this filter function together with the data fetching method, we can modify the code as follows:

async getCollectionData(collection, conditions) {
    let docsRef = await this.db.collection(collection);
    const mainDocs = [];

    // filter the docs
    if (conditions) {
      conditions.forEach((condition) => {
        docsRef = this.filter(docsRef, condition.field, condition);
      });
    }

    const docs = await docsRef.get();
    docs.forEach(async (doc) => {
      mainDocs.push({ ...doc.data(), _id: doc.id });
    });

    return mainDocs;
  }

If the conditions are passed to the getCollectionData() method, the documents will be filtered based on the inputs

const data = await firestoreConnection.getCollectionData("connections", {
    field: "type",
    operator: "==",
    value: "api",
});

console.log("data", data);

Note how in this example we passed an extra parameter to the method and it contains the required data to only return the data with field = "api". The filter function covers a wide range of other operators so feel free to try all of them out.

Cleaning reference fields in the collection data

If you have fields of reference type in your data, the SDK will return strangely formatted data. This is not always the desired outcome, especially when we're dealing with reporting and data visualization where we need to flatten data structures. Next up, we are going to write a function that will look for these reference fields within the collection data and attach them to the result.

function determineType(data) {
  if (data !== null && typeof data === "object" && data instanceof Array) {
    return "array";
  }
  if (data !== null && typeof data === "object" && !(data instanceof Array)) {
    return "object";
  }

  return "other";
}

function populateReferences(docs) {
  const nestedCheckedDocs = [];
  let index = -1;
  docs.forEach((data) => {
    const doc = data;

    nestedCheckedDocs.push({ ...doc });
    index++;
    Object.keys(doc).forEach((key) => {
      if (determineType(doc[key]) === "array") {
        doc[key].forEach((subDoc, subIndex) => {
          if (determineType(subDoc) === "object") {
            if (subDoc._firestore) {
              nestedCheckedDocs[index][key][subIndex] = subDoc.id;
            } else {
              Object.keys(subDoc).forEach((subKey) => {
                if (subDoc[subKey] && subDoc[subKey]._firestore) {
                  nestedCheckedDocs[index][key][subIndex][subKey] = subDoc[subKey].id;
                }
              });
            }
          }
        });
      } else if (doc[key] && doc[key]._firestore) {
        nestedCheckedDocs[index][key] = doc[key].id;
      }
    });
  });

  return nestedCheckedDocs;
}

This function is a bit more complex because of the extra checks to identify the reference fields. When Firestore returns a reference, it will be an object with a _firestore key inside. The function above will check if a field is of object type and if it has the _firestore key inside. The function also populates the data if the document has an array of references.

Final code

Feel free to use and modify this code as you wish and hopefully these functions can give you a better idea of how the Node SDK for Firestore works.

const firebase = require("firebase-admin");

const serviceAccount = require("./path/to/serviceAccountKey.json");

function filter(docsRef, condition) {
  let newRef = docsRef;
  switch (condition.operator) {
    case "==":
      newRef = docsRef.where(condition.field, "==", condition.value);
      break;
    case "!=":
      newRef = docsRef.where(condition.field, "!=", condition.value);
      break;
    case "isNull":
      newRef = docsRef.where(condition.field, "==", null);
      break;
    case "isNotNull":
      newRef = docsRef.where(condition.field, "!=", null);
      break;
    case ">":
      newRef = docsRef.where(condition.field, ">", condition.value);
      break;
    case ">=":
      newRef = docsRef.where(condition.field, ">=", condition.value);
      break;
    case "<":
      newRef = docsRef.where(condition.field, "<", condition.value);
      break;
    case "<=":
      newRef = docsRef.where(condition.field, "<=", condition.value);
      break;
    case "array-contains":
      newRef = docsRef.where(condition.field, "array-contains", condition.value);
      break;
    case "array-contains-any":
      newRef = docsRef.where(condition.field, "array-contains-any", condition.value);
      break;
    case "in":
      newRef = docsRef.where(condition.field, "in", condition.value);
      break;
    case "not-in":
      newRef = docsRef.where(condition.field, "not-in", condition.value);
      break;
    default:
      break;
  }

  return newRef;
}

function determineType(data) {
  if (data !== null && typeof data === "object" && data instanceof Array) {
    return "array";
  }
  if (data !== null && typeof data === "object" && !(data instanceof Array)) {
    return "object";
  }

  return "other";
}

function populateReferences(docs) {
  const nestedCheckedDocs = [];
  let index = -1;
  docs.forEach((data) => {
    const doc = data;

    nestedCheckedDocs.push({ ...doc });
    index++;
    Object.keys(doc).forEach((key) => {
      if (determineType(doc[key]) === "array") {
        doc[key].forEach((subDoc, subIndex) => {
          if (determineType(subDoc) === "object") {
            if (subDoc._firestore) {
              nestedCheckedDocs[index][key][subIndex] = subDoc.id;
            } else {
              Object.keys(subDoc).forEach((subKey) => {
                if (subDoc[subKey] && subDoc[subKey]._firestore) {
                  nestedCheckedDocs[index][key][subIndex][subKey] = subDoc[subKey].id;
                }
              });
            }
          }
        });
      } else if (doc[key] && doc[key]._firestore) {
        nestedCheckedDocs[index][key] = doc[key].id;
      }
    });
  });

  return nestedCheckedDocs;
}

class FirestoreConnection {
  constructor(uuid = "test") {
    let admin;
    const firebaseAppName = `firestore-${uuid}`;

    // first check if there is a firebase app already created for this uuid
    firebase.apps.forEach((firebaseApp) => {
      if (firebaseApp.name === firebaseAppName) {
        admin = firebaseApp;
      }
    });

    if (!admin) {
      admin = firebase.initializeApp({
        credential: firebase.credential.cert(serviceAccount),
      }, firebaseAppName);
    }

    this.db = admin.firestore();
  }

  listCollections() {
    return this.db.listCollections();
  }

  async getCollectionData(collection, conditions) {
    let docsRef = await this.db.collection(collection);
    const mainDocs = [];

    // filter the docs
    if (conditions) {
      conditions.forEach((condition) => {
        docsRef = this.filter(docsRef, condition.field, condition);
      });
    }

    const docs = await docsRef.get();
    docs.forEach(async (doc) => {
      mainDocs.push({ ...doc.data(), _id: doc.id });
    });

    const finalDocs = populateReferences(mainDocs);

    return finalDocs;
  }
}

module.exports = FirestoreConnection;

Visualizing Firestore data with Chartbrew

Chartbrew is a data visualization and reporting platform with an official integration for Firestore and Realtime Database. The code in this article is based on the integration code in the platform.

In Chartbrew, you can create a Firestore or Realtime Database connection and then use the UI to query for data. You can then use this data to create visualizations and get to know your data a bit better. The query editor also allows you to query and filter sub-collections alongside your main collections.

If you are looking to visualize data from Firebase, you can sign up for Chartbrew here and try it out free for 30 days. Chartbrew is also an open-source platform and can be self-hosted for free. You can find more details on this over on the official Github repository.

Read more about how you can visualize your Firebase data in this tutorial blog post: