Zvonimir Stipanović

In this article, you can learn how to use Cloud Firestore import and export features and migrate project data. 

Firestore Data migration between projects - Prototyped 

Keep reading and learn how to use Cloud Firestore import and export features and
migrate project data.

This example demonstrates how to export data from a source project and import
that data into the destination project. In this example, the source project is
called production-firestore and the destination project is 
development-firestore.

First things first -  we will need to export production data from Firestore to
the Cloud Storage Bucket. Once we have that exported, we need to transfer the
exported data to the development storage bucket, after which we will finally be
able to import the data.

Before we start transferring data, there are some requirements.

 1. You’ll need to enable billing for both projects. Only Google Cloud projects
    with billing enabled can use the export and import functionality.
 2. Make sure your account has the necessary Cloud IAM permission for both
    projects.

If you are the project owner for both projects, you have the required
permissions. Otherwise, the following Cloud IAM roles grant the necessary IAM
permissions:

Owner, Cloud Datastore owner, Cloud Datastore Import Export Admin

3. Access gcloud from the Google Cloud Console using Cloud Shell
[https://cloud.google.com/shell/].

Be careful! Export and import operations are charged for document reads and
writes at the rates listed in Cloud Firestore pricing
[https://firebase.google.com/docs/firestore/pricing]. The costs of export and
import operations do not count towards your spending limit. Export or import
operations will not trigger your Google Cloud budget alerts until after
completion.

Reads and writes performed during these steps are applied to your daily quota
after the operation is complete.

Export data from the source project
 1. Create a Cloud Storage bucket
    [https://cloud.google.com/storage/docs/creating-buckets]

Create a bucket in the source project - in our case, we will create a
production-bucket.

Your bucket name should meet bucket name requirements
[https://cloud.google.com/storage/docs/naming-buckets].

2. Export data using gcloud

We will use the gcloud firestore export command to export data from your source
project.

Make sure that you are in the correct Firestore location (production-firestore

There are 2 options:

1. Export all data

Replace [SOURCE_BUCKET] with the name of your Cloud Storage bucket

> gcloud firestore export gs://[SOURCE_BUCKET] --async
In our case it will be:

> gcloud firestore export gs://production-bucket --async
2. Export specific collections

> gcloud firestore export gs://[SOOURCE_BUCKET]
--collection-ids=[COLLECTION_ID_1],[COLLECTION_ID_2] --async
For example, in our case, we want to export users and reports collection

> gcloud firestore export gs://production-bucket --collection-ids=users,reports
--async
Take a note of your export operation’s outputUriPrefix as we will use this later
on. By default, Cloud Firestore adds a prefix to our export files based on a
timestamp.

> outputUriPrefix: gs://production-bucket/2022-01-18T10:58:23_56418
Transfer data from one bucket to another
 1. Create a Cloud Storage bucket
    [https://cloud.google.com/storage/docs/creating-buckets]

Just like we did for the source/production environment, we will need to create a
bucket in the destination project - in our case, we will create a 
development-bucket.

Important: Your bucket has to be in the same location as your Cloud Firestore
(destination Firestore Database).

2. Open the Transfer page
[https://console.cloud.google.com/transfer/cloud?_ga=2.148493681.779314474.1642429950-1865278465.1629470715] 
in the Google Cloud Console

First of all, we will need to create a transfer job.

Choose your source bucket as a source and destination bucket as a destination.

After selecting the correct source and destination buckets, update the other
settings as you’d like (you can choose to overwrite or delete data). Finally,
schedule the transfer and the data will start migrating between environments.
When all data has been transferred, we are ready to import data from the
destination bucket to the destination Firestore Database.

Import data into the destination project
Before we can start an import operation, we need to make sure that our
destination project can access our Cloud Firestore data files.

 1. Make sure that gcloud is configured for the correct project

You can configure gcloud to the correct project with this command:

> gcloud config set project [DESTINATION_PROJECT_ID]
In our case, we will set our project to development-firestore:

> gcloud config set project development-firestore
2. Use gcloud firestore import

We will use the gcloud firestore import command to import the data into our
destination bucket.

> gcloud firestore import gs://[SOURCE_BUCKET]/[EXPORT_PREFIX] --async
Our source bucket will be development-bucket and the export prefix will be our
export operation’s ouputUriPrefix timestamp (see the last step in exporting
data).

In our case, it should look like this:

> gcloud firestore import gs://development-bucket/2022-01-18T10:58:23_56418
--async
While the export operation is running, you can check your operation’s progress
with this command:

> gcloud firestore operations list
And that's it! 
We have successfully moved data from the source project to the destination
project! For more detailed information, you can check the official Firestore
documentation [https://firebase.google.com/docs/firestore/manage-data/move-data]
.


--------------------------------------------------------------------------------

> Thank you for your the time to read this blog! Feel free to share your thoughts
about this topic in the comments or drop us an email at hello@prototyp.digital.

AuthorZvonimir Stipanović

Start a Project