How to move data between projects in Firebase

Firebase is an excellent choice as MBaaS (Mobile Backend as a Service). In this tutorial, we will see how to move data between projects in Firebase.

In one of our android apps, we stored lot of images in Firebase storage and then provided user to browse them based on category. The data about image categories was stored as Metadata in Firestore database in Firebase. When we created another app which was using different project, we wanted same data again in new project. So we had to copy same data from one project to another.

There are two options to do this :

1. First one is using “Export Import” functionality provided by Firebase. If you have billing enabled in both projects, you can use this functionality.
   e.g From one project, export data to Google Cloud Storage in a bucket, then from new project, import from that bucket.
   This will bring all data from first project to second one. If you want to bring selected Collections from Firestore, there is an option in export command, but if you use that option, it will not export subcollections inside the collections. For us, it was thennot useful as we had multiple levels of subcollections. So we exported and imported all collections and then deleted few in new project which were not needed.
   
2. Another option is do it programmatically. If you have to alter or filter data in between export and import, then another option is doing it programmatically. We used Python to automate that. Check our next post here to know more.

In this tutorial, we will see the steps for using option 1.

Move data using Firebase Export Import feature

Make sure you have billing enabled in both projects to use this feature and you have enough access rights.

Let’s say you have to move data from PROJECT_1 to PROJECT_2.
So our steps will be like this :
For PROJECT_1,

• Create Storage bucket (BUCKET_1) in Google Cloud Storage in PROJECT_1
• Export firestore data to BUCKET_1 using Cloud shell
• Get service account of PROJECT_2 and give it Storage admin permission in BUCKET_1

From PROJECT_2,

• Import data from BUCKET_1

Let’s see the steps in detail.

Firestore – Create Storage bucket

For our source project (PROJECT_1), we need to create storage bucket. For that, in google cloud console, from Menu, Select Storage > Browser option. or Click here (https://console.cloud.google.com/storage/browser). Make sure you have selected your source project from Project list. Create a new storage bucket (for example, BUCKET_1, you may need to give some unique name) and make sure to create in same location as your Firebase project’s location. You can get your firebase Project’s location from “Project Settings” page.

Firestore – Export firestore data to Bucket.

1. From Google cloud console, start cloud shell. (Make sure again that you are in correct project.)
2. Export using this command :
   >gcloud firestore export gs://<bucket_name> –async

> gcloud firestore export gs://backup-gm-data --async

Output might looks like below : 
metadata:
  '@type': type.googleapis.com/google.firestore.admin.v1.ExportDocumentsMetadata
  operationState: PROCESSING
  outputUriPrefix: gs://backup-gm-data/2020-08-11T03:39:34_99880
  startTime: '2020-08-11T03:39:34.974679Z'
name: projects/<project-name>/databases/(default)/operations/ASAzNzEwMDI0MjMJGnRsdWFmZWQHEjFzLXNhLXNib2otbmltZGEQCigS

Copy the outputUriPrefix from above output.
Since we used async option, you can see that operationState is PROCESSING.
Export may take time. You can check the status of above command by using following:

> gcloud firestore operations list
done: true
metadata:
  '@type': type.googleapis.com/google.firestore.admin.v1.ExportDocumentsMetadata
  endTime: '2020-08-11T03:39:49.370518Z'
  operationState: SUCCESSFUL
  outputUriPrefix: gs://backup-gm-data/2020-08-11T03:39:34_99880
  progressBytes:
    completedWork: '341770'
    estimatedWork: '301940'
  progressDocuments:
    completedWork: '1368'
    estimatedWork: '1368'
  startTime: '2020-08-11T03:39:34.974679Z'
name: projects/<project-name>/databases/(default)/operations/ASAzNzEwMDI0MjMJGnRsdWFmZWQHEjFzLXNhLXNib2otbmltZGEQCigS
response:
  '@type': type.googleapis.com/google.firestore.admin.v1.ExportDocumentsResponse

3. Give permission to the service account of PROJECT_2 for BUCKET_1.
   In Google cloud console, now change project to destination project PROJECT_2. You can access list of all service accounts from Menu > IAM & Admin > Service Accounts option. Get the defaul service account (it should be @appspot.gserviceaccount.com)

Now in PROJECT_1, Go to Storage bucket /Permissions tab, Click add member, add new project’s service account email id and give storage admin role.

Now in in Google Cloud shell, change to PROJECT_2

gcloud config set project <PROJECT_2_ID>

Use below command to import data from bucket.

gcloud firestore import --async

Check operation’s status.

gcloud firestore operations list

Once operation is successful, you can check in PROJECT_2’s firestore database menu, now the data is imported from PROJECT_1.

You can choose to export selective collections also from firestore database like below option. But currently it does’t export/import subcollections inside.

gcloud firestore export gs://[SOURCE_BUCKET] --collection-ids=[COLLECTION_ID_1],[COLLECTION_ID_2] --async

This firebase documentation link here has more details.

In next tutorial, we will see how to move data between firebase projects using Python script.