All posts
Blog

How to migrate Parse app to AWS – our story

Krzysztof Zielonka

Parse announced that they were going to retire on January 28, 2017. It is a big problem for all customers using Parse for their apps because they have to migrate all data and set up their own backend. We have already moved one of our apps and we want to share with you our experience.

In this article you can find:

  • general info of Parse, Parse Server and Parse Dashboard,
  • info about database migration and which option has we chosen,
  • how to set up minimal Parse Server locally,
  • info about problem with files migration,
  • why we have chosen Amazon Elastic Beanstalk and how to deploy Parse Server there,
  • how to set up and deploy Parse Dashboard.

parse_amazon_mlab

Parse introduction

Parse is a SaaS (Software as a Service) backend provider for mobile and web applications. While using it you will be able to build your application easily without the necessity of building your own backend. Out of the box you have ready features like:

  • data storage in document-oriented database,
  • files storage,
  • users password and third-party authentication (for example with Facebook),
  • push notifications,
  • Apple In-App Purchases validation,

and much more with REST api and ready iOS, Android, JavaScript and others SDKs (even for Arduino). The usage of such solution can speed up development and reduce the cost of application. All docs and SDKs are available on parse.com/docs.

Parse is going to retire on January 28, 2017 and all hosted services will be disabled. It might sound like a disaster for all existing apps although Parse team has done a great job and hasn’t left their customers on their own. The team has moved all software to open source so you are enabled to set your own instance of Parse Server and use SDKs without any need to build your custom backend. So if you have existing app you should start migrating your data and setting up server. If you want to start a new application with Parse you can also do it with Parse Server. All Parse software you can find on GitHub.

Database migration

Migration process is described very clearly in Parse Server documentation on Migrating an Existing Parse App. The first thing which you have to do is to move your database. It is very essential to do it before April 28, 2016 because it will give you more time to configure and set up server. However the most important is that Parse team will place such customers issues with higher priority. Please check What happens if I don’t migrate my data by April 28, 2016? at Parse documentation section for more info. After data migration, according to the instruction you should set your local instance of Parse Server, configure it and update all your client apps to work with the Parse Server. If everything works well you should deploy it to the production environment.

We were wondering how to update old iOS apps which were already on store and start using a new backend with the database which would be soon migrated. We were afraid, that they would stop working. According to the documentation there is no need to worry about it. Because after having the database migration completed, hosted Parse server will be running until January 28, 2017 and will be using your newly migrated database. The old apps should still be working without any problems. So you can easily migrate only database before you will start working on Parse Server and before updating client apps.

parse_migration

As you can see on picture below on the top is presented how clients like iOS app requests for data. It uses hosted Parse server which gets its from its hosted db. On the bottom we can see how looks data flow when we replace Parse Hosted DB with our own MongoDB. The old clients can still communicate with api.parse.com which gets and stores data in your own database this time. The full visualization is presented on Visual Overview in documentation.

mlab_rocket_object

Parse recommends the usage of mLab or RocketObject as your new database hosting. Both of those providers fully manage your database, so you don’t need to worry of setup. Of course if you want you can use any other solution including creation your own cluster. We have chosen mLab shared server option for our project which is the cheapest solution of $15 per month for 1GB. For our app it is enough and the advantage of mLab is that we have the choice between Amazon S3 Bucket, Microsoft Azure or Google Cloud Platform where our data will be stored. We decided to choose Amazon because our intention was to deploy there our backend as well. You can also use mLab free sandbox plan to set up there your staging version of database. Whatever you will chose for your project, follow the Migrate Parse DB to Self-Hosted MongoDB documentation to move your database.

Local Parse Server set up

The next step is to set up your local Parse server which is core of the system. It is the open source backend built on Node.js Express framework. It implements all of Parse features, but it is still under development and might have some bugs. We found some of them while we were migrating our app so please check your app carefully in local environment to make sure that everything is working correct.

parse-server-logo

You can start your application cloning Parse Server Example which have reach configuration example for almost all deployment options. It is good documented and probably it will be all what you need. However we decided to start our project from scratch and create parse server embedded in express app. To start minimal version of Parse server install parse npm package and express npm pacakge. Create yours server main file like below.

Presented configuration is enough to start server as middleware before SDK and MongoDB. There are many other configuration options which are well documented on Parse guide. To start your server run node app.js in a console. When it starts, you can test it using below curl command which creates new object.

Files migration

Parse Objects can have File fields which allows you to store files like images or pdf documents. Parse keeps files in it’s own AWS S3 Bucket and Mongo database stores only references (URLs) to them. Migrating database as described above copies only database data and files references, but not files itself. You have also to move the files to your storage before January 28, 2017 or they will be deleted. However there is problem because there is no easy solution or tool for moving files for now. The Parse team is looking for one as we can read on migration guide, but now we can only find their plan how to do it (files migration plan).

Parse Server can be configured to work with Parse S3 Bucket so it will still serve files without need to migrate them. You need to set fileKey parameter initializing ParseServer in app.js which is shown above (check parse guide for more info). The key can be found in Parse Dashboard > YOUR APP > App settings > Security & Keys. Remember that setting this key doesn’t solve the problem, but only allows you to focus on other things until the tool for files migration will be introduced.

You should also set up your own storage for any new files which you will want to add. Parse only require to provide fileAdapter property with adapter which exposes methods like: createFile, deleteFile, getFileData and getFileLocation (check files adapter base). There are build in adapters for stores like Amazon S3, Google Cloud Storage or MongoDB which you can use. Check documentation for more info.

Parse Server deployment

We considered between Amazon Elastic Beanstalk, Google App Engine and Heroku, and finally decided to use Amazon Elastic Beanstalk. Each of those solutions are PaaS (Platform as a Service). It means that we don’t need to set any software like load balancers or web server. We only deploy our code and the rest is handled by platform. However each of these differs in details what is presented in table below. Our considerations are tuned for our project which is small application with no special requirements. Each of those three solutions are quite cheap and scale easily. Amazon and Google scale automatically, increasing and decreasing number of instances depending on requests traffic, but Heroku needs to be managed. All of them support Node.js environment, but Google App Engine is in beta version.

Amazon Elastic BeanstalkGoogle App EngineHeroku
Price~$10
for t2.micro
1vCPU 1GiB
free
28 instance/day
$7
per instance
Scalingautoautomanuall
NodeJs environmentOKBETAOK

State on 4 March 2016

We decided to deploy on Amazon Elastic Beanstalk with Node.js environment.

elastic_beanstalk_and_nodejs

We need to create package.json which specifies what npm packages are needed to be installed and how to start server.

Next create Amazon application and deploy our code.

  1. Before start zip together app.js created on “Settings up local parse server” section above and just created pacakge.json.
  2. Go to Amazon Elastic Beanstalk.
  3. In the right top corner click “Create New Application”.
  4. Type application name (for example “parse-server”) and click “Next”.
  5. Next you will see “New Environment” page. Click there “Create web server” button.
  6. You should be on “Environment Type” page. You have to select “Node.js” form “Predefined configuration:” and click “Next”.
  7. On “Application Version” page set “Source:” as “Upload your own” and select zipped files. Click “Next”.
  8. On next “Environment Information” page type your “Environment name” (for example “parse-server-env”) and type “Environment URL:” which will be your server url. Check is URL available and if not type another one. If everything will be valid click “Next”.
  9. On next pages “Additional Resource”, “Configuration Details”, “Environment Tags” and “Permissions” just click next.
  10. You should land on “Review” page on which just click “Launch” and wait until you environment will be ready.
  11. Now you can access your parse dashboard using “Environment URL” which you have typed in point 8.

Parse Dashboard deployment

Parse dashboard is separate project which is also open source and you can find it on GitHub. You can just install it using npm (parse-dashboard package) and launch it from console.

Dashboard can be accessed through browser entering http://localhost:4040. If you need it only for yourself you can launch it in such way every time when you need without deploying. In our case we need it to be online and like the Parse Server we deployed it to Amazon Elastic Beanstalk, but this time as Docker container.

elastic_beanstalk_and_docker

To do this you will have to create three files: Dockerfile, parse-dashboard-config.json and Dockerrun.aws.json. Below you can see Dockerfile which is recipe for container build instructions.

It just imports parseplatform/parse-dashboard container as a base, adds configuration, exposes port 4040 and runs the server with two parameters. The first is path of configuration file and the second allows to access dashboard using http (default https is required). It is not recommended, but our server will be deployed to elastic beanstalk which is accessed by proxy. If we want we can later set Amazon to use domain with https. Next file presented below is dashboard configuration.

The first property “users” is a list of users who can access dashboard. Parse dashboard implements HTTP basic access authentication so it is the reason why https connection is recommended. The next one “apps” is the list of apps which are managed by dashboard. If we have production and staging app we can add both. The last file is Dockerrun.aws.json configuration which just tells AWS Elastic Beanstalk which docker container port is exposed.

All files should be in the same directory. Next we have to create Amazon Elastic Beanstalk environment. To do that you should repeat the same process like for Parse Server deployment described above with same changes. In step 1 zip presented files: Dockerfile, parse-dashboard-config.json and Dockerrurn.aws.json. In point 5 you should choose “Docker” environment instead of “Node.js”. The rest steps are the same. You should also set domain with https to make the dashboard secure with HTTP basic access authentication.

Summary

Migrating the Parse backend is quite easy, but it depends on application complexity. Standarized simple server can be deployed in few hours, but if you have a lot of Cloud Code you might need more time. However due to that we can develop our app more easily. We are able to set up our Parse Server locally what hasn’t been possible before. App performance should be also improved as soon as you have your own dedicated Parse instance deployed. When you set it up, maintenance will not cost much due to the fact that solutions like Amazon Elastic Beanstalk are self-managing. The only problem which has not been solved yet is the migration of files, but we do hope that it will be fixed up soon.

Newsletter

The post is created by Droids On Roids Team Member.
We would love to take care of your business.

Leave comment

  • Jonathan Brown

    Krzysztof, great article! This has been a great help understanding Parse.

    I’m stuck at deploying Parse-Dashboard. I have AWS Docker environment ready and working, I see the dashboard load, but when it asks for username/password, I cannot log in (used default admin/password).
    When click cancel, I see the dashboard, but no app visible.
    Can you provide guidance?

    Thank you!

    • Krzysztof Zielonka

      Thank you, I am glad that I can help you.

      The problem is because of my mistake in parse-dashboard-config.json file.
      The “password” property is incorrect and should be replaced with “pass”.
      I have already updated it in example above so please compare it with your file.
      Very sorry for that.

      • Jonathan Brown

        Krzysztof, Thank you very much for your correction! It now works well!

  • oojr

    Would you use parse-server on a new project?

    • Krzysztof Zielonka

      Yes of course.
      It requires some work to set it up but creating your backend from the scratch requires much more.
      Parse gives you SDK for iOS, Android, Javascript and more, it handles authentication, permissions and a lot of more.
      If you need some special functionality which Parse doesn’t have you can always implement it as Cloud Code and use easily with SDK.
      You can also embed Parse in Express framework and use it as a your app base but more complex operation implement as separate Express endpoints.
      Of course Parse has also it’s limitations.
      If you need some complex queries like in relational data bases Parse is not for you because it uses NoSQL MongoDB database.
      Because of same reason the performance can also be a problem.
      However in my opinion if your focus is on frontend app, you should give Parse a try.
      In the worst case you will lose a day on setting up it but implementing your own backend with same functionalities will take weeks or even months.
      You can also implement your frontend app with additional adapters layer between Parse and your application code so you will be able to easily replace it when you decide to build your custom backend.

      • oojr

        I created my backend “from scratch” using hapi and mongodb but I am quickly seeing that the parse server open source is growing faster than what I can do on my own (LiveQueries, Push Notifications), thanks for your input, I believe in NoSQL database so that is also not a problem. I can use any backend under sun even something with Java but I think I will be able to save time with Parse and time is a major key when creating new software projects

        • Krzysztof Zielonka

          I agree with you. Parse will help you to save a lot of time. Good luck with your project 🙂

  • cs

    Do you have any information on setting up Parse Cloud Code logs (what you used to send with console.log / console.error) in this environment? All the documentation I’m finding is spotty in different areas.

    • Krzysztof Zielonka

      In the case of AWS Elastic Beanstalk NodeJS environment logging is already preconfigured. The concatenated stdout and stderr logs are stored in /var/log/nodejs/nodejs.log on each instance. You can access them through ssh access, eb cli or your environment web console. If you are on web console just choose “Logs” from the left menu and then click drop down button “Request logs”. Whatever you choose there after a few seconds a new “Logs” table row will appear with download button. So whatever log method like console.info, console.error, console.warn from https://nodejs.org/api/console.html you use, it’s result will be in the downloaded file. It aggregates also logs from other services so you have to scroll to find Node.js logs. By default logs entries are stored only for 15 minutes and if you need it to be persistent you have enable it. Instructions how to do it you can find on EB documentation: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html
      with a lot more info about logging.

    • Krzysztof Zielonka

      In the case of AWS Elastic Beanstalk NodeJS environment logging is already preconfigured. The concatenated stdout and stderr logs are stored in /var/log/nodejs/nodejs.log on each instance. You can access them through ssh access, eb cli or your environment web console. If you are on web console just choose “Logs” from the left menu and then click drop down button “Request logs”. Whatever you choose there after a few seconds a new “Logs” table row will appear with download button. So whatever log method like console.info, console.error, console.warn from https://nodejs.org/api/console.html you use, it’s result will be in the downloaded file. It aggregates also logs from other services so you have to scroll to find Node.js logs. By default logs entries are stored only for 15 minutes and if you need it to be persistent you have enable it. Instructions how to do it you can find on EB documentation: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html
      with a lot more info about logging.

      • cs

        Got it. Thanks for the tip.

  • cs

    We’ve actually encountered a bizarre problem with PFFiles – curious as to whether you, or anyone else here has experienced the same? One of our classes saves a PFFile (profile picture) in JPEG format, as one of its keys. All pre-existing files (which are still hosted on Parse’s S3 account and accessed via the FileKey) are fine, but any newly saved file seems to get mutated in transmission, and rather than saving as its usual type, ends up as a ‘.bin’ file which is unreadable (even locally, if you try to unzip it it becomes a cpgz file, then unzipping that turns it back into a bin file, etc.).

    Originally we were just using the GridStore adapter and saving the files in mLab, so we thought that might be the problem, but even using the S3 adapter and hosting the files there ourselves, they still get converted into the unusable type. I also tried enabling directAccess with the adapter, so they wouldn’t get proxied through parse-server, but that doesn’t help either.

    I can find obliquely related issues others have run into, but nothing conclusive or especially helpful out there. Has anyone else experienced this or anything like it?

    Even more fun – if I try converting the image to a PNG and sending it as that format instead, Parse Server actually *closes the connection* when you try to save and you need to kill and restart the app entirely.

    • Krzysztof Zielonka

      Sorry for late answer, we had holiday in Poland last week and after then we had a lot of work.

      We haven’t had problem like your.
      However we had other problem related to files.
      New files uploaded to our own S3 bucket waren’t display on clients with old sdk.
      Theirs urls were invalid but old files which were on Parse Server S3 bucket were ok.
      It was easy to solve, we just had to upgrade our client’s sdk to latest version.
      If you have also your client sdk and parse server in not latest version maybe you should try to upgrade it.

      I checked the code of parse server and the place when your file extension can be changed is here: https://github.com/ParsePlatform/parse-server/blob/2.2.11/src/Controllers/FilesController.js#L22
      It adds extension to file name when it doesn’t have one but valid content type is defined.
      So it would add .bin extension if you send such file which name hasn’t extension and with content type like: application/octet-stream.
      However if you tried to send it directy to S3 avoiding parse server and it still changes it’s extension parse-server should not be the cause of a problem.
      It is still very strange that S3 modifies your file name or anything on uploading it.
      So maybe you should check once again does it for sure evade parse server.

      You mentioned that when you unzip downloaded bin file it isn’t image anymore.
      Maybe try to just change its extension from “.bin” to original file extension like “.jpg” or “.png” instead of unziping it .

      Reffering to your mention about server crash.
      If you are able to download log for this accident I think you should report this issue on parse-server github project page:
      https://github.com/ParsePlatform/parse-server/issues
      It can be more common problem and I am sure that it might be fastly resolved.

      If you can attach logs here or give us more details please don’t hestitate.
      Maybe we will be able to help you more. 🙂

      • cs

        No worries! Thanks for your suggestions. Actually, upon digging into the Parse SDK a bit more, it seems that any file where we don’t explicitly mention a MIME type is similarly corrupted on save. My guess is Parse’s hosted service had some internal checks to assign a type on the server side. So anything saved via ‘PFFile fileWithName:’ will fail, but if you add the ‘contentType:’ parameter, it works as normal. This is only true after moving to Parse server.

  • Raspberry Pi

    How long does this process take for someone who has only used Parse before for apps

    • Krzysztof Zielonka

      It has hard to predict how long exactly it will take.
      It depends on your app complexity and on yours skills.
      I predict that the proces described in the article should take few hours.
      Of course you will have to add mail service and others but all is very good described at official documentation.
      The only thing which is still hard is moving uploaded files from parse server to your S3 bucket or any other storage provider.
      For now parse team haven’t introduced any easy method for this problem so you can wait or write your own script which will migrate them.

      Migrating one of our first projects took me about two weeks but during that time I was mostly learning about Parse Server and I was refactoring our client apps to use the newest version of SDK what wasn’t necessary in fact.
      Now I think that I would be able to migrate similar project in one or two days but assuming that clients code and cloud code would not require a lot of refactorings.
      Migration procedure is getting better and better documented on https://parse.com/migration so it should be easier now.

      Summarizing I guess that moving database and setting parse server with parse dashboard as I described in article should take you no more that one day.
      Even if you are doing it first time.
      But to fully understand it and make it functional with services like mail service, push notifications, updating cloud code if needed and update clients should take you 2 – 3 weeks (included time needed to learn all those things).

  • Hamza Lhioui

    live query is not working, it seem’s that aws is not supporting web socket. have you tried it ?

    • Krzysztof Zielonka

      Sorry I haven’t tried WebSockets on AWS yet. However I checked it and websockets are supported. For sure it requires some additional load balancer configuration and maybe proxy server (nginx) too. As I checked load balancer requires to change protocal HTTP to TCP (or HTTPS to SSL). Here is some GitHub Gist which can help you: https://gist.github.com/obolton/071be4c926f9cf0b6fd8.

  • T Jay

    Hi, would you be able to do this our our app? Looking for an estimate. Thanks.

    • tmuter

      T Jay, please contact me on [email protected]. We will prepare for you an estimation together with Krzysiek.

      • Ilya Zatulovskiy

        I’m sending you an email as well with the same request

        • tmuter

          Ilya, I got it. See you on email!

  • Is there a section on what changes need to be made to the app in order to get the SDK pointed to the new parse server?

    • Krzysztof Zielonka

      Unfortunately there is not.

      If you starts new application it should be enough to just change versions of packages in package.json file to the newest versions. The newest parse-server version is 2.3.1 and express is 4.14.0 but you need also to check your Node.js version. As you can check here: https://github.com/ParsePlatform/parse-server/blob/2.3.1/package.json#L81
      the newest parse-server requires node at least at version 4.5 so you have also to install such or greater version and updated you package.json to use it.

      If you want to upgrade existing project there can be problem with dependencies, if you have any. In the simplest case It should be enough to just change their versions to the newest ones. In the hardest, if some of interfaces have changed, you will need to adjust your code.

  • Todd

    Why did you deploy parse-dashboard using docker vs deploying directly to elastic beanstalk? My parse-server app is long since migrated to AWS Elastic Beanstalk but I’ve been putting off migrating parse-dashboard. It seems like deploying using docker is overkill.

  • ami

    Hi,

    I have a couple of questions:

    1. What kind of file is the Dockerfile? It doesn’t have json extension like the others.
    2. Where the folder with the 3 files should be located? Should it be inside the parse-dashboard-master.zip and then “upload and deploy” parse-dashboard-master.zip to AWS or should I deploy only the zipped folder with the 3 files?

    Thanks,
    Ami

  • ami

    Hi,

    I’ve done all steps which describe how to deploy the parse dashboard and it failed.
    This is the message I’ve received:

    [Instance: i-0cced1c67e27fc664] Command failed on instance. Return code: 1 Output: cat: Dockerrun.aws.json: No such file or directory cat: Dockerrun.aws.json: No such file or directory No Docker image specified in either Dockerfile or Dockerrun.aws.json. Abort deployment. Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/03build.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.

    What is wrong?

    Ami