Introduction

Amazon Web Services is a subsidiary of Amazon providing on-demand cloud computing platforms and APIs: using this platform it is possible to create large, automated, scalable applications in the cloud.

It is possible then, to create on-demand applications that leverage our Photogrammetry SDK FlowEngine on amazon AWS, so that when you can automatically scale up or down your processing power in a transparent and seamless way.

To achieve this objective, we will use Amazon AWS, FlowEngine and Docker, a set of platform as a service products that use OS-level virtualization to deliver software in packages called containers. By pre-configuring a docker container to our needs, we will then be able to spawn as many containers on-demand.

Make sure to get familiar with Amazon IAM as you will need to create credentials in order to use the container on AWS, hence, your development machine must also be configured to use the Amazon CLI – a great way to get started quickly is to simply fire up a Linux virtual machine on Amazon EC2 as most linux images come pre-configured with said tools.

Tutorial Architecture

This tutorial aims to build a solution that:

  • Can interact with Amazon S3 to download the data that will be processed
  • Can spawn on-demand tasks with FlowEngine on a cluster in ECS
  • Can interact with Amazon S3 to upload the results of the reconstruction

The automation on how the data is uploaded (and how the tasks are supposed to be spawned) is left to the user as this tutorial focuses on the FlowEngine interaction with the S3 ecosystem – how you bring data in, your policy to spawn tasks and what you do with the resulting data, is out of the scope of this tutorial.

The Docker Component

We assume you are already familiar with Docker and that you have already installed docker in your development machine, however we will walk you through the required step to create a CUDA ready container that will hold your custom FlowEngine component. Once we have prepared this Docker container, it will be uploaded to your Amazon account so that it can be defined as a task in Amazon ECS.

The FlowEngine Component

We assume that you are already familiar with FlowEngine and that you have already created the perfect photogrammetry solution for your needs with it. You will likely need some additional customization to integrate this tutorial with your workflow, so for this tutorial we will keep things simple and use the basic FlowEngine example: our executable will simply run a reconstruction based on the Default.xml file we will upload and save an output .3dk file as result.

The S3 Component

We assume that you are already familiar with Amazon S3. For readability purposes on this tutorial, we assume that each task will have all the required data (photos and XML settings for the reconstruction) uploaded on an S3 bucket in a single zip file, in the format ProjectName/ProjectName.zip – our solution will fetch that archive, unpack it, run it in the FlowEngine Component, and upload back the result in the same bucket in the format ProjectName/ProjectName.3dk

The ECS and ECR Components

We assume that you are already familiar with Amazon ECS. For the purpose of this tutorial, after we have created our cluster and infrastructure, we will launch the task manually from the AWS web interface – additional customization to spawn tasks as needed are left to the reader according to their own scalability needs. If you plan in creating your own custom conatiner, we will also need to push our docker image on an Amazon ECR repository before we can consume it on an ECS cluster.

Picking the right FlowEngine solution for you

You can either use your own existing FlowEngine license (full or free) to completely customize your flow. You can pick any cloud provider you may like, although the orchestration may differ.