Upload directly to Amazon S3

Uploading using ftp has been used by some of our customers as an alternative to uploading using the vms. The main reason for using ftp is uploading multiple files at once. Now that we've transitioned from physical datacenter to the cloud based AWS, another way upload files has presented itself.

This example using fineuploader is mostly based on the documentation available at https://docs.fineuploader.com/. As a side note, this is just an example and only one way to upload files to S3. There may be a better way to upload files to S3, and we encourage you to keep your options open.

Example code can be found at this location: fineuploader-example.zip

So let's get started.

In this example we're using a macbook pro, so this will probably differ from linux/unix and definately differ for Windows. To run this code you will need php which is pre-installed on osx. Please make sure php.ini contains the correct settings: variables_order = "EGPCS" and error_reporting = E_ERROR These settings will make sure that environment variables can be used in php and errors will not show up in the json that's returned from the fineuploader endpoint.

If you're reluctant to install npm and composer on your mac, there's example code with all of the requirements to start working right away.

The easiest way to install node.js is using homebrew https://brew.sh/.

  1. brew install npm

Let's install composer https://getcomposer.org/ as well. Just follow the instructions to install composer here https://getcomposer.org/download/.

Now we can start working on building our application. My favorite editor is Vim, please replace this command by your editor of choice.

  1. mkdir fineuploader-example
  2. cd fineuploader-example

To install the javascript version of fine-uploader:

  1. echo "{}" > package.json
  2. npm install fine-uploader

Any warnings from npm can be safely ignored.

To install the php backend of fine-uploader:

  1.         vim composer.json

Add:

  1. {
  2.   "require": {
  3.     "fineuploader/php-s3-server": "1.1.0"
  4.   }
  5. }

Install the fineuploader php backend:

  1. ./composer.phar install

The next step is to show something basic in your favorite browser.

  1. vim index.php

Add:

  1. <!DOCTYPE html>
  2.     <meta charset="utf-8">
  3.     <meta name="viewport" content="width=device-width, initial-scale=1">
  4.     <link href="node_modules/fine-uploader/s3.fine-uploader/fine-uploader-gallery.min.css" rel="stylesheet">
  5.     <script src="node_modules/fine-uploader/s3.fine-uploader/s3.fine-uploader.min.js""></script>
  6.     <script type="text/template" id="qq-template">
  7.         <div class="qq-uploader-selector qq-uploader qq-gallery" qq-drop-area-text="Drop files here">
  8.             <div class="qq-total-progress-bar-container-selector qq-total-progress-bar-container">
  9.                 <div role="progressbar" aria-valuenow="0" aria-valuemin="0" aria-valuemax="100" class="qq-total-progress-bar-selector qq-progress-bar qq-total-progress-bar"></div>
  10.             </div>
  11.             <div class="qq-upload-drop-area-selector qq-upload-drop-area" qq-hide-dropzone>
  12.                 <span class="qq-upload-drop-area-text-selector"></span>
  13.             </div>
  14.             <div class="qq-upload-button-selector qq-upload-button">
  15.                 <div>Upload a file</div>
  16.             </div>
  17.             <span class="qq-drop-processing-selector qq-drop-processing">
  18.                 <span>Processing dropped files...</span>
  19.                 <span class="qq-drop-processing-spinner-selector qq-drop-processing-spinner"></span>
  20.             </span>
  21.             <ul class="qq-upload-list-selector qq-upload-list" role="region" aria-live="polite" aria-relevant="additions removals">
  22.                 <li>
  23.                     <span role="status" class="qq-upload-status-text-selector qq-upload-status-text"></span>
  24.                     <div class="qq-progress-bar-container-selector qq-progress-bar-container">
  25.                         <div role="progressbar" aria-valuenow="0" aria-valuemin="0" aria-valuemax="100" class="qq-progress-bar-selector qq-progress-bar"></div>
  26.                     </div>
  27.                     <span class="qq-upload-spinner-selector qq-upload-spinner"></span>
  28.                     <div class="qq-thumbnail-wrapper">
  29.                         <img class="qq-thumbnail-selector" qq-max-size="120" qq-server-scale>
  30.                     </div>
  31.                     <button type="button" class="qq-upload-cancel-selector qq-upload-cancel">X</button>
  32.                     <button type="button" class="qq-upload-retry-selector qq-upload-retry">
  33.                         <span class="qq-btn qq-retry-icon" aria-label="Retry"></span>
  34.                         Retry
  35.                     </button>
  36.  
  37.                     <div class="qq-file-info">
  38.                         <div class="qq-file-name">
  39.                             <span class="qq-upload-file-selector qq-upload-file"></span>
  40.                             <span class="qq-edit-filename-icon-selector qq-btn qq-edit-filename-icon" aria-label="Edit filename"></span>
  41.                         </div>
  42.                         <input class="qq-edit-filename-selector qq-edit-filename" tabindex="0" type="text">
  43.                         <span class="qq-upload-size-selector qq-upload-size"></span>
  44.                         <button type="button" class="qq-btn qq-upload-delete-selector qq-upload-delete">
  45.                             <span class="qq-btn qq-delete-icon" aria-label="Delete"></span>
  46.                         </button>
  47.                         <button type="button" class="qq-btn qq-upload-pause-selector qq-upload-pause">
  48.                             <span class="qq-btn qq-pause-icon" aria-label="Pause"></span>
  49.                         </button>
  50.                         <button type="button" class="qq-btn qq-upload-continue-selector qq-upload-continue">
  51.                             <span class="qq-btn qq-continue-icon" aria-label="Continue"></span>
  52.                         </button>
  53.                     </div>
  54.                 </li>
  55.             </ul>
  56.  
  57.             <dialog class="qq-alert-dialog-selector">
  58.                 <div class="qq-dialog-message-selector"></div>
  59.                 <div class="qq-dialog-buttons">
  60.                     <button type="button" class="qq-cancel-button-selector">Close</button>
  61.                 </div>
  62.             </dialog>
  63.  
  64.             <dialog class="qq-confirm-dialog-selector">
  65.                 <div class="qq-dialog-message-selector"></div>
  66.                 <div class="qq-dialog-buttons">
  67.                     <button type="button" class="qq-cancel-button-selector">No</button>
  68.                     <button type="button" class="qq-ok-button-selector">Yes</button>
  69.                 </div>
  70.             </dialog>
  71.  
  72.             <dialog class="qq-prompt-dialog-selector">
  73.                 <div class="qq-dialog-message-selector"></div>
  74.                 <input type="text">
  75.                 <div class="qq-dialog-buttons">
  76.                     <button type="button" class="qq-cancel-button-selector">Cancel</button>
  77.                     <button type="button" class="qq-ok-button-selector">Ok</button>
  78.                 </div>
  79.             </dialog>
  80.         </div>
  81.     </script>
  82.  
  83.     <title>Fine Uploader Gallery UI</title>
  84. </head>
  85.     <div id="uploader"></div>
  86.     <script type="text/javascript">
  87.         var uploader = new qq.s3.FineUploader({
  88.             element: document.getElementById("uploader"),
  89.             debug: true,
  90.             request: {
  91.                 endpoint: "http://s3-eu-west-1.amazonaws.com/YOUR_BUCKET_NAME",
  92.                 accessKey: "YOUR_PUBLIC_ACCESS_KEY"
  93.             },
  94.             objectProperties: {
  95.                 acl: "public-read-write",
  96.                 key: function( id ) {
  97.                         var date = new Date();
  98.                         var year = date.getUTCFullYear();
  99.                         var month = date.getMonth() < 10 ? '0' + date.getMonth() : date.getMonth();
  100.                        var day = date.getDay() < 10 ? '0' + date.getDay() : date.getDay();
  101.                        // In order to process the file it should be placed in the upload "folder" on S3
  102.                        var filename = 'upload/' + '<publication_name>' + '/' + year + '/' + month + '/' + day + '/' +  this.getName(id);
  103.                         return filename;
  104.                 }
  105.             },
  106.             signature: {
  107.                 endpoint: "/vendor/fineuploader/php-s3-server/endpoint.php"
  108.             },
  109.             uploadSuccess: {
  110.                 endpoint: "/vendor/fineuploader/php-s3-server/endpoint.php?success"
  111.             },
  112.             iframeSupport: {
  113.                 localBlankPagePath: "success.html"
  114.             },
  115.             chunking: {
  116.                 enabled: true,
  117.                 concurrent: {
  118.                     enabled: true
  119.                 }
  120.             },
  121.             resume: {
  122.                 enabled: true
  123.             },
  124.             retry: {
  125.                 enableAuto: true,
  126.                 showButton: true
  127.             },
  128.             deleteFile: {
  129.                 enabled: true,
  130.                 endpoint: "/vendor/fineuploader/php-s3-server/endpoint.php"
  131.             }
  132.         })
  133.     </script>
  134. </body>
  135. </html>

Important to note is that the upload key should always start with "upload/" otherwise the item will not be uploaded into the vms. Also good to know is that the request endpoint needs to be http://s3-eu-west-1.amazonaws.com/YOUR_BUCKET_NAME because of the dots (mm.YOUR_BUCKET_NAME.bbvms.com) used in the name.

Create a startup script to start a development server:

  1. vim startServer.sh

Add:

  1. #!/bin/bash
  2. export AWS_SERVER_PUBLIC_KEY=YOUR_PUBLIC_ACCESS_KEY
  3. export AWS_SERVER_PRIVATE_KEY=YOUR_SECRET_KEY
  4. export AWS_CLIENT_SECRET_KEY=$AWS_SERVER_PRIVATE_KEY
  5. export S3_BUCKET_NAME=YOUR_BUCKET_NAME
  6. php -S 0.0.0.0:8080

Make sure the file is executable:

  1. chmod +x startupServer.sh

Now to run the development server:

  1. ./startupServer.sh

Use your favorite browser to go to http://127.0.0.1:8080

This should show the image below:

Now you should be able to upload images to S3.

Add metadata to a file that's uploaded

To upload the metadata to the VMS we recommend using the VMS Rpc for php or java.

To be able to add metadata to a file that has been uploaded using fineuploader, we're using the filename. Uploading a file with filename test-2313425536.mxf will translate to a sourceid of test-2313425536.mxf.

  1. <?php
  2. require_once( 'bootstrap.php' );
  3.  
  4. $token = 'TOKEN_PROVIDED_BY_SUPPORT';
  5. $vms = new VMSRpc('https://<PUBLICATION_NAME>.bbvms.com', null, null, $token);
  6.  
  7. $properties = array('srcid'=>'test-2313425536.mxf,'xml' => '<media-clip title="New mediaclip" status="published" sourceid="test-2313425536.mxf"></media-clip>');
  8. $result = $vms->doAction('mediaclip', 'put', $properties);
  9.  
  10. print_r( $result );
  11. ?>