FrankTheDevop FrankTheDevop FrankTheDevop FrankTheDevop FrankTheDevop

MLAND Series – Tips II – Loopback AthenaPDF

150 150 Frank

Purpose

When developing an Infrastructure for a Client Project I faced the situation that I needed to support 50 concurrent request to generate a PDF. The purpose of this post is to show you how I did this using AthenaPDF and RabbitMQ.

Setup

The Setup used for this Tip is a machine on Digital Ocean*, created and managed through Docker Cloud and with an Loopback API Container, an AthenaPDF Container and RabbitMQ hosted on CloudAMQP .

 

Mixin Definition

You can use a mixin definition similar to this:

"mixins": {
    "MessageQueue": {
      "producers": {
        "producerSendGeneratePDF": {
          "exchange": "generate.pdf",
          "options": {
            "routingKey": "pdf",
            "type": "company.project.messages.generatepdf",
            "contentType": "application/json"
          }
        },
        "producerPdfGenerated": {
          "exchange": "generate.pdf",
          "options": {
            "routingKey": "pdf",
            "type": "company.project.messages.pdfgenerated",
            "contentType": "application/json"
          }
        }
      },
      "consumers": {
        "consumerGeneratePDF": {
          "queue": "client.generate.pdf",
            "type": "company.project.messages.generatepdf",
        },
        "consumerPDFGenerated": {
          "queue": "client.generate.pdf",
            "type": "company.project.messages.pdfgenerated",
        }
      }
    }
  }

 

With this configuration you can execute  Model.producerSendGeneratePDF({data}) and send a message of the following type:

"company.project.messages.generatepdf"

 

The message with call Model.consumerGeneratePDF where you can prepare the HTML Document you want to be converted and call the AthenaPDF Container to create it. After the work is done in Model.consumerGeneratePDF you can call producerPdfGenerated({data}) to start the next step of your workflow.

If you go with my config, you implement Model.consumerPDFGenerated. There you can execute whatever next step you want to do, send an e-mail to your customer, Upload to Amazon S3 or whatever your next step is.

 

How to limit the AthenaPDF Container Resource consumption

To limit the Memory Usage of the AthenaPDF Container at this to the Docker Stackfile definition of the Container: mem_limit: Xm.

X is the number of MB you allow the Container to get. 256 MB is a good starting point if you start measuring.

How to fix Xlib: extension “RANDR” missing on display “:99”

If you are getting this error it means the shared memory area /dev/shm isn´t shared between the host and the Container.  To fix this add these two Lines to the PDF Container definition of the Stackfile:

 

volumes:
  - '/dev/shm:/dev/shm'

 

How to not overload the AthenaPDF Container

Especially when you limit the amount of  Memory the PDF Container can use you want to limit the number of concurrent PDFs being generated.

Referring to the RabbitMQ Tip you can limit the number of concurrent tasks that the API Endpoint does at any given moments:

Check the Section server/component-config.json, in the loopback-component-mq definition

{  
   "loopback-component-mq":{  
      "path":"loopback-component-mq",
      "options":{  
         "restPort":15672,
         "acls":[  
            {  
               "accessType":"",
               "principalType":"ROLE",
               "principalId":"$unauthenticated",
               "permission":"DENY"
            }
         ]
      }
   },
   "topology":{  
      "connection":{  
         "uri":"amqp://user:password@host/vhost",
         "timeout": timeoutinmilliseconds
      },
      "exchanges":[  
         {  
            "name":"exchangename",
            "type":"exchangetype",
            "persistent":persistent
         }
      ],
      "queues":[  
         {  
            "name":"queuename",
            "subscribe":true,
            "limit":concurrentnonacknoledgedtasknumber
         }
      ],
      "bindings":[  
         {  
            "exchange":"exchangename",
            "target":"queuename",
            "keys":[  

            ]
         }
      ],
      "logging":{  
         "adapters":{  
            "stdOut":{  
               "level":5,
               "bailIfDebug":true
            }
         }
      }
   }
}

Experiment with the limit in the queues definition and see how many concurrent tasks your infrastructure can handle with you configuration.

 

Scaling

Vertically

The great point about this way of setting up your infrastructure is that you can easily scale horizontally. Just give the PDF Container more Memory, get a bigger Machine and increase the limit of the queue definition.

Horizontally

If you want to scale vertically you need to add a haproxy container and change your configuration a bit. I will show you how to do it in the next Tip.

 

Sources

AthenaPDF

Digital Ocean* (Affiliate Link)

Docker Cloud

RabbitMQ

CloudAMQP

Leave a Reply

Your email address will not be published.

16 − 10 =