FrankTheDevop FrankTheDevop FrankTheDevop FrankTheDevop FrankTheDevop

Monthly Archives :

February 2019

What´s a Dockerfile?

150 150 Frank

Hey everyone,

after the introduction of Docker I thought I show you the recipe to create such a container. The recipe is like a recipe for your favorite lasagne and is called Dockerfile.
It looks similar to this:
FROM node:8.15.0-alpine

# Add application folder
RUN mkdir /app
WORKDIR /app

# Add package.json and install deps
ADD package.json /app/package.json
RUN npm install

COPY . .

# Expose the listening port
EXPOSE 1235

# Start the server
CMD [“pm2-docker”, “start”, “process.yml”]

As you see it is quite short and nothing to be afraid about. So lets get through it:
FROM node:8.15.0-alpine
As mentioned in the introduction Docker uses multiple layer to form a container. With this we tell it which container in which version to use as our starting point.

# Add application folder
Lines starting with a # are comments. Use the to explain anything that isn´t clear to you

RUN mkdir /app
RUN is the command to execute a custom command. In this case create a directory app in the root of the container filesystem

WORKDIR /app
WORKDIR is similar to the commandline command cd. It changes the current working directory for the following commands.
Here we change into the newly created directory /app.

ADD package.json /app/package.json
Adding a file to the container in the mentioned path (/app/package.json).

RUN npm install
Again we run a custom command, here npm install to install all dependencies for the used Nodejs project.

COPY . .
COPY copies all content in the current folder recursively into the mentioned path of the container (. translates to the last WORKDIR command, so we copy to /app).
COPY respects a .dockerignore file, with which you can filter out files you don´t want to copy over (e.g. log files).

CMD [“pm2-docker”, “start”, “process.yml”]
Docker needs a starting point / software. If you want to let the container running the software needs to run.
With CMD we define which software we want to use for this.
Here we use pm2-docker from the pm2 package to keep (nodejs) projects running (restart when necessary, etc.).

And that´s it. Basically you can now already package your own project into a docker container. Just execute “docker build -t $yourname/$your_container_name:$your_version .” and it will be build. An example can be “docker build -t frankthedevop/MyTodolistAPI:v0.0.1 .”.
You can then run it with “docker run $yourname/$your_container_name:$your_version” or in my case “docker run frankthedevop/MyTodolistAPI:v0.0.1”. More details about the commands in a later post.

Just remember that we are locker inside the container, that means that all paths are inside the container too. If you have e.g. a configuration file in your personal user directory make sure to copy it to the correct point inside the container, otherwise it doesn´t exist.

This is just the beginning, but I wanted you to see how short and easy such a recipe can be.
You find further links in the sources. And I will create a more extensive post about it soon too.

Feel free to contact me about how to create a Dockerfile for your project. I will do my best to help you :).

Yours sincerely,
Frank

Sources:
(1) Docker Dockerfile Documentation
(2) Docker 101: Fundamentals & The Dockerfile
(3) Inspiration how to use Dockerfiles

What is Docker? Or: Why choose Docker?

150 150 Frank

Hey everyone,

today it´s about one of the virtualisation solutions out there: Docker.
I want to help you understand quickly how it works.

What is Docker?
As mentioned Docker is a software virtualisation solution. There are others out
there which are different. The important point is that it uses container virtualisation.
That means with Docker you don´t have to install a whole virtual machine if you want to
e.g. setup a webserver. You package the webserver into a container and docker reuses the
operating system it runs on. This means a difference in size, both storage and main memory.

In short it looks like this:
Host OS -> Docker -> Docker Container
-> Docker Container
-> Docker Container
.
.
.

When to use it?
I personally use Docker for most of my projects of custom developed online hosted software.
You probably heard of the MEAN (Mongo, Express, Angular, Nodejs) Stack. To host those Docker
is often used.

Whats the benefit?
When you create software whose fundamentals work similarly (e.g. the same Nodejs Framework)
you write the Recipe to package the Software (Dockerfile) once and can copy it over with
little changes.

Docker container are stateless. You can rest assured that you can start a software from the same
configuration every time. You change the configuration of the data storage but else it works everywhere the same.
No more “but it works on my computer”.

Easy to combine into Systems (Stacks). You take multiple containers with different software and can combine them
to achieve your goal. You can reuse the same containers, just point the configuration to e.g. another Database and it works.

Works like a charm with the concept of microservices. With Docker you can set up microservices easier. I have e.g. one container
to handle a generic JobSystem for distributed workflows. I need it in a new stack? No problem, point to it and it basically is available.
The new System creates to much load? Start another instance of that container to do load balancing.

Online Hosting is available and affordable. It is not as cheap as a $3.95/month web hosting but it doesn´t need to cost $1000/month either.
The exact cost will of course depend on the size of your project.

What are the negatives?
Docker does virtualisation and virtualisation itself takes up more resources.

Docker has layers of containers. If you have to many layers the logic slows down.

Sometimes it takes more time to find the correct base image layer and install software in the correct way.

Conclusion
Like everything Docker has it´s pros and cons. Every user has to weigh them and draw his/her own conclusion.
Personally I use a microservice approach. And Docker is a great help for that after you figured a few things out.
I love that I can package my software into a stateless container, configure it externally with connections to data and systems and know that it works the same on my laptop as well as my server.
If it doesn´t work on the server then I know 99.9% of the time that it is a configuration problem. That alone already saves a headache or two ;).
But I am not at the point of an enterprise that uses 1000s of container yet. That might be a bigger challenge with Docker. But the knowledge from Docker will make a transition even then easier.

I hope I was able to give you a quick overview of Docker as a base for following posts and your own decision whether to use it or not.

Yours Sincerely,
Frank

Sources:
(1) Docker Curriculum.com
(2) Docker from the Beginning I

Script in Node.js to iterate a directory and extract information from it´s files

150 150 Frank

Hi everyone,

after we did the template last time, I want to show you how to put the single pieces together.
Based on a task at hand I choose the example of iterating and working with files in a directory.
The exact task was:
– Iterate a directory
– find all JSON files in it
– read them
– extract all objects in them
– extract the property email from them
– extract the unique domains of the email addresses
– count how often each domain occurs
– write this information to a summary file for further processing / display

'use strict'

const Promise = require('bluebird')
const fs = require('fs');
const path = require('path');
const util = require('util');

// Promisify only readdir as we don´t need more
const readdirAsync = Promise.promisify(fs.readdir);
const writeFileAsync = Promise.promisify(fs.writeFile);

// Commandline handling
const optionDefinitions = [
  { name: 'folder', alias: 'f', type: String }
]
const commandLineArgs = require('command-line-args')
const options = commandLineArgs(optionDefinitions)

// Add the path to your files
const folder = options.folder
// e.g. '/Users/$Yourusername/Downloads/customerdata';

// This will hold all entries from all files
//  Not unique
const all = []

// Read all files in our directory
return readdirAsync(folder)
  .then(files => {
    files
      .map(entry => {
        if(entry.indexOf('.') > 0 && path.extname(entry) === '.json') {
          // In case there are .json files in the folder that are not in JSON format
          try {
            const temp = require(path.join(folder, entry))

	    // I know for sure that all entries have an filled email column so I can just split here
            // and extract the domain name without checking
            temp.map(entry => (all.push(entry.email.split('@')[1])))
          } catch (e) {}
          return null
        }
      })

      console.log(all)
  })
  .then(() => {
    // Create a unique array
    // Use the new Set feature of ES 6
   return Promise.resolve([...new Set(all)])
  })
  .then(allUnique => {
    // Get a list of unique entries with the number of times it appears
    let newList = []

    allUnique.map(entry => {
      const t = all.filter(innerEntry => innerEntry === entry)
      newList.push({name: entry, count: t.length})
    })

    return Promise.resolve(newList)
  })
  .then(allUnique => {
    allUnique.sort((a,b) => b.count - a.count)
    return Promise.resolve(allUnique)
   })
  .then(allUnique => {
    let content = allUnique.reduce((a, b) => a + `${b.name};${b.count}\n`, '')
    return writeFileAsync(path.join(folder, 'all.txt'), content)
  })
  .then(data => {
    console.log('Wrote file successfully')
  })
  .catch(err => {
    console.log('An error occurred:', err)
  })

You find the repository for it here.

If you are looking for the explanation, continue to read. Otherwise be happy with the template and change it to your hearts desire ;).

This one is a bit longer but stay with me, we will go through it together.
At first we have the standard block where we import all required libraries in Line 3-6.

Then we convert the async, callback-based functions for readdir and writeFile to Promises (we promisify them) for easier
and more elegant handling in Line 9-10.

Next comes the handling of command line (CLI) parameters as we did before in Line 12 – 21.

We define an array all which will receive all email domains from the read files (not unique) in Line 25 .

Now we have everything together to start:
We read the directory content in Line 28.
In Line 29 it returns an array of all found files
With Line 30-31 we start iterating all elements of the array and with Line 32 with make sure that only files ending with .json are accepted, all others are ignored
Line 34-41 is a bit of cheating, Node.js is able to require an JSON file. So instead of reading the file, parsing it and having to handle it all myself, I use the functionality of require.
In case there is a JSON file that can not be parsed I wrap it into a try error, so that it continues on an error
Line 39 does a few things at once:
– With .map I iterate over all entries in the file
– I know each object contains an email property, therefore I act on it without checking
– An email address is in the form username@domainname.domainextension, I need the domain name and extension, so I split the email property and take the second half of the email address, which is the domain part
– Each of these domain parts is pushed into the array for further processing

After all the processing I make a debug output in Line 45.

JavaScript ES 6 introduced some nice new features, one is a Set (an “array” of unique values) and the decomposition operator. In Line 50 I return an new array that is created by decomposing the Set,
so in short: In one line I get an unique array of domains.

In the next function we create a new array of object with the domain name and the number of occurrences. For that we iterate over each entry of the unique array in Line 56,
use the filter method of the not unique array with all entries in Line 57. The filter method returns an array, so I can create the JSON object with the number of occurrences easily by using array.length in Line 58.

After I have the array with the number of occurrences I want to see it sorted. The sort function allows use to provide a function how to sort. And thanks to (5) & (6) I found an short way to do in as you see
in Line 64.

In the last function I use the array.reduce function to create a string from the JSON objects. You can see this post-processing step in Line 68.

All that is left is to write the data to a file as you see in Line 69.

This is followed by a simple message to signal that the script has successfully finished (Line 72) or the output of the error if one occurred in Line 75.

I hope I could help you save time again in your race against the clock and you found the explanations useful.

Yours sincerely,
Frank

Sources:
(1) How to escape Callback Hell
(2) Explanation of Node.js CLI Argument handling
(3) Explanation of Node.js CLI Argument handling II
(4) My own short example of an template for Node.js CLI Argument handling
(5) Sorting an array
(6) Sorting an array of objects by their property
(7) How to write to a file in Node.js
(8) How to avoid making mistakes with Promises
(9) Repository for the scrip
(10) Escape Callback Hell with Promises
(11) My article about how to convert (promisify) an async function with callback to a Promise based one

Commandline tools with Nodejs

150 150 admin

Hi everyone,

sometimes you need a small tool like but you might be working for an extended period of time in Node.js so that you don´t want to
switch languages and loose time and momentum on it. You want to do it quickly, but correctly to be able to reuse it in one way or another at a later point.
This is what today is about about.

I will show you quickly how to do a template for commandline arguments and handling them comfortably, so that you have this off your plate.

Here is the code for it:

'use strict'

const commandLineArgs = require('command-line-args')

// Commandline handling
const optionDefinitions = [
{ name: 'folder', alias: 'f', type: String }
]

const options = commandLineArgs(optionDefinitions)

// Add the path to your files
const folder = options.folder
// '/Users/$YourUsername/Downloads/customerdata';
console.log(`Given folder:${folder}`)

Explanation:

I use the npm package command-line-args to be able to handle commandline arguments easily.
Line 3: At first we import the command-line-args package.
Line 6: Then we define the options we want to be able to use. I chose an option folder with the type string.

Line 10: After we defined them we feed then to commandLineArgs and it parses them for use and returns a json document with the result.

Line 13: In that result we have properties with the name we defined in our options and we can extract them like we are used to.

If you save it as template.js, the following syntax is supported on the commandline:

node template.js --folder $YourFolder
node template.js --folder=$YourFolder
node template.js -f $YourFolder

As you can see we do have defined long and short form of the parameter that is required.

You can find the file on https://github.com/FrankTheDevop/cli-template too.
The npm package you find on https://www.npmjs.com/package/command-line-args and it´s repo on https://github.com/75lb/command-line-args.

I hope I could help you save some time i research and trial & error with this short nugget.

Yours Sincerely,
Frank

Sources:
(1) https://flaviocopes.com/node-cli-args/
(2) https://code-maven.com/argv-raw-command-line-arguments-in-nodejs
(3) https://codeburst.io/need-for-promises-and-rookie-mistakes-to-avoid-when-using-promises-9cabba215e04