Deploy Apache NiFi on Docker with AWS EC2 instance and Connect to Web Interface

Here First, you have to create an AWS EC2 instance. Here I’m using Amazon Linux 2 free tier eligible instance.

Log into the created Linux instance

STEP 01 — Install Docker on EC2 instance.

Update the installed packages and package cache on your instance.

sudo yum update -y

Install the latest Docker Engine package.

sudo amazon-linux-extras install docker

Start the Docker service.

sudo service docker start

(Optional) On Amazon Linux 2, to ensure that the Docker daemon starts after each system reboot, run the following command

sudo systemctl enable docker

Add the ec2-user to the docker group, so you can execute Docker commands without using sudo.

sudo usermod -a -G docker ec2-user

Now, log out and log in again to have the group access permission. Alternatively, you can do this by closing the current SSH terminal and connecting using a new terminal.

Here, you may be a little confused about how I got this ec2-user. Actually, it is the username that is used to log in to the console. You can find it here.

user

Also, you can find all the user groups and users using getent group

User Groups

To list all the members of a group, use getent groupcommand followed by the group name.

getent group docker

If you want to get all the groups, the current user belongs to, simply use groupscommand.

groups

Alright, now you can verify whether ec2-usercan runs Docker commands without sudo.

docker info

Verifying ec2-user can run docker commands without sudo

STEP 02— Build Docker Container with NiFi Image

Now we pull NiFi Docker Registry Image from the official docker registry and build inside a Container

docker pull apache/nifi:1.12.1

Hereapache/nifiis the repository(Docker Registry) and 1.12.1 is the Tag (REPOSITORY/IMAGE:TAG) of this specific image among other NiFi images. The reason for using this image is it is a pure bug fix release according to the official Apache NiFi release site. https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.12.1

Now we run the pulled image inside a container.

docker run --name nifi -p 8080:8080 -d apache/nifi:1.12.1 --restart=always

Here, -p 8080:8080is the port configuration and how it is published to the outside world. You can get a proper idea about this by referring the following note from the official Docker site.

Docker ports

It's something like this,

How ports work

STEP 03 — Setup relevant timezone and getting necessary connectors for NiFi to process properly.

For this, we have to jump into the container’s shell(previously we ran commands on host OS’s shell) using following command.

docker exec -i -t nifi /bin/bash

Now you can execute commands from the Container’s bash shell. In order to setup the time of the NiFi application, you have to change the time zone of the JVM(Java Virtual Machine)

echo "java.arg.8=-Duser.timezone=Asia/Colombo" >> conf/bootstrap.conf

Now go to the lib directory and get the following connectors as you wish.

cd lib
wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.23/mysql-connector-java-8.0.23.jar
wget https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc8/21.1.0.0/ojdbc8-21.1.0.0.jar
wget https://repo1.maven.org/maven2/org/apache/nifi/nifi-kite-nar/1.12.1/nifi-kite-nar-1.12.1.nar
wget https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem
wget https://jdbc.postgresql.org/download/postgresql-42.2.19.jar

Now get out of the container’s shell simply by Ctrl+D or using exitcommand. That’s all. In addition to this if you face any running issues with the container please execute following commands.

Update the restart policy

docker update --restart=always nifi

Read the available restarting policies

docker inspect -f "{{ .HostConfig.RestartPolicy }}" nifi

Important:

Since we running this docker container on an EC2 instance, if we want to connect to it using exposed ports (8080of the host and 8080 of the container) we have to enable access for incoming connection requests to that EC2 instance. For that go to EC2>Security Groups>Inbound rules>Edit inbound rules. Now add a Custom TCP Port as 8080 to the instance and give 0.0.0.0/0 IP as the source(It means anyone/any IP has access) and save.

Now open your browser and connect to the running NiFi application on Docker container through the exposed port.

http://EC2instancepublicip:8080/nifi/

NiFi Web interface