DockerでApache NiFi 1.7.1、PostgreSQL10.5がインストールされたコンテナを構築する

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

○Apache NiFiの画面

○構築方法
以下の手順で、Apache NiF i.7.1とPostgreSQL10.5のコンテナを構築・実行します。

1. Apache NiFi公式にPostgreSQL JDBCドライバを追加したイメージの作成(Dockerfileがあるフォルダで実行)

docker build -t nifi171pgsql .

Dockerfile

FROM apache/nifi:1.7.1
USER root
RUN wget https://jdbc.postgresql.org/download/postgresql-42.2.4.jar -P /usr/share/java
USER nifi
VOLUME /opt/nifi

2. Apache NiFiとPostgreSQLコンテナの構築・実行(docker-compose.ymlがあるフォルダで実行)

COMPOSE_HTTP_TIMEOUT=180 docker-compose up -d

docker-compose.yml

version: "2"
services:
  db:
    image: postgres:10.5-alpine
    container_name: "nifi-db"
    ports:
      - "5432:5432"
    volumes:
      - "nifidb-data:/var/lib/postgresql/data"
    environment:
        POSTGRES_DB: nifi
        POSTGRES_PASSWORD: nifi
  nifi171pgsql:
    image: nifi171pgsql
    container_name: "nifi171pgsql"
    volumes:
      - "nifi-data:/opt/nifi"
    ports:
      - "8080:8080"
    depends_on:
      - db
volumes:
  nifidb-data:
    driver: local
  nifi-data:
    driver: local

3.ブラウザから以下のURLにアクセス
http://<Dockerホスト名またはIP>:8080/nifi/

※DBへの接続
4.DBCPConnectionPoolで、以下のようにパラメータを設定してPostgreSQLにアクセスします。
Database Connection -> URL jdbc:postgresql://db:5432/nifi
Database Driver Class Name -> org.postgresql.Driver
Database Driver Location(s) -> /usr/share/java/postgresql-42.2.4.jar
Database User -> postgres
Password -> nifi

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

DockerでApache NiFi 1.7.1、MySQL5.7がインストールされたコンテナを構築する

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

○Apache NiFiの画面

○構築方法
以下の手順で、Apache NiF i.7.1とMySQL 5.7のコンテナを構築・実行します。

1. Apache NiFi公式にMySQL JDBCドライバを追加したイメージの作成(Dockerfileがあるフォルダで実行)

docker build -t nifi171mysql .

Dockerfile

FROM apache/nifi:1.7.1
USER root
RUN wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.12.tar.gz -P /tmp \
  && tar xvfz /tmp/mysql-connector-java-8.0.12.tar.gz -C /tmp \
  && cp /tmp/mysql-connector-java-8.0.12/mysql-connector-java-8.0.12.jar /usr/share/java \
  && rm -Rf /tmp/mysql-connector*
USER nifi
VOLUME /opt/nifi

2. Apache NiFiとMySQLコンテナの構築・実行(docker-compose.ymlがあるフォルダで実行)

COMPOSE_HTTP_TIMEOUT=180 docker-compose up -d

docker-compose.yml

version: "2"
services:
  db:
    image: mysql:5.7
    command: mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
    container_name: "nifi-db"
    volumes:
      - "nifidb-data:/var/lib/mysql"
    ports:
      - "3306:3306"
    environment:
      MYSQL_DATABASE: nifi
      MYSQL_ROOT_PASSWORD: nifi
  nifi171mysql:
    image: nifi171mysql
    container_name: "nifi171mysql"
    volumes:
      - "nifi-data:/opt/nifi"
    ports:
      - "8080:8080"
    depends_on:
      - db
volumes:
  nifidb-data:
    driver: local
  nifi-data:
    driver: local

3.ブラウザから以下のURLにアクセス
http://<Dockerホスト名またはIP>:8080/nifi/

※DBへの接続
DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのPostgreSQLにアクセスします。
Database Connection -> URL jdbc:mysql://db:5432/nifi
Database Driver Class Name -> com.mysql.jdbc.Driver
Database Driver Location(s) -> /usr/share/java/mysql-connector-java-8.0.12.jar
Database User -> root
Password -> nifi

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

AnsibleでApache NiFiとPostgreSQL10をインストールする(Ubuntu18.04用)

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache NiFiの画面

〇インストール方法
1.下準備
~/.ansible.cfgに以下の内容を設定します

[ssh_connection]
pipelining=True
[defaults]
host_key_checking = False

2. インストール対象ホストのユーザやパスワードを環境に合わせてinventoryファイルに記入します

inventory例

[nifi-pg]
192.168.55.61

[all:vars]
ansible_ssh_port=22
ansible_ssh_user=vagrant
ansible_ssh_pass=vagrant
ansible_sudo_pass=vagrant

3.inventory、kanboard.ymlを準備し、以下のコマンドを実行します。

ansible-playbook -i inventory nifi-pg.yml

nifi-pg.yml

- hosts: nifi-pg
  vars:
    - dbname: test
    - dbuser: test
    - dbpassword: test
  tasks:
    - name: configure sources.list
      blockinfile:
        dest: /etc/apt/sources.list.d/pgdg.list
        create: yes
        block: |
          deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main
      become: true
    - name: add an apt key
      apt_key:
        url: https://www.postgresql.org/media/keys/ACCC4CF8.asc
        state: present
      become: yes
    - name: install packages
      apt:
        name: ca-certificates
        state: latest
      become: yes
    - name: Install required software
      apt: name={{ item }} state=latest
      with_items:
        - postgresql-10
        - libpq-dev
        - python-dev
        - python-pip
      become: true
    - name: install psycopg2 using pip
      pip:
        name: "{{ item }}"
        state: forcereinstall
      with_items:
        - psycopg2-binary
      become: true
    - name: configure postgresql.conf
      blockinfile:
        dest: /etc/postgresql/10/main/postgresql.conf
        create: yes
        block: |
          listen_addresses='*'
      become: true
    - name: edit pg_hba.conf
      replace: dest=/etc/postgresql/10/main/pg_hba.conf regexp="host.*all.*all.*127.0.0.1" replace="#host    all             all             127.0.0.1"
      become: true
    - name: edit pg_hba.conf
      replace: dest=/etc/postgresql/10/main/pg_hba.conf regexp="^host.*all.*all.*::1/128.*ident" replace="host    all             all             ::1/128    password"
      become: true
    - name: configure postgresql.conf
      blockinfile:
        dest: /etc/postgresql/10/main/pg_hba.conf
        create: yes
        block: |
          host    all         all         127.0.0.1/32          password
          host    all         all         192.168.1.0/24          password
          host    all         all         192.168.55.0/24          password
      become: true
    - name: enable and start postgresq.service
      systemd:
        daemon_reload: yes
        enabled: yes
        state: started
        name: postgresql.service
      become: true
    - name: create PostgreSQL user
      postgresql_user:
        name: "{{ dbuser }}"
        password: "{{ dbpassword }}"
        login_user: postgres
        encrypted: yes
      become: true
      become_user: postgres
    - name: create a database
      postgresql_db:
        name: "{{ dbname }}"
        owner: "{{ dbuser }}"
        encoding: 'UTF-8'
        lc_collate: 'ja_JP.UTF-8'
        lc_ctype: 'ja_JP.UTF-8'
        template: 'template0'
        login_user: postgres
      become: true
      become_user: postgres
    - name: download jdbc driver
      get_url:
        url: https://jdbc.postgresql.org/download/postgresql-42.2.4.jar
        dest: /usr/share/java
      become: true
    - name: configure limits.conf
      blockinfile:
        dest: /etc/security/limits.conf
        create: yes
        block: |
          *  hard  nofile  50000
          *  soft  nofile  50000
          *  hard  nproc  10000
          *  soft  nproc  10000
      become: true
    - name: Install openjdk
      apt: name={{ item }} state=present
      with_items:
        - openjdk-8-jdk
      become: true
    - name: download nifi
      get_url:
        url: http://ftp.riken.jp/net/apache/nifi/1.7.1/nifi-1.7.1-bin.tar.gz
        dest: /opt
      become: true
    - name: extract nifi
      unarchive:
        remote_src: yes
        src: /opt/nifi-1.7.1-bin.tar.gz
        dest: /opt/
      become: true
    - name: setup systemd
      blockinfile:
        dest: /etc/systemd/system/nifi.service
        create: yes
        block: |
          [Unit]
          Description=Apache Nifi
          After=syslog.target network.target
          [Service]
          Type=forking
          ExecStart=/opt/nifi-1.7.1/bin/nifi.sh start
          ExecStop=/opt/nifi-1.7.1/bin/nifi.sh stop
          KillMode=none
          [Install]
          WantedBy=multi-user.target
      become: true
    - name: clean up
      file:
        state: absent
        path: /opt/nifi-1.7.1-bin.tar.gz
      become: yes
    - name: enable and start nifi.service
      systemd:
        daemon_reload: yes
        enabled: yes
        state: started
        name: nifi.service
      become: true

4.DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのPostgreSQLにアクセスします。
Database Connection -> URL jdbc:postgresql://localhost:5432/test
Database Driver Class Name -> org.postgresql.Driver
Database Driver Location(s) -> /usr/share/java/postgresql-42.2.4.jar
Database User -> test
Password -> test

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

・psycopg2に関する他の記事はこちらを参照してください。

・その他のAnsibleの記事

AnsibleでApache NiFiとPostgreSQL10をインストールする(CentOS7.5用)

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache NiFiの画面

〇インストール方法
1.下準備
~/.ansible.cfgに以下の内容を設定します

[ssh_connection]
pipelining=True
[defaults]
host_key_checking = False

v 2. インストール対象ホストのユーザやパスワードを環境に合わせてinventoryファイルに記入します

inventory例

[nifi-postgresql]
192.168.1.107

[all:vars]
ansible_ssh_port=22
ansible_ssh_user=vagrant
ansible_ssh_pass=vagrant
ansible_sudo_pass=vagrant

3.inventory、kanboard.ymlを準備し、以下のコマンドを実行します。

ansible-playbook -i inventory nifi-postgresql.yml

nifi-postgresql.yml

- hosts: nifi-postgresql
  vars:
    - dbname: test
    - dbuser: test
    - dbpassword: test
  tasks:
    - name: install epel-release
      yum:
        name: epel-release
        state: present
      become: yes
    - name: download postgresql repos
      get_url:
        url: https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/pgdg-centos10-10-2.noarch.rpm
        dest: /tmp
      become: true
    - name: install repos
      yum: state=present name=/tmp/pgdg-centos10-10-2.noarch.rpm
      become: true
    - name: install PostgreSQL
      yum: name={{ item }} state=latest enablerepo=epel
      with_items:
        - postgresql10-server
        - postgresql10-devel
        - postgresql10-contrib
        - python-devel
        - python36-devel
        - python-pip
      become: true
    - name: install psycopg2 using pip
      pip:
        name: "{{ item }}"
        state: forcereinstall
      with_items:
        - psycopg2-binary
      become: true
    - name: initdb
      shell: /usr/pgsql-10/bin/postgresql-10-setup initdb
      become: yes
    - name: configure postgresql.conf
      blockinfile:
        dest: /var/lib/pgsql/10/data/postgresql.conf
        create: yes
        block: |
          listen_addresses='*'
      become: true
    - name: edit pg_hba.conf
      replace: dest=/var/lib/pgsql/10/data/pg_hba.conf regexp="host.*all.*all.*127.0.0.1" replace="#host    all             all             127.0.0.1"
      become: true
    - name: edit pg_hba.conf
      replace: dest=/var/lib/pgsql/10/data/pg_hba.conf regexp="^host.*all.*all.*::1/128.*ident" replace="host    all             all             ::1/128    password"
      become: true
    - name: configure postgresql.conf
      blockinfile:
        dest: /var/lib/pgsql/10/data/pg_hba.conf
        create: yes
        block: |
          host    all         all         127.0.0.1/32          password
          host    all         all         192.168.1.0/24          password
          host    all         all         192.168.55.0/24          password
      become: true
    - name: enable and start postgresql-10.service
      systemd:
        daemon_reload: yes
        enabled: yes
        state: started
        name: postgresql-10.service
      become: true
    - name: create PostgreSQL user
      postgresql_user:
        name: "{{ dbuser }}"
        password: "{{ dbpassword }}"
        login_user: postgres
        encrypted: yes
      become: true
      become_user: postgres
    - name: create a database
      postgresql_db:
        name: "{{ dbname }}"
        owner: "{{ dbuser }}"
        encoding: 'UTF-8'
        lc_collate: 'ja_JP.UTF-8'
        lc_ctype: 'ja_JP.UTF-8'
        template: 'template0'
        login_user: postgres
      become: true
      become_user: postgres
    - name: configure limits.conf
      blockinfile:
        dest: /etc/security/limits.conf
        create: yes
        block: |
          *  hard  nofile  50000
          *  soft  nofile  50000
          *  hard  nproc  10000
          *  soft  nproc  10000
      become: true
    - name: Install openjdk
      yum: name={{ item }} state=present
      with_items:
        - java-1.8.0-openjdk
      become: true
    - name: download connectorJ
      get_url:
        url: https://jdbc.postgresql.org/download/postgresql-42.2.4.jar
        dest: /usr/share/java
      become: true
    - name: download nifi
      get_url:
        url: http://ftp.riken.jp/net/apache/nifi/1.7.1/nifi-1.7.1-bin.tar.gz
        dest: /opt
      become: true
    - name: extract nifi
      unarchive:
        remote_src: yes
        src: /opt/nifi-1.7.1-bin.tar.gz
        dest: /opt/
      become: true
    - name: setup systemd
      blockinfile:
        dest: /etc/systemd/system/nifi.service
        create: yes
        block: |
          [Unit]
          Description=Apache Nifi
          After=syslog.target network.target
          [Service]
          Type=forking
          ExecStart=/opt/nifi-1.7.1/bin/nifi.sh start
          ExecStop=/opt/nifi-1.7.1/bin/nifi.sh stop
          KillMode=none
          [Install]
          WantedBy=multi-user.target
      become: true
    - name: setup systemd
      blockinfile:
        dest: /opt/nifi-1.7.1/bin/nifi-env.sh
        create: yes
        block: |
          
          JAVA_HOME=/usr/lib/jvm/jre-1.8.0
      become: true
    - name: clean up
      file:
        state: absent
        path: /opt/nifi-1.7.1-bin.tar.gz
      become: yes
    - name: enable and start nifi.service
      systemd:
        daemon_reload: yes
        enabled: yes
        state: started
        name: nifi.service
      become: true

4.DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのPostgreSQLにアクセスします。
Database Connection -> URL jdbc:postgresql://localhost:5432/test
Database Driver Class Name -> org.postgresql.Driver
Database Driver Location(s) -> /usr/share/java/postgresql-42.2.4.jar
Database User -> test
Password -> test

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

・psycopg2に関する他の記事はこちらを参照してください。

・その他のAnsibleの記事

AnsibleでApache NiFiとMariaDBをインストールする(CentOS7.5用)

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache NiFiの画面

〇インストール方法
1.下準備
~/.ansible.cfgに以下の内容を設定します

[ssh_connection]
pipelining=True
[defaults]
host_key_checking = False

2. インストール対象ホストのユーザやパスワードを環境に合わせてinventoryファイルに記入します

inventory例

[nifi-mariadb]
192.168.1.107

[all:vars]
ansible_ssh_port=22
ansible_ssh_user=vagrant
ansible_ssh_pass=vagrant
ansible_sudo_pass=vagrant

3.inventory、kanboard.ymlを準備し、以下のコマンドを実行します。

ansible-playbook -i inventory nifi-mariadb.yml

nifi-mariadb.yml

- hosts: nifi-mariadb
  vars:
    - dbname: test
    - dbuser: test
    - dbpassword: test
    - dbhost: localhost
    - dbport: 3306
  tasks:
    - name: install mariadb
      yum:
        name: mariadb-server
        state: present
      become: yes
    - name: enable and start mariadb.service
      systemd:
        name: mariadb
        enabled: yes
        state: started
      become: yes
    - name: install epel-release
      yum:
        name: epel-release
        state: present
      become: yes
    - name: Install required software
      yum: name={{ item }} state=present enablerepo=epel
      with_items:
        - mariadb-devel
        - python-devel
        - python36-devel
        - python-pip
      become: true
    - name: install MySQL-python using pip
      pip:
        name: "{{ item }}"
        state: forcereinstall
      with_items:
        - MySQL-python
      become: true
    - name: create db
      mysql_db:
        name={{ dbname }}
        state=present
        encoding=utf8
        login_user=root
      become: true
    - name: create and grant a database user
      mysql_user:
        name={{ dbuser }}
        password={{ dbpassword }}
        priv="test.*:ALL"
        state=present
      become: true
    - name: configure limits.conf
      blockinfile:
        dest: /etc/security/limits.conf
        create: yes
        block: |
          *  hard  nofile  50000
          *  soft  nofile  50000
          *  hard  nproc  10000
          *  soft  nproc  10000
      become: true
    - name: Install openjdk
      yum: name={{ item }} state=present
      with_items:
        - java-1.8.0-openjdk
      become: true
    - name: download connectorJ
      get_url:
        url: https://downloads.mariadb.com/Connectors/java/connector-java-2.2.6/mariadb-java-client-2.2.6.jar
        dest: /usr/share/java
      become: true
    - name: download nifi
      get_url:
        url: http://ftp.riken.jp/net/apache/nifi/1.7.1/nifi-1.7.1-bin.tar.gz
        dest: /opt
      become: true
    - name: extract nifi
      unarchive:
        remote_src: yes
        src: /opt/nifi-1.7.1-bin.tar.gz
        dest: /opt/
      become: true
    - name: setup systemd
      blockinfile:
        dest: /etc/systemd/system/nifi.service
        create: yes
        block: |
          [Unit]
          Description=Apache Nifi
          After=syslog.target network.target
          [Service]
          Type=forking
          ExecStart=/opt/nifi-1.7.1/bin/nifi.sh start
          ExecStop=/opt/nifi-1.7.1/bin/nifi.sh stop
          KillMode=none
          [Install]
          WantedBy=multi-user.target
      become: true
    - name: setup systemd
      blockinfile:
        dest: /opt/nifi-1.7.1/bin/nifi-env.sh
        create: yes
        block: |
          
          JAVA_HOME=/usr/lib/jvm/jre-1.8.0
      become: true
    - name: clean up
      file:
        state: absent
        path: /opt/nifi-1.7.1-bin.tar.gz
      become: yes
    - name: enable and start nifi.service
      systemd:
        daemon_reload: yes
        enabled: yes
        state: started
        name: nifi.service
      become: true

4.DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのMariaDBにアクセスします。
Database Connection -> URL jdbc:mariadb://localhost/test
Database Driver Class Name -> org.mariadb.jdbc.Driver
Database Driver Location(s) -> /usr/share/java/mariadb-java-client-2.2.6.jar
Database Userr -> test
Password -> test

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

・その他のAnsibleの記事

AnsibleでApache NiFiとMariaDBをインストールする(Debian Stretch/9.5用)

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache NiFiの画面

〇インストール方法
1.下準備
~/.ansible.cfgに以下の内容を設定します

[ssh_connection]
pipelining=True
[defaults]
host_key_checking = False

2. インストール対象ホストのユーザやパスワードを環境に合わせてinventoryファイルに記入します

inventory例

[nifi-mariadb]
192.168.1.107

[all:vars]
ansible_ssh_port=22
ansible_ssh_user=vagrant
ansible_ssh_pass=vagrant
ansible_sudo_pass=vagrant

3.inventory、kanboard.ymlを準備し、以下のコマンドを実行します。

ansible-playbook -i inventory nifi-mariadb.yml

nifi-mariadb.yml

- hosts: nifi-mariadb
  vars:
    - dbname: test
    - dbuser: test
    - dbpassword: test
    - dbhost: localhost
    - dbport: 3306
  tasks:
    - name: set password for root
      shell: echo "mariadb-server-10.1 mysql-server/root_password password root" | sudo debconf-set-selections
      become: yes
    - name: set password for root
      shell: echo "mariadb-server-10.1 mysql-server/root_password_again password root" | sudo debconf-set-selections
      become: yes
    - name: install mariadb
      apt:
        name: mariadb-server
        state: present
      become: yes
    - name: enable and start mariadb
      systemd:
        name: mysql
        enabled: yes
        state: started
      become: yes
    - name: Install required software
      apt: name={{ item }} state=present
      with_items:
        - default-libmysqlclient-dev
        - python-pip
      become: true
    - name: install MySQL-python using pip
      pip:
        name: "{{ item }}"
        state: forcereinstall
      with_items:
        - pip
        - MySQL-python
      become: true
    - name: create db
      mysql_db:
        name={{ dbname }}
        state=present
        encoding=utf8
        login_user=root
      become: true
    - name: create and grant a database user
      mysql_user:
        name={{ dbuser }}
        password={{ dbpassword }}
        priv="test.*:ALL"
        state=present
      become: true
    - name: download connectorJ
      get_url:
        url: https://downloads.mariadb.com/Connectors/java/connector-java-2.2.6/mariadb-java-client-2.2.6.jar
        dest: /usr/share/java
      become: true
    - name: configure limits.conf
      blockinfile:
        dest: /etc/security/limits.conf
        create: yes
        block: |
          *  hard  nofile  50000
          *  soft  nofile  50000
          *  hard  nproc  10000
          *  soft  nproc  10000
      become: true
    - name: Install openjdk
      apt: name={{ item }} state=present
      with_items:
        - openjdk-8-jdk
      become: true
    - name: download nifi
      get_url:
        url: http://ftp.riken.jp/net/apache/nifi/1.7.1/nifi-1.7.1-bin.tar.gz
        dest: /opt
      become: true
    - name: extract nifi
      unarchive:
        remote_src: yes
        src: /opt/nifi-1.7.1-bin.tar.gz
        dest: /opt/
      become: true
    - name: setup systemd
      blockinfile:
        dest: /etc/systemd/system/nifi.service
        create: yes
        block: |
          [Unit]
          Description=Apache Nifi
          After=syslog.target network.target
          [Service]
          Type=forking
          ExecStart=/opt/nifi-1.7.1/bin/nifi.sh start
          ExecStop=/opt/nifi-1.7.1/bin/nifi.sh stop
          KillMode=none
          [Install]
          WantedBy=multi-user.target
      become: true
    - name: clean up
      file:
        state: absent
        path: /opt/nifi-1.7.1-bin.tar.gz
      become: yes
    - name: enable and start nifi.service
      systemd:
        daemon_reload: yes
        enabled: yes
        state: started
        name: nifi.service
      become: true

4.DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのMariaDBにアクセスします。
Database Connection -> URL jdbc:mariadb://localhost/test
Database Driver Class Name -> org.mariadb.jdbc.Driver
Database Driver Location(s) -> /usr/share/java/mariadb-java-client-2.2.6.jar
Database Userr -> test
Password -> test

VagrantでApache Nifi 1.7.1とMariaDBがインストールされた仮想マシン(CentOS7.5)を構築する。

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache Nifiの画面

〇構築方法
1.以下のVagrantfileを使用して、Apache Nifi 1.7.1とMariaDBがインストールされた仮想マシン(CentOS7.5)を構築します。

Vagrantfile

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "bento/centos-7.5"
  config.vm.hostname = "co75nifi171mariadb"
  config.vm.provider :virtualbox do |vbox|
     vbox.name = "co75nifi171mariadb"
     vbox.cpus = 4
     vbox.memory = 8192
     vbox.customize ["modifyvm", :id, "--nicpromisc2","allow-all"]
  end
  # private network
config.vm.network "private_network", ip: "192.168.55.114", :netmask => "255.255.255.0"
  # bridge netwrok
config.vm.network "public_network", ip: "192.168.1.114", :netmask => "255.255.255.0"
config.vm.network "forwarded_port", guest:22, host:18022, id:"ssh"
  config.vm.provision "shell", inline: <<-SHELL
localectl set-locale LANG=ja_JP.UTF-8
timedatectl set-timezone Asia/Tokyo

# install mariadb
yum -y install mariadb mariadb-server
sudo systemctl enable mariadb.service
sudo systemctl start mariadb.service
mysql -uroot -e "SET PASSWORD = PASSWORD('root'); FLUSH PRIVILEGES;"
mysql -uroot -proot -e "CREATE DATABASE test DEFAULT CHARACTER SET utf8;"
mysql -uroot -proot -e "CREATE USER test@localhost IDENTIFIED BY 'test';"
mysql -uroot -proot -e "GRANT ALL PRIVILEGES ON test.* TO 'test'@'localhost';"
mysql -uroot -proot -e "FLUSH PRIVILEGES;"


# maximum file handles & maximum forked processes
echo '*  hard  nofile  50000' >> /etc/security/limits.conf
echo '*  soft  nofile  50000' >> /etc/security/limits.conf
echo '*  hard  nproc  10000' >> /etc/security/limits.conf
echo '*  soft  nproc  10000' >> /etc/security/limits.conf

echo '*  soft  nproc  10000' >> /etc/security/limits.d/90-nproc

# install openjdk and JDBC driver
yum -y install java-1.8.0-openjdk

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.12-1.el7.noarch.rpm
rpm -i mysql-connector-java-8.0.12-1.el7.noarch.rpm

# download and install Apache Nifi
wget http://ftp.riken.jp/net/apache/nifi/1.7.1/nifi-1.7.1-bin.tar.gz
tar xvfz nifi-1.7.1-bin.tar.gz
mv nifi-1.7.1 /opt
echo "" >> /opt/nifi-1.7.1/bin/nifi-env.sh
echo "JAVA_HOME=/usr/lib/jvm/jre-1.8.0" >> /opt/nifi-1.7.1/bin/nifi-env.sh

cat << EOF > /etc/systemd/system/nifi.service
[Unit]
Description=Apache Nifi
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/opt/nifi-1.7.1/bin/nifi.sh start
ExecStop=/opt/nifi-1.7.1/bin/nifi.sh stop
KillMode=none

[Install]
WantedBy=multi-user.target
EOF
systemctl enable nifi.service
systemctl start nifi.service


echo 'access url -> http://192.168.1.114:8080/nifi/'
SHELL
end

2.DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのMariaDBにアクセスします。
Database Connection -> URL jdbc:mysql://localhost/test
Database Driver Class Name -> com.mysql.jdbc.Driver
Database Driver Location(s) -> /usr/share/java/mysql-connector-java-8.0.12.jar
Database Userr -> test
Password -> test

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

AnsibleでApache NiFiとMariaDBをインストールする(Ubuntu18.04用)

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache NiFiの画面

〇インストール方法
1.下準備
~/.ansible.cfgに以下の内容を設定します

[ssh_connection]
pipelining=True
[defaults]
host_key_checking = False

2. インストール対象ホストのユーザやパスワードを環境に合わせてinventoryファイルに記入します

inventory例

[nifi-mariadb]
192.168.1.61

[all:vars]
ansible_ssh_port=22
ansible_ssh_user=vagrant
ansible_ssh_pass=vagrant
ansible_sudo_pass=vagrant

3.inventory、kanboard.ymlを準備し、以下のコマンドを実行します。

ansible-playbook -i inventory nifi-mariadb.yml

nifi-mariadb.yml

- hosts: nifi-mariadb
  vars:
    - dbname: test
    - dbuser: test
    - dbpassword: test
    - dbhost: localhost
    - dbport: 3306
  tasks:
    - name: set password for root
      shell: echo "mariadb-server-10.1 mysql-server/root_password password root" | sudo debconf-set-selections
      become: yes
    - name: set password for root
      shell: echo "mariadb-server-10.1 mysql-server/root_password_again password root" | sudo debconf-set-selections
      become: yes
    - name: install mariadb
      apt:
        name: mariadb-server
        state: present
      become: yes
    - name: enable and start mariadb
      systemd:
        name: mysql
        enabled: yes
        state: started
      become: yes
    - name: Install required software
      apt: name={{ item }} state=present
      with_items:
        - libmysqlclient-dev
        - python-pip
      become: true
    - name: install MySQL-python using pip
      pip:
        name: "{{ item }}"
        state: forcereinstall
      with_items:
        - pip
        - MySQL-python
      become: true
    - name: create db
      mysql_db:
        name={{ dbname }}
        state=present
        encoding=utf8
        login_user=root
      become: true
    - name: create and grant a database user
      mysql_user:
        name={{ dbuser }}
        password={{ dbpassword }}
        priv="test.*:ALL"
        state=present
      become: true
    - name: download connectorJ
      get_url:
        url: https://downloads.mariadb.com/Connectors/java/connector-java-2.2.6/mariadb-java-client-2.2.6.jar
        dest: /usr/share/java
      become: true
    - name: configure limits.conf
      blockinfile:
        dest: /etc/security/limits.conf
        create: yes
        block: |
          *  hard  nofile  50000
          *  soft  nofile  50000
          *  hard  nproc  10000
          *  soft  nproc  10000
      become: true
    - name: Install openjdk
      apt: name={{ item }} state=present
      with_items:
        - openjdk-8-jdk
      become: true
    - name: download nifi
      get_url:
        url: http://ftp.riken.jp/net/apache/nifi/1.7.1/nifi-1.7.1-bin.tar.gz
        dest: /opt
      become: true
    - name: extract nifi
      unarchive:
        remote_src: yes
        src: /opt/nifi-1.7.1-bin.tar.gz
        dest: /opt/
      become: true
    - name: setup systemd
      blockinfile:
        dest: /etc/systemd/system/nifi.service
        create: yes
        block: |
          [Unit]
          Description=Apache Nifi
          After=syslog.target network.target
          [Service]
          Type=forking
          ExecStart=/opt/nifi-1.7.1/bin/nifi.sh start
          ExecStop=/opt/nifi-1.7.1/bin/nifi.sh stop
          KillMode=none
          [Install]
          WantedBy=multi-user.target
      become: true
    - name: clean up
      file:
        state: absent
        path: /opt/nifi-1.7.1-bin.tar.gz
      become: yes
    - name: enable and start nifi.service
      systemd:
        daemon_reload: yes
        enabled: yes
        state: started
        name: nifi.service
      become: true

4.DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのMariaDBにアクセスします。
Database Connection -> URL jdbc:mariadb://localhost/test
Database Driver Class Name -> org.mariadb.jdbc.Driver
Database Driver Location(s) -> /usr/share/java/maradb-java-client-2.2.6.jar
Database Userr -> test
Password -> test

VagrantでApache Nifi 1.7.1とMariaDBがインストールされた仮想マシン(Ubuntu18.04)を構築する。

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache Nifiの画面

〇構築方法
1.以下のVagrantfileを使用して、Apache Nifi 1.7.1とMariaDBがインストールされた仮想マシン(Ubuntu18.04)を構築します。

Vagrantfile

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "bento/ubuntu-18.04"
  config.vm.hostname = "ub1804nifi171mariadb"
  config.vm.provider :virtualbox do |vbox|
     vbox.name = "ub1804nifi171mariadb"
     vbox.cpus = 2
     vbox.memory = 2048
     vbox.customize ["modifyvm", :id, "--nicpromisc2","allow-all"]
  end
config.vm.network "private_network", ip: "192.168.55.114", :netmask => "255.255.255.0"
config.vm.network "public_network", ip:"192.168.1.114", :netmask => "255.255.255.0"
  config.vm.provision "shell", inline: <<-SHELL
sed -i -e 's/# ja_JP.UTF-8 UTF-8/ja_JP.UTF-8 UTF-8/' /etc/locale.gen
locale-gen
localectl set-locale LANG=ja_JP.UTF-8
localectl set-keymap jp106
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade
timedatectl set-timezone Asia/Tokyo

# install mariadb
echo "mariadb-server-10.0 mysql-server/root_password password root" | sudo debconf-set-selections
echo "mariadb-server-10.0 mysql-server/root_password_again password root" | sudo debconf-set-selections
apt-get -y install mariadb-server
mysql -uroot -proot -e "CREATE DATABASE test DEFAULT CHARACTER SET utf8;"
mysql -uroot -proot -e "CREATE USER test@localhost IDENTIFIED BY 'test';"
mysql -uroot -proot -e "GRANT ALL PRIVILEGES ON test.* TO 'test'@'localhost';"
mysql -uroot -proot -e "FLUSH PRIVILEGES;"


# install jdbc driver for mysql
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java_8.0.12-1ubuntu18.04_all.deb
dpkg -i mysql-connector-java_8.0.12-1ubuntu18.04_all.deb

# maximum file handles & maximum forked processes
echo '*  hard  nofile  50000' >> /etc/security/limits.conf
echo '*  soft  nofile  50000' >> /etc/security/limits.conf
echo '*  hard  nproc  10000' >> /etc/security/limits.conf
echo '*  soft  nproc  10000' >> /etc/security/limits.conf

echo '*  soft  nproc  10000' >> /etc/security/limits.d/90-nproc

# install java
apt-get -y install openjdk-8-jdk

# download and install Apache Nifi
wget http://ftp.riken.jp/net/apache/nifi/1.7.1/nifi-1.7.1-bin.tar.gz
tar xvfz nifi-1.7.1-bin.tar.gz
mv nifi-1.7.1 /opt

cat << EOF > /etc/systemd/system/nifi.service
[Unit]
Description=Apache Nifi
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/opt/nifi-1.7.1/bin/nifi.sh start
ExecStop=/opt/nifi-1.7.1/bin/nifi.sh stop
KillMode=none

[Install]
WantedBy=multi-user.target
EOF
systemctl enable nifi.service
systemctl start nifi.service

echo 'access url -> http://192.168.1.114:8080/nifi/'
SHELL
end

2.DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのMariaDBにアクセスします。
Database Connection -> URL jdbc:mysql://localhost/test
Database Driver Class Name -> com.mysql.jdbc.Driver
Database Driver Location(s) -> /usr/share/java/mysql-connector-java-8.0.12.jar
Database Userr -> test
Password -> test

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

VagrantでApache Nifi 1.7.1とMariaDBがインストールされた仮想マシン(Ubuntu16.04)を構築する。

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache Nifiの画面

〇構築方法
1.以下のVagrantfileを使用して、Apache Nifi 1.7.1とMariaDBがインストールされた仮想マシン(Ubuntu16.04)を構築します。

Vagrantfile

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "bento/ubuntu-16.04"
  config.vm.hostname = "ub1604nifi171mariadb"
  config.vm.provider :virtualbox do |vbox|
     vbox.name = "ub1604nifi171mariadb"
     vbox.cpus = 2
     vbox.memory = 2048
     vbox.customize ["modifyvm", :id, "--nicpromisc2","allow-all"]
  end
config.vm.network "private_network", ip: "192.168.55.114", :netmask => "255.255.255.0"
config.vm.network "public_network", ip:"192.168.1.114", :netmask => "255.255.255.0"
  config.vm.provision "shell", inline: <<-SHELL
sed -i -e 's/# ja_JP.UTF-8 UTF-8/ja_JP.UTF-8 UTF-8/' /etc/locale.gen
locale-gen
localectl set-locale LANG=ja_JP.UTF-8
localectl set-keymap jp106
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade
timedatectl set-timezone Asia/Tokyo

# install mariadb
echo "mariadb-server-10.0 mysql-server/root_password password root" | sudo debconf-set-selections
echo "mariadb-server-10.0 mysql-server/root_password_again password root" | sudo debconf-set-selections
apt-get -y install mariadb-server
mysql -uroot -proot -e "CREATE DATABASE test DEFAULT CHARACTER SET utf8;"
mysql -uroot -proot -e "CREATE USER test@localhost IDENTIFIED BY 'test';"
mysql -uroot -proot -e "GRANT ALL PRIVILEGES ON test.* TO 'test'@'localhost';"
mysql -uroot -proot -e "FLUSH PRIVILEGES;"


# install jdbc driver for mysql
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java_8.0.12-1ubuntu16.04_all.deb
dpkg -i mysql-connector-java_8.0.12-1ubuntu16.04_all.deb

# maximum file handles & maximum forked processes
echo '*  hard  nofile  50000' >> /etc/security/limits.conf
echo '*  soft  nofile  50000' >> /etc/security/limits.conf
echo '*  hard  nproc  10000' >> /etc/security/limits.conf
echo '*  soft  nproc  10000' >> /etc/security/limits.conf

echo '*  soft  nproc  10000' >> /etc/security/limits.d/90-nproc

# install java
apt-get -y install openjdk-8-jdk

# download and install Apache Nifi
wget http://ftp.riken.jp/net/apache/nifi/1.7.1/nifi-1.7.1-bin.tar.gz
tar xvfz nifi-1.7.1-bin.tar.gz
mv nifi-1.7.1 /opt

cat << EOF > /etc/systemd/system/nifi.service
[Unit]
Description=Apache Nifi
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/opt/nifi-1.7.1/bin/nifi.sh start
ExecStop=/opt/nifi-1.7.1/bin/nifi.sh stop
KillMode=none

[Install]
WantedBy=multi-user.target
EOF
systemctl enable nifi.service
systemctl start nifi.service

echo 'access url -> http://192.168.1.114:8080/nifi/'
SHELL
end

2.DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのMariaDBにアクセスします。
Database Connection -> URL jdbc:mysql://localhost/test
Database Driver Class Name -> com.mysql.jdbc.Driver
Database Driver Location(s) -> /usr/share/java/mysql-connector-java-8.0.12.jar
Database Userr -> test
Password -> test

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

VagrantでApache Nifi 1.7.1とMariaDBがインストールされた仮想マシン(Debian Stretch/9.5)を構築する

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache Nifiの画面

〇構築方法
1.以下のVagrantfileを使用して、Apache Nifi 1.7.1とMariaDBがインストールされた仮想マシン(Debian Stretch/9.5)を構築します。

Vagrantfile

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "bento/debian-9.5"
  config.vm.hostname = "db95nifi171mariadb"
  config.vm.provider :virtualbox do |vbox|
     vbox.name = "db94nifi160mariadb"
     vbox.cpus = 2
     vbox.memory = 2048
     vbox.customize ["modifyvm", :id, "--nicpromisc2","allow-all"]
  end
config.vm.network "private_network", ip: "192.168.55.114", :netmask => "255.255.255.0"
config.vm.network "public_network", ip:"192.168.1.114", :netmask => "255.255.255.0"
  config.vm.provision "shell", inline: <<-SHELL
apt-get -y install task-japanese
sed -i -e 's/# ja_JP.UTF-8 UTF-8/ja_JP.UTF-8 UTF-8/' /etc/locale.gen
locale-gen
update-locale LANG=ja_JP.UTF-8
localectl set-locale LANG=ja_JP.UTF-8
localectl set-keymap jp106
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade
timedatectl set-timezone Asia/Tokyo

# install mariadb
echo "mariadb-server-10.0 mysql-server/root_password password root" | sudo debconf-set-selections
echo "mariadb-server-10.0 mysql-server/root_password_again password root" | sudo debconf-set-selections
apt-get -y install mariadb-server
mysql -uroot -proot -e "CREATE DATABASE test DEFAULT CHARACTER SET utf8;"
mysql -uroot -proot -e "CREATE USER test@localhost IDENTIFIED BY 'test';"
mysql -uroot -proot -e "GRANT ALL PRIVILEGES ON test.* TO 'test'@'localhost';"
mysql -uroot -proot -e "FLUSH PRIVILEGES;"


# install jdbc driver for mysql
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java_8.0.11-1debian9_all.deb
dpkg -i mysql-connector-java_8.0.11-1debian9_all.deb

# maximum file handles & maximum forked processes
echo '*  hard  nofile  50000' >> /etc/security/limits.conf
echo '*  soft  nofile  50000' >> /etc/security/limits.conf
echo '*  hard  nproc  10000' >> /etc/security/limits.conf
echo '*  soft  nproc  10000' >> /etc/security/limits.conf

echo '*  soft  nproc  10000' >> /etc/security/limits.d/90-nproc

# install java
apt-get -y install openjdk-8-jdk

# download and install Apache Nifi
wget http://ftp.riken.jp/net/apache/nifi/1.7.1/nifi-1.7.1-bin.tar.gz
tar xvfz nifi-1.7.1-bin.tar.gz
mv nifi-1.7.1 /opt

cat << EOF > /etc/systemd/system/nifi.service
[Unit]
Description=Apache Nifi
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/opt/nifi-1.7.1/bin/nifi.sh start
ExecStop=/opt/nifi-1.7.1/bin/nifi.sh stop
KillMode=none

[Install]
WantedBy=multi-user.target
EOF
systemctl enable nifi.service
systemctl start nifi.service

echo 'access url -> http://192.168.1.114:8080/nifi/'
SHELL
end

2.DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのMariaDBにアクセスします。
Database Connection -> URL jdbc:mysql://localhost/test
Database Driver Class Name -> com.mysql.jdbc.Driver
Database Driver Location(s) -> /usr/share/java/mysql-connector-java-8.0.11.jar
Database Userr -> test
Password -> test

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

VagrantでApache Nifi 1.6.0とPercona Serverがインストールされた仮想マシン(Debian Stretch/9.4)を構築する

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache Nifiの画面

〇構築方法
1.以下のVagrantfileを使用して、Apache Nifi 1.6.0とPercona Serverがインストールされた仮想マシン(Debian Stretch/9.4)を構築します。

Vagrantfile

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "bento/debian-9.4"
  config.vm.hostname = "db94nifi160percona"
  config.vm.provider :virtualbox do |vbox|
     vbox.name = "db94nifi160percona"
     vbox.cpus = 2
     vbox.memory = 2048
     vbox.customize ["modifyvm", :id, "--nicpromisc2","allow-all"]
  end
config.vm.network "private_network", ip: "192.168.55.104", :netmask => "255.255.255.0"
config.vm.network "public_network", ip:"192.168.1.104", :netmask => "255.255.255.0"
  config.vm.provision "shell", inline: <<-SHELL
apt-get -y install task-japanese
sed -i -e 's/# ja_JP.UTF-8 UTF-8/ja_JP.UTF-8 UTF-8/' /etc/locale.gen
locale-gen
update-locale LANG=ja_JP.UTF-8
localectl set-locale LANG=ja_JP.UTF-8
localectl set-keymap jp106
apt-get update
#DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade

# install percona server
export DEBIAN_FRONTEND=noninteractive
echo "percona-server-server-5.7 percona-server-server/root_password password root" | sudo debconf-set-selections
echo "percona-server-server-5.7 percona-server-server/root_password_again password root" | sudo debconf-set-selections
wget https://repo.percona.com/apt/percona-release_0.1-4.$(lsb_release -sc)_all.deb
dpkg -i percona-release_0.1-4.$(lsb_release -sc)_all.deb
apt-get update
apt-get -y install percona-server-server-5.7
mysql -uroot -proot -e "CREATE DATABASE test DEFAULT CHARACTER SET utf8;"
mysql -uroot -proot -e "CREATE USER test@localhost IDENTIFIED BY 'test';"
mysql -uroot -proot -e "GRANT ALL PRIVILEGES ON test.* TO 'test'@'localhost';"
mysql -uroot -proot -e "FLUSH PRIVILEGES;"


# install jdbc driver for mysql
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java_8.0.11-1debian9_all.deb
dpkg -i mysql-connector-java_8.0.11-1debian9_all.deb

# maximum file handles & maximum forked processes
echo '*  hard  nofile  50000' >> /etc/security/limits.conf
echo '*  soft  nofile  50000' >> /etc/security/limits.conf
echo '*  hard  nproc  10000' >> /etc/security/limits.conf
echo '*  soft  nproc  10000' >> /etc/security/limits.conf

echo '*  soft  nproc  10000' >> /etc/security/limits.d/90-nproc

# install java
apt-get -y install openjdk-8-jdk

# download and install Apache Nifi
wget http://ftp.riken.jp/net/apache/nifi/1.6.0/nifi-1.6.0-bin.tar.gz
tar xvfz nifi-1.6.0-bin.tar.gz
mv nifi-1.6.0 /opt

cat << EOF > /etc/systemd/system/nifi.service
[Unit]
Description=Apache Nifi
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/opt/nifi-1.6.0/bin/nifi.sh start
ExecStop=/opt/nifi-1.6.0/bin/nifi.sh stop
KillMode=none

[Install]
WantedBy=multi-user.target
EOF
systemctl enable nifi.service
systemctl start nifi.service

echo 'access url -> http://192.168.1.104:8080/nifi/'
SHELL
end

2.DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのPercona Serverにアクセスします。
Database Connection -> URL jdbc:mysql://localhost/test
Database Driver Class Name -> com.mysql.jdbc.Driver
Database Driver Location(s) -> /usr/share/java/mysql-connector-java-8.0.11.jar
Database Userr -> test
Password -> test

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

VagrantでApache Nifi 1.6.0とMariaDBがインストールされた仮想マシン(Debian Stretch/9.4)を構築する。

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache Nifiの画面

〇構築方法
1.以下のVagrantfileを使用して、Apache Nifi 1.6.0とMariaDBがインストールされた仮想マシン(Debian Stretch/9.4)を構築します。

Vagrantfile

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "bento/debian-9.4"
  config.vm.hostname = "db94nifi160mariadb"
  config.vm.provider :virtualbox do |vbox|
     vbox.name = "db94nifi160mariadb"
     vbox.cpus = 2
     vbox.memory = 2048
     vbox.customize ["modifyvm", :id, "--nicpromisc2","allow-all"]
  end
config.vm.network "private_network", ip: "192.168.55.104", :netmask => "255.255.255.0"
config.vm.network "public_network", ip:"192.168.1.104", :netmask => "255.255.255.0"
  config.vm.provision "shell", inline: <<-SHELL
apt-get -y install task-japanese
sed -i -e 's/# ja_JP.UTF-8 UTF-8/ja_JP.UTF-8 UTF-8/' /etc/locale.gen
locale-gen
update-locale LANG=ja_JP.UTF-8
localectl set-locale LANG=ja_JP.UTF-8
localectl set-keymap jp106
apt-get update
#DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade

# install mariadb
echo "mariadb-server-10.3 mysql-server/root_password password root" | sudo debconf-set-selections
echo "mariadb-server-10.3 mysql-server/root_password_again password root" | sudo debconf-set-selections
apt-get -y install mariadb-server
mysql -uroot -proot -e "CREATE DATABASE test DEFAULT CHARACTER SET utf8;"
mysql -uroot -proot -e "CREATE USER test@localhost IDENTIFIED BY 'test';"
mysql -uroot -proot -e "GRANT ALL PRIVILEGES ON test.* TO 'test'@'localhost';"
mysql -uroot -proot -e "FLUSH PRIVILEGES;"


# install jdbc driver for mysql
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java_8.0.11-1debian9_all.deb
dpkg -i mysql-connector-java_8.0.11-1debian9_all.deb

# maximum file handles & maximum forked processes
echo '*  hard  nofile  50000' >> /etc/security/limits.conf
echo '*  soft  nofile  50000' >> /etc/security/limits.conf
echo '*  hard  nproc  10000' >> /etc/security/limits.conf
echo '*  soft  nproc  10000' >> /etc/security/limits.conf

echo '*  soft  nproc  10000' >> /etc/security/limits.d/90-nproc

# install java
apt-get -y install openjdk-8-jdk

# download and install Apache Nifi
wget http://ftp.riken.jp/net/apache/nifi/1.6.0/nifi-1.6.0-bin.tar.gz
tar xvfz nifi-1.6.0-bin.tar.gz
mv nifi-1.6.0 /opt

cat << EOF > /etc/systemd/system/nifi.service
[Unit]
Description=Apache Nifi
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/opt/nifi-1.6.0/bin/nifi.sh start
ExecStop=/opt/nifi-1.6.0/bin/nifi.sh stop
KillMode=none

[Install]
WantedBy=multi-user.target
EOF
systemctl enable nifi.service
systemctl start nifi.service

echo 'access url -> http://192.168.1.104:8080/nifi/'
SHELL
end

2.DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのMariaDBにアクセスします。
Database Connection -> URL jdbc:mysql://localhost/test
Database Driver Class Name -> com.mysql.jdbc.Driver
Database Driver Location(s) -> /usr/share/java/mysql-connector-java-8.0.11.jar
Database Userr -> test
Password -> test

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

VagrantでApache Nifi 1.6.0とmosquittoがインストールされた仮想マシン(Debian Stretch/9.4)を構築する。

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache Nifiの画面

ブラウザからhttp://192.168.1.105:8080/nifi/にアクセスします。

〇構築方法
1.以下のVagrantfileを使用して、Apache Nifi 1.6.0とmosquittoがインストールされた仮想マシン(Debian Stretch/9.4)を構築します。

Vagrantfile

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "bento/debian-9.4"
  config.vm.hostname = "db94nifi160mosquitto"
  config.vm.provider :virtualbox do |vbox|
     vbox.name = "db94nifi160mosquitto"
     vbox.cpus = 2
     vbox.memory = 2048
     vbox.customize ["modifyvm", :id, "--nicpromisc2","allow-all"]
  end
config.vm.network "private_network", ip: "192.168.55.105", :netmask => "255.255.255.0"
config.vm.network "public_network", ip:"192.168.1.105", :netmask => "255.255.255.0"
  config.vm.provision "shell", inline: <<-SHELL
apt-get -y install task-japanese
sed -i -e 's/# ja_JP.UTF-8 UTF-8/ja_JP.UTF-8 UTF-8/' /etc/locale.gen
locale-gen
update-locale LANG=ja_JP.UTF-8
localectl set-locale LANG=ja_JP.UTF-8
localectl set-keymap jp106
apt-get update
#DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade

# install mosquitto
apt-get -y install mosquitto mosquitto-clients


# maximum file handles & maximum forked processes
echo '*  hard  nofile  50000' >> /etc/security/limits.conf
echo '*  soft  nofile  50000' >> /etc/security/limits.conf
echo '*  hard  nproc  10000' >> /etc/security/limits.conf
echo '*  soft  nproc  10000' >> /etc/security/limits.conf

echo '*  soft  nproc  10000' >> /etc/security/limits.d/90-nproc

# install java
apt-get -y install openjdk-8-jdk

# download and install Apache Nifi
wget http://ftp.riken.jp/net/apache/nifi/1.6.0/nifi-1.6.0-bin.tar.gz
tar xvfz nifi-1.6.0-bin.tar.gz
mv nifi-1.6.0 /opt

cat << EOF > /etc/systemd/system/nifi.service
[Unit]
Description=Apache Nifi
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/opt/nifi-1.6.0/bin/nifi.sh start
ExecStop=/opt/nifi-1.6.0/bin/nifi.sh stop
KillMode=none

[Install]
WantedBy=multi-user.target
EOF
systemctl enable nifi.service
systemctl start nifi.service

echo 'access url -> http://192.168.1.105:8080/nifi/'

echo 'execute command below for test.'
echo 'mosquitto_pub -t mytopic/test -h localhost -m "test message."'

SHELL
end

2.ConsumeMQTT Processorで、以下のようにパラメータを設定してローカルのmosquittoにアクセスします。
Broker URI : tcp://localhost:1883
Client ID : nifi
Topi Filter : mytopic/test
Quality of Service(QoS) : 0 - At most once
Max Queue Size : 100

※ローカルのmosquittoにメッセージをpublishするには、以下のようなコマンドを実行します。

mosquitto_pub -t mytopic/test -h localhost -m "test message."

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。


○関連情報
・mosquittoに関する他の記事はこちらを参照してください。

VagrantでApache Nifi 1.6.0とApache Cassandraがインストールされた仮想マシン(Debian Stretch/9.4)を構築する。

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache Nifiの画面

〇構築方法
1.以下のVagrantfileを使用して、Apache Nifi 1.6.0とApache Cassandraがインストールされた仮想マシン(Debian Stretch/9.4)を構築します。

Vagrantfile

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "bento/debian-9.4"
  config.vm.hostname = "db94nifi160cassandra"
  config.vm.provider :virtualbox do |vbox|
     vbox.name = "db94nifi160cassandra"
     vbox.cpus = 2
     vbox.memory = 2048
     vbox.customize ["modifyvm", :id, "--nicpromisc2","allow-all"]
  end
config.vm.network "private_network", ip: "192.168.55.105", :netmask => "255.255.255.0"
config.vm.network "public_network", ip:"192.168.1.105", :netmask => "255.255.255.0"
  config.vm.provision "shell", inline: <<-SHELL
apt-get -y install task-japanese
sed -i -e 's/# ja_JP.UTF-8 UTF-8/ja_JP.UTF-8 UTF-8/' /etc/locale.gen
locale-gen
update-locale LANG=ja_JP.UTF-8
localectl set-locale LANG=ja_JP.UTF-8
localectl set-keymap jp106
apt-get update
#DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade

apt-get -y install curl
echo "deb http://www.apache.org/dist/cassandra/debian 311x main" >> /etc/apt/sources.list.d/cassandra.sources.list
curl https://www.apache.org/dist/cassandra/KEYS | apt-key add -
apt-get update
apt-get -y install cassandra

# configure authentication.
sed -i -e 's/authenticator: AllowAllAuthenticator/authenticator: PasswordAuthenticator/' /etc/cassandra/cassandra.yaml
systemctl enable cassandra.service
systemctl start cassandra.service

# CQL for test
cat << EOF > /home/vagrant/sample.cql
create keyspace mykeyspace with replication = {'class':'SimpleStrategy', 'replication_factor':1};
use mykeyspace;
create table mytable (
name text PRIMARY KEY,
value text
);
insert into mytable (name, value) values ('test1', 'cassandra');
select * from mytable;
EOF

# wait until it listens on 9042 port.
while netstat -lnt | awk '$4 ~ /:9042$/ {exit 1}'; do sleep 10; done
sleep 10

cqlsh -u cassandra -p cassandra -f /home/vagrant/sample.cql


# maximum file handles & maximum forked processes
echo '*  hard  nofile  50000' >> /etc/security/limits.conf
echo '*  soft  nofile  50000' >> /etc/security/limits.conf
echo '*  hard  nproc  10000' >> /etc/security/limits.conf
echo '*  soft  nproc  10000' >> /etc/security/limits.conf

echo '*  soft  nproc  10000' >> /etc/security/limits.d/90-nproc

# download and install Apache Nifi
wget http://ftp.riken.jp/net/apache/nifi/1.6.0/nifi-1.6.0-bin.tar.gz
tar xvfz nifi-1.6.0-bin.tar.gz
mv nifi-1.6.0 /opt

cat << EOF > /etc/systemd/system/nifi.service
[Unit]
Description=Apache Nifi
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/opt/nifi-1.6.0/bin/nifi.sh start
ExecStop=/opt/nifi-1.6.0/bin/nifi.sh stop
KillMode=none

[Install]
WantedBy=multi-user.target
EOF
systemctl enable nifi.service
systemctl start nifi.service

echo 'access url -> http://192.168.1.105:8080/nifi/'

SHELL
end

2.Query Cassandra Processorで、以下のようにパラメータを設定してローカルのCassandraにアクセスします。
Cassandra Constact Points : localhost:9042
Keyspace : mykeyspace
Username : cassandra
Password : cassandra
CQL select query : select name, value from mytable

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

・Apache Cassandraに関する他の記事はこちらを参照してください。

VagrantでApache Nifi 1.6.0とMySQLがインストールされた仮想マシン(Debian Stretch/9.4)を構築する。

Apache Nifiは様々なデータを処理・分配するためのソフトウェアです。

〇Apache Nifiの画面

〇構築方法
1.以下のVagrantfileを使用して、Apache Nifi 1.6.0とMySQLがインストールされた仮想マシン(Debian Stretch/9.4)を構築します。

Vagrantfile

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "bento/debian-9.4"
  config.vm.hostname = "db94nifi160mysql"
  config.vm.provider :virtualbox do |vbox|
     vbox.name = "db94nifi160mysql"
     vbox.cpus = 2
     vbox.memory = 2048
     vbox.customize ["modifyvm", :id, "--nicpromisc2","allow-all"]
  end
config.vm.network "private_network", ip: "192.168.55.104", :netmask => "255.255.255.0"
config.vm.network "public_network", ip:"192.168.1.104", :netmask => "255.255.255.0"
  config.vm.provision "shell", inline: <<-SHELL
apt-get -y install task-japanese
sed -i -e 's/# ja_JP.UTF-8 UTF-8/ja_JP.UTF-8 UTF-8/' /etc/locale.gen
locale-gen
update-locale LANG=ja_JP.UTF-8
localectl set-locale LANG=ja_JP.UTF-8
localectl set-keymap jp106
apt-get update
#DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade

# install mysql
wget https://dev.mysql.com/get/mysql-apt-config_0.8.9-1_all.deb
export DEBIAN_FRONTEND=noninteractive
echo mysql-apt-config mysql-apt-config/enable-repo select mysql-5.7-dmr | sudo debconf-set-selections
echo "mysql-community-server mysql-community-server/root-pass password root"| debconf-set-selections
echo "mysql-community-server mysql-community-server/re-root-pass password root" | debconf-set-selections
dpkg -i mysql-apt-config_0.8.9-1_all.deb
apt-get update
apt-get -y install mysql-server
mysql -uroot -proot -e "CREATE DATABASE test DEFAULT CHARACTER SET utf8;"
mysql -uroot -proot -e "CREATE USER test@localhost IDENTIFIED BY 'test';"
mysql -uroot -proot -e "GRANT ALL PRIVILEGES ON test.* TO 'test'@'localhost';"
mysql -uroot -proot -e "FLUSH PRIVILEGES;"


# install jdbc driver for mysql
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java_8.0.11-1debian9_all.deb
dpkg -i mysql-connector-java_8.0.11-1debian9_all.deb

# maximum file handles & maximum forked processes
echo '*  hard  nofile  50000' >> /etc/security/limits.conf
echo '*  soft  nofile  50000' >> /etc/security/limits.conf
echo '*  hard  nproc  10000' >> /etc/security/limits.conf
echo '*  soft  nproc  10000' >> /etc/security/limits.conf

echo '*  soft  nproc  10000' >> /etc/security/limits.d/90-nproc

# install java
apt-get -y install openjdk-8-jdk

# download and install Apache Nifi
wget http://ftp.riken.jp/net/apache/nifi/1.6.0/nifi-1.6.0-bin.tar.gz
tar xvfz nifi-1.6.0-bin.tar.gz
mv nifi-1.6.0 /opt

cat << EOF > /etc/systemd/system/nifi.service
[Unit]
Description=Apache Nifi
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/opt/nifi-1.6.0/bin/nifi.sh start
ExecStop=/opt/nifi-1.6.0/bin/nifi.sh stop
KillMode=none

[Install]
WantedBy=multi-user.target
EOF
systemctl enable nifi.service
systemctl start nifi.service

echo 'access url -> http://192.168.1.104:8080/nifi/'
SHELL
end

2.DBCPConnectionPoolで、以下のようにパラメータを設定してローカルのMySQLにアクセスします。
Database Connection -> URL jdbc:mysql://localhost/test
Database Driver Class Name -> com.mysql.jdbc.Driver
Database Driver Location(s) -> /usr/share/java/mysql-connector-java-8.0.11.jar
Database Userr -> test
Password -> test

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。

Apache NiFiがインストールされた仮想マシン(CentOS7.3)をVagrantで作成する

Apache NiFiがインストールされた仮想マシン(CentOS7.3)をVagrantで作成するには、以下のVagrantfile, nifi.serviceを同一フォルダに配置してvagrant upコマンドを実行します。仮想マシンのビルド完了後、http://192.168.1.88:8080/nifi/でApache NiFiの画面にアクセスできます。 Vagrantfile

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "bento/centos-7.3"
  config.vm.hostname = "nifi"
  config.vm.provider :virtualbox do |vbox|
     vbox.name = "nifi"
     vbox.cpus = 4
     vbox.memory = 8192 
     vbox.customize ["modifyvm", :id, "--nicpromisc2","allow-all"]
  end
  # private network
  config.vm.network "private_network", ip: "192.168.55.88", :netmask => "255.255.255.0"
  # bridge netwrok
  config.vm.network "public_network", ip: "192.168.1.88", :netmask => "255.255.255.0"
  config.vm.network "forwarded_port", guest:22, host:18022, id:"ssh"
  config.vm.provision "shell", inline: <<-SHELL
# 8080ポート
firewall-cmd --permanent --add-port=8080/tcp

# maximum file handles & maximum forked processes
echo '*  hard  nofile  50000' >> /etc/security/limits.conf
echo '*  soft  nofile  50000' >> /etc/security/limits.conf
echo '*  hard  nproc  10000' >> /etc/security/limits.conf
echo '*  soft  nproc  10000' >> /etc/security/limits.conf

echo '*  soft  nproc  10000' >> /etc/security/limits.d/90-nproc

# javaをインストール
yum -y install java-1.8.0-openjdk

# Apache Nifiのダウンロード
wget http://ftp.riken.jp/net/apache/nifi/1.3.0/nifi-1.3.0-bin.tar.gz
tar xvfz nifi-1.3.0-bin.tar.gz
mv nifi-1.3.0 /opt

#サービスとして登録
cp /vagrant/nifi.service /etc/systemd/system
systemctl enable nifi.service
systemctl start nifi.service

echo 'access url -> http://192.168.55.88:8080/nifi/' 

SHELL
end

nifi.service

[Unit]
Description=Apache Nifi
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/opt/nifi-1.3.0/bin/nifi.sh start
ExecStop=/opt/nifi-1.3.0/bin/nifi.sh stop
KillMode=none

[Install]
WantedBy=multi-user.target

○関連情報
・Apache NiFiに関する他の記事はこちらを参照してください。