Software engineering notes

Docker

介紹

有點像 VM 但又不全然是 VM, 最主要它跟一般 VM 的差別在於 VM 所需的 OS 及 bins/libs 是安裝在 VM 本身裡面的

但 docker 的 container 概念會共用 OS 及 bins/libs, 但他們仍是獨立的, 由 Docker Engine 管理

最大好處可以快速啟動環境並且設定好, 很適合用在自動化測試或開發上

基本觀念

名詞解釋

Image

Container

Quick start based on particular scenarios

ngrok: forward port 8080 from inside the Docker container to your local machine

docker run --rm -it -p 4040:4040 ngrok/ngrok http host.docker.internal:8080

Run the local daemon listening port 8080

Then open the domain that ngrok generates for you on browser, the request should be forwarded to local daemon

安裝

mac

可從官網下載

ubuntu

sudo apt-get update
sudo apt-get install docker.io
sudo service docker start

執行 sudo adduser <username> docker, 將你加入到 docker group

windows

下載頁面

Exec : C:\Program Files\Boot2Docker for Windows\start.sh

putty / xshell connect to docker@127.0.0.1:2022

測試是否安裝成功

docker run hello-world

開始試玩

將 mysql 跑起來

Pull image mysql from docker hub

docker pull mysql

Run container mysql

docker run -t -d -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql

Test

mysql -h 127.0.0.1 -u root -p

將 redis 跑起來

Pull Docker Hub 上的 redis image

docker pull redis

將 redis image 啟動成 container, 並且將 6379 port 接出來

docker run -t -d -p 6379:6379 redis:latest

測試用 redis-cli 是否可以連進去

將 ubuntu image 跑起來 (建立一個 container)

Pull Docker Hub 上的 ubuntu image

docker pull ubuntu

以 ubuntu image 建立一個 container

docker run -t -d ubuntu

看看執行中的 container 有哪些，你可以看到剛剛建立的 container

docker ps

CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                               NAMES
bba4d449e053        ubuntu                       "/bin/bash"              52 seconds ago      Up 51 seconds                                           sharp_mestorf

Stop 後 docker ps 就看不到它了

docker stop bba4d449e053

但你顯示全部的 container 就看的到了

docker ps -a

你可以完全地刪除它

docker rm bba4d449e053

How can i run docker command inside a docker container?

在一台 container 裡面安裝 docker 啟動失敗:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

必須要 mount 外部的 docker socket 進去

docker run -td -v /var/run/docker.sock:/var/run/docker.sock ubuntu:18.04
docker exec -it a1b6a80b0bb8 bash

or
docker run -it -v /var/run/docker.sock:/var/run/docker.sock ubuntu:latest sh -c "apt-get update ; apt-get install docker.io -y ; bash"

常用指令

build : 建立寫好的 Dockerfile

docker build -t jxxx/golang-env .

without cache

docker build --no-cache .

images : image list

docker images

rmi : 刪除 image

docker rmi eg_sshd

// 刪除全部 image
docker rmi $(docker images -q)

inspect : 查看此 container 的 info (不常用)

docker inspect f244dc51e1d9

ps : running container list (有哪些主機正在跑)

docker ps

stop : 停掉在跑的 container (相當於關機)

docker stop {CONTAINER ID}

// 停掉全部在跑的 container
docker stop $(docker ps -a -q)

start : 將一個已經被 stop 的 container 再跑起來 (相當於開機)

docker start {CONTAINER ID}

rm : 刪除 container (相當於刪除這台主機)

docker rm {CONTAINER ID}

// 刪除全部 container
docker rm $(docker ps -a -q)

stop vs rm

start vs run

run

以 ubuntu image 建立 container 並進入到 container 裡面 (有點像 SSH 連進去，但 exit 後就是 stopped 的狀態了)

docker run -t -i ubuntu:14.04 /bin/bash

背景執行 container

docker run -t -d ubuntu:14.04

在 container 裡執行指令

docker run jxxx/golang-env touch ff.md

本機與 container 所有的 port

docker run -d -P jxxx/golang-env

-P flag to publish all of the exposed ports.

50080 port -> 80 port

docker run -d -p 50080:80 jxxx/golang-env

-p flag to publish a range of ports

將 domain 對應到 image name

docker run -d --link my_mysql:db.example.com jxxx/golang-env

將目錄名稱 folder_src mount 到 container 家目錄下(?)

docker run -v `pwd`/folder_src:/home/test/into_container jxxx/golang-env ls -al into_container

Exec

進入正在執行中的container

docker exec -it <containerIdOrName> bash

Cp

將外部檔案複製到 container 裡

docker cp foo.txt {CONTAINER ID}:/foo.txt

將檔案從 container 裡複製出來

docker cp {CONTAINER ID}:/foo.txt ./

Commit

docker commit -m "Add ff.md" e3015 jxxx/golang-env

Push

docker push jxxx/golang-env

Attach

Logs

將 container 的標準輸出接出來

docker logs {CONTAINER ID}

在 Docker config 設定將 log 接到標準輸出

RUN ln -sf /dev/stdout /var/log/test.log

container

Remove the stopped containers

docker container prune

其他指令

prune

docker container prune
docker image prune
docker volume prune
docker network prune
docker system prune -a

Docker machine command

docker-machine is used to connect to VirtualBox and create virtual machine

Show version

docker-machine version

Create machine

docker-machine create --driver virtualbox {name}

Show machine list

docker-machine ls

Show environment variables

docker-machine env {name}

SSH into machine

docker-machine ssh {name}

Start machine

docker-machine start {name}

Stop machine

docker-machine {name}

Remove machine

docker-machine {name}

Show machine status

docker-machine status {name}

Dockerfile 怎麼寫

FROM : 使用每個 image 為基礎

FROM ubuntu
FROM ubuntu:14.04

MAINTAINER : 設定產生的 imgage 的 Author

MAINTAINER test "xxxxxxx@gmail.com"

USER : 使用哪個 user 去執行 RUN, CMD etc.

USER daemon

RUN : 執行指令 (使用預設的 shell /bin/sh -c)

RUN apt-get update && apt-get -y upgrade

CMD : 執行 command

CMD echo "This is a test." | wc -       // 預設是使用 shell 執行  /bin/sh -c

CMD ["/usr/bin/wc","--help"]            // 執行其他指令不使用 shell 的話要使用 JSON format

ENV : 設定環境變數

ENV HOME /root

WORKDIR 設定工作目錄 (for any RUN, CMD, ENTRYPOINT, COPY and ADD in the Dockerfile)

WORKDIR /root
或
WORKDIR ${HOME}

ADD : 新增檔案

ADD xx.md /root
ADD xx.md $HOME

ADD test relativeDir/          # adds "test" to `WORKDIR`/relativeDir/
ADD test /absoluteDir/         # adds "test" to /absoluteDir/

COPY : 複製檔案

COPY \$HOME /tmp

COPY test relativeDir/   # adds "test" to `WORKDIR`/relativeDir/
COPY test /absoluteDir/  # adds "test" to /absoluteDir/

EXPOSE : 開放哪些 Port

EXPOSE 80 443

HEALTHCHECK (https://docs.docker.com/engine/reference/builder/#/healthcheck)

HEALTHCHECK --interval=5m --timeout=3s \
  CMD curl -f http://localhost/ || exit 1

SHELL : 更改預設的 shell (linux default : /bin/sh -c)

SHELL ["executable", "parameters"]

# Executed as cmd /S /C echo hello
SHELL ["cmd", "/S"", "/C"]
RUN echo hello

docker-compose

To start and run the app

docker-compose up -d

Build/rebuild the service

docker-compose build

Build/rebuild the service and run the app

docker-compose up --build

As long as Dockerfile is changed, run docker build --no-cache . first

Allow 2 services from different docker-compose config to connect the same service

For example, allow ‘worker’ and ‘api’ connect to the same db server

In order to achieve this, it needs to use docker network

Create the network first

docker network create mynetwork

docker-compose.yml from repo 1

services:
  db:
    image: mysql:latest
    networks:
      - mynetwork
  worker:
    build: .
    depends_on:
      - db
    networks:
      - mynetwork
networks:
  mynetwork:
    external: true

docker-compose.yml from repo 2

services:
  api:
    build: .
    networks:
      - mynetwork
networks:
  mynetwork:
    external: true

Now, these 2 services are able to connect to db (root:@tcp(db:3306)/mytable)

mysql server and client login

docker-compose.yml

version: "3"
services:
  app:
    build: .
    volumes:
      - .:/usr/src/app
    depends_on:
      - db
  db:
    image: mysql:latest
    restart: always
    environment:
      MYSQL_DATABASE: mydb
      MYSQL_ALLOW_EMPTY_PASSWORD: yes
    ports:
      - "3306:3306"
    volumes:
      - ./mysql:/var/lib/mysql
    networks:
      - mynetwork
networks:
  mynetwork:

Run

docker-compose up --build

Find the network name

docker network ls

NETWORK ID     NAME                                     DRIVER    SCOPE
2e31601337aa   myproject_mynetwork                      bridge    local

Connect to mysql server

docker run -it --network myproject_mynetwork mysql:latest mysql -hdb -uroot -p
(no need to enter password)

config for mysql, redis and phpmyadmin

docker-compose.yml

version: "2"
services:

  test-redis:
    container_name: test-redis
    image: redis
    expose:
      - "6379"
    ports:
      - "6379:6379"
    volumes:
      - redis-data-test:/data-test

  test-mysql:
    container_name: test-mysql
    image: mysql/mysql-server:5.7
    expose:
      - "3306"
    ports:
      - "3306:3306"
    environment:
      - MYSQL_ALLOW_EMPTY_PASSWORD=yes
    volumes:
      - dbdata-test:/var/lib/mysql

  phpmyadmin:
    container_name: phpmyadmin
    image: phpmyadmin/phpmyadmin
    links:
      - test-mysql:db
    ports:
      - "8888:80"

volumes:
  redis-data-test:
  dbdata-test:

phpmyadmin: http://localhost:8888

Docker Swarm

A distributed system tool,used to manage a cluster of Docker engines, allows you to deploy your services.

Features

Concepts

Networking

Routing Mesh

Request -> ingress network on published port -> any node (use IPVS to decide which service to go) -> ingress network (outer LB)-> overlay network (inner LB) -> node

Docker swarm command

Init

docker swarm init --advertise-addr {ip}    // local ip

Show node list

docker node ls

Show network list

docker network ls

Create an overlay network

docker network create --opt encrypted --subnet 100.0.0.0/24 -d overlay {name}

Create service

docker service create --name nginx --network {network name} -p {ex_port:in_port} {image_name}
docker service create --name nginx --network {network name} -p 1080:80 nvbeta/swarm_nginx

Get started

  1. Use docker machine command to create 3 machines.
  2. SSH into one of machines and do swarm initialisation.
  3. SSH into the rest of machines and execute token command that is generated by step 2.
  4. Show node list to see if expected.

Setting

Open protocols and ports between the hosts

Troubleshootings

dial unix /var/run/docker.sock: permission denied

2014/12/23 14:14:33 Get http:///var/run/docker.sock/v1.12/images/json: dial unix /var/run/docker.sock: permission denied

sudo adduser <username> docker

exec: "./wait-for-it.sh": permission denied: unknown

If adding below into Dockerfile doesn’t work, make sure running docker build --no-cache . before docker-compose up --build

RUN chmod +x /usr/src/app/wait-for-it.sh

If docker is loading forever and can’t be launched:

  1. Move docker from application to trash bin
  2. Remove this file
rm -rf ~/Library/Containers/com.docker.docker
rm -rf ~/Library/Application\ Support/Docker\ Desktop
rm -rf ~/Library/Group\ Containers/group.com.docker