Software engineering notes

Linux Shell Script

Basics

接收傳入參數

建立test.sh內容:

#!/bin/bash
echo $0
echo $1
echo $2

執行你的shell檔案，並將要傳入的參數加在後面

$ sh test.sh hello 11 22
test.sh         # 檔名
hello           # 第一個參數
11              # 第二個參數

讀寫檔案內容

建立test.sh內容:

cat > qq.php << EOF
this is line 1
this is line 1
EOF

執行後就會產生qq.php檔案了，檔案內容為EOF框起來的內容

寫檔換行 :

echo -e "line1\nline2" | tee -a /tmp/q.txt

append到檔案最後面

tee -a /tmp/t.txt << EOF
line 1
line 2
EOF

讀取檔案，將每行放入陣列

file=1.txt

seq=1
while read line
do
    lines[$seq]=$line
    ((seq++))
done < $file

for ((i=1;i<=${#lines[@]};i++))
do
    echo ${lines[$i]}
done

使用sh執行會有錯誤，但bash執行沒問題

或 :

array_1=($(cat /tmp/t.txt))
echo ${array_1[0]}

使用sh執行會有錯誤，但bash執行沒問題

set

錯誤直接停止, 在最上面加上:

set -e

錯誤不停止，繼續執行

set +e

其他:

ref:

if else

t.sh :

#!/bin/bash

num=$1

if [ "${num}" == "XD" ]; then
    echo "= XD"
elif [ "${num}" != "QD" ]; then
    echo "!= QD"
else
    echo "Nothing"
fi

執行 :

$ bash t.sh XD
= XD
$ bash t.sh something
!= QD
$ bash t.sh QD
Nothing

AND / OR :

[ "${confirm}" == "y" ] && [ "${confirm}" == "Y" ]
[ "${confirm}" == "y" ] || [ "${confirm}" == "Y" ]

switch

case $1 in
    "aa")
        echo "aa"
        ;;
    "cc")
        echo "cc"
        ;;
    "dd")
        echo "dd"
        ;;
    *)
        echo
        echo "  ***\$1 doesn't match!"
        echo
        exit
        ;;
esac

迴圈

for

for var in con1 con2 con3 ...
do
    // do something
done

or

for ((i=0; i<6; i++)); do
  echo $i
  if [ ${i} == 3 ]; then
        continue
        // break
  fi
  echo "done"
done

while

i=1
while [ $i -le 5 ]      // i < 5
do
    ((i++))
    echo $i
done

until

until [ condition ]
do
    程式段落
done

陣列

lines=('line1' 'line2' 'line3')

${#array[@]} 為array的總數 ${array[0]} 索引從0開始, -1為最後一個值

判斷

檔案是否存在

if test -e /etc/network/if-pre-up.d/firewall; then
    echo "/etc/network/if-pre-up.d/firewall exists.."
fi

or

if [ -f /tmp/${filename}.tar.gz ]; then
    echo "exist"
fi

判斷檔案是否在該目錄

if (ls | grep -q "XXX"); then
    echo "XXX existed"
fi

判斷沒有目錄直接離開:

if [ ! -d "/var/www/$dir" ]; then
    echo "/var/www/$dir doesn't exist."
    exit
fi

判斷變數是否存在

if [ -z $1 ]; then
        # 不存在
else
        # 存在
fi

判斷只能輸入數字

echo "$1" | grep -o "^[0-9]*$"
if [ $? -eq 0 ] ;then
    echo "match found"
else
    echo "match not found"
fi

AND 條件

if (( $hour >= 0 )) && (( $hour <= 9 )); then
    # do something
else
    # do something
fi

亂數 10~19 中其中一個數字

shuf -i 10-19 -n 1

執行command結果給變數

random_num=$(shuf -i 10-19 -n 1)

or

count=$(grep "2017-07-07 $hour_str:$min_str" /var/log/php/ci/log-2017-07-07.php | wc -l)

How to run mysql commands through shell script?

mysql -u root test_database << EOF
SELECT * FROM test
EOF

有設定免密碼所以不用加 -p

在shell script執行 發生Bad substitution錯誤

可能發生的原因 ex: shell script 可能使用 ${str:3} (取得str變數第三個字元以後的字串)

因為sh不支援，所以執行時改用bash執行就可以解決了

bash test.sh

執行 ssh 後要怎麼離開

以下是沒有用的:

ssh qq.com
exit

要改成:

ssh qq.com <<ENDHERE
exit
ENDHERE

shell script 回覆 yes

當我要reload supervisor這個套件，但是會有新的一行yes/no，所以要使用這種方法才可以讓shell script幫我填入yes

sudo supervisorctl <<EOF
reload
y | command-that-asks-for-input
EOF

or

yes | sudo sensors-detect
sudo sensors

shell script 下 sudo echo 發生 permission denied

sudo echo "123" >> /etc/fstab

>> 是bash執行，不具root權限，以tee 代替>>

改成:

echo "123" | sudo tee -a /etc/fstab

-a, –append : append to the given FILEs, do not overwrite

match string

只比對test, 其餘不要

bash :

#!/bin/bash

test=$(echo $1 | grep -o "test[0-9]*")
echo ${test}

-o, –only-matching

取得指定字串長度

str='123456'
echo ${str:3:2}        # 結果為45

read讓使用者輸入變數

sudo fdisk -l
read -p "Please input XFS partition: " xfs_partition
echo $xfs_partition

執行漸進式選項

EX: fdisk partition

(echo o; echo n; echo p; echo 1; echo ; echo; echo w) | sudo fdisk

判斷 command 是否執行成功

some_command
if [ $? -eq 0 ]; then
    echo OK
else
    echo FAIL
fi

The return value is stored in $?. 0 indicates success, others indicates error.

比對檔案是否有符合字串

grep "127.0.1.1" /tmp/hosts > /dev/null 2>&1
if [ $? -eq 0 ]; then
    echo 'match'
fi

function

的第一種寫法，並返回值

function test ()
{
    return 255
}

test
echo $?

結果 : 255 return 範圍是 0~255，超過的話 : 256 為 0 257 為 1 使用 $? 取得 return 的值

第二種寫法，傳值並返回

test2 () {
 echo $1$2
}

string=$(test2 "Hello" " World!")
echo $string

結果 : Hello World! function 並不像 php 一樣順序可以放在執行後，shell script 的 function 必須放在執行前

背景執行不顯示 output

host www.google.com > /dev/null 2>&1

將 command 結果存入陣列

function test_dev_list()
{
    echo "/dev/sda5"
    echo "/dev/sda6"
    echo "/dev/sda7"
    echo "/dev/sda10"
    echo "/dev/sda11"
    echo "/dev/sda14"
}

dev_array=(`echo $(test_dev_list) | cut  -d " " --output-delimiter=" " -f 1-`)
for ((i=0; i<${#dev_array[@]}; i++)); do
    echo $i : ${dev_array[$i]}
done

結果:

0 : /dev/sda5
1 : /dev/sda6
2 : /dev/sda7
3 : /dev/sda10
4 : /dev/sda11
5 : /dev/sda14

減法

a=5
b=3
c=$(($a-$b))
echo $c

結果 : 2

輸出顏色文字及控制背景顏色

printf "\33[0;35;44m"
echo " Menu of available command:"
printf "\33[0m"
printf "\E[0;31;40m"
echo " Menu of available command:"
printf "\E[0m"
printf "\e[0;36;43m"
echo " Menu of available command:"
printf "\e[0m"

\33 = \E = \e 用 printf 來輸出，而不是一般的 echo

註 :

樣式：

文字顏色：

30 黑色    90 暗灰
31 紅色    91 亮紅
32 綠色    92 亮綠
33 黃色    93 亮黃
34 藍色    94 亮藍
35 紫色    95 亮紫
36 青藍綠  96 亮青藍綠
37 亮灰    97 白

背景顏色：

40 黑色    100 暗灰
41 紅色    101 亮紅
42 綠色    102 亮綠
43 黃色    103 亮黃
44 藍色    104 亮藍
45 紫色    105 亮紫
46 青藍綠  106 亮青藍綠
47 亮灰    107 白

ref : 參考顏色 http://mark528.pixnet.net/blog/post/7267334-shell-script%3A-%E6%8E%A7%E5%88%B6%E6%96%87%E5%AD%97%E9%A1%8F%E8%89%B2

printf + read

printf "Choose Monkey, are you sure (y/n) ? "
read xfs_choice_confirm

read 後面不需接 -p

補充 if 參數

ref: http://www.troubleshooters.com/linux/quickhacks.htm#ShellscriptFileTests

確認是否要執行以下指令

confirm() {
    echo "Press ENTER to continue, or ^C to cancel.";
    read -e ignored
}
confirm

判斷 Ubuntu 版本

LTS="Ubuntu 10.04"
ISSUE=`cat /etc/issue`
if [[ $ISSUE != Ubuntu* ]]; then
    echo "This script is intended for use on Ubuntu, but this system appears
    echo "to be something else. Your results may vary."
    echo
elif [[ `expr match "$ISSUE" "$LTS"` -eq ${#LTS} ]]; then
    echo "what"
fi

是否擁有 sudo 權限

echo "Testing sudo..."
sudo true
if [ $? -ne 0 ]
then
    echo "ERROR: You must be able to sudo to run this script.";
    exit 1;
fi;

動態偵測輸入變數

for TOKEN in $*
do
   echo $TOKEN
done

判斷第幾個參數

if [ $# -lt 1 ]; then

line 10: php: command not found

在使用 shell 的 bash 指令裡使用 php 指令記得要用絕對路徑

可使用 which php 查路徑在哪

指定 USER 執行命令

/usr/bin/sudo -H -u $USER bash -c "php -l qq.php"

截取目錄名

test@106-185-47-26:~/dockerfile/33-fsx$ echo ${PWD#*-}
fsx

當使用echo 增加資料發生Permission denied

$ sudo echo 'xxx' >> /etc/projects
-bash: /etc/projects: Permission denied

解決方法：

sudo echo 'xxx' | sudo tee -a /etc/projects

multi commands

alias lock='gnome-screensaver; gnome-screensaver-command --lock'

or

lock() {
    gnome-screensaver
    gnome-screensaver-command --lock
}

Examples

每秒執行一次

while true
do
    /bin/sleep 1s
done

Add cron by command line

先印出已存在的 crontab, 然候新增你要的存進去

(crontab -l; echo "0 * * * * your_command") |uniq - | crontab -

or

echo "0   0  *  *  * your_command" | sudo tee -a /var/spool/cron/crontabs/test

Remove cron by command line

先印出已存在的 crontab, 再用 sed 去刪除

crontab -l | sed "/pattern/d" | crontab -

other way

sudo sed -i "/pattern/.*$/d" /var/spool/cron/crontabs/test

取得網卡 mac

pi@raspbmc:~$ ifconfig |awk '/eth/{print $1,$5}'
eth0 b8:27:eb:7e:6e:3f

測試複製(cp)速度

#!/bin/bash
begin=`date "+%s%N"`
cp /test/iso /test/qq/iso
end=`date "+%s%N"`
rel=$(($end-$begin))
echo $rel

shell script 判斷套件是否已安裝, 如果未安裝自動安裝(自動輸入yes)

if dpkg -s ${package} | grep -q installed; then
    echo "${package} installed"
else
    echo "${package} isn't installed."
    echo "Install ${package}.."
    sudo apt-get --force-yes --yes install ${package}
fi

取得 route IP (Gateway IP)

route -n | grep 'UG[ \t]' | awk '{print $2}'

or :

ip r | awk '/^def/{print $3}'

or :

netstat -rn |awk '{if($1=="0.0.0.0") print $2}'

檢查是否能上網

router_ip=$(netstat -rn |awk '{if($1=="0.0.0.0") print $2}')
ping -c1 ${router_ip} >> /dev/null 2>&1
if [ $? -eq 0 ]; then
    echo OK
else
    echo FAIL
fi

檢查 dns 是否運作正常

# Check DNS
set +e
echo "Check DNS ..."
function check_dns()
{
    is_successful=0
    for ((i=0; i<2; i++)); do
        host www.google.com > /dev/null 2>&1
        if [ $? -eq 0 ]; then
            is_successful=1
            break;
        else
            echo "DNS can't function ..."
            echo "Restart networking and check again ..."
            sudo service networking restart
        fi
    done
    return $is_successful
}
check_dns
if [ $? == 0 ]; then
    echo "DNS can't function ..."
    exit
fi
set -e

選擇 partition (private)

set +e
dev_array=(`echo $(sudo fdisk -l | grep -E '^\/dev\/sda([3-9]|\d{2})' | awk '{print $1}') | cut  -d " " --output-delimiter=" " -f 1-`)
dev_array_count=$((${#dev_array[@]}-1))
until [ "${xfs_choice_confirm}" == "y" ] || [ "${xfs_choice_confirm}" == "Y" ]
do
    df
    sudo fdisk -l
    echo -e "\nDevice list :"
    echo "----------------------"
    for ((i=0; i<=$dev_array_count; i++)); do
        printf " \E[0;33;40m${i}\E[0m : ${dev_array[$i]}\n"
    done
    echo "----------------------"
    printf "Please choose XFS partition by entering a number : "
    read  dev_choice
    echo "${dev_choice}" | grep -o "^[0-9]*$" >> /dev/null 2>&1
    if [ ! $? -eq 0 ] || [ -z ${dev_array[${dev_choice}]} ]; then
        continue
    fi
    printf "You choose \E[0;33;40m${dev_array[${dev_choice}]}\E[0m, are you sure \E[0;31;40m(y/n)\E[0m ? "
    read xfs_choice_confirm
done
xfs_partition=${dev_array[${dev_choice}]}
set -e

顯示資料夾下的檔案及空間

d.sh :

for x in `find /tmp/destination -type f | sort`
do
  sum_y=`sum $x`
  echo "$sum_y $x";
done

顯示結果

$ bash d.sh
34198    19 /tmp/destination/dd.php
63612     3 /tmp/destination/qq/fff/ffddxx.php
53866     2 /tmp/destination/qq/fff/pp.php
07572     1 /tmp/destination/qq/qq.php

第一列數字不清楚 第二列數字(19, 3, 2, 1)是換送成kb的容量

php syntax checking

#!/bin/bash

# For checking php and js syntax

if [ -z $1 ]; then
    echo
    echo "  ***Missing arguments"
    echo
    exit
fi
for PATH in $*
do
    reg=".*(\.php|\.js)"
    if [[ $PATH =~ $reg  ]]; then
        if [ ${BASH_REMATCH[1]} == ".php" ]; then
            printf "\33[0;35;44m$PATH : \33[0m\n"
            /usr/bin/php -l $PATH
            echo "-------------------"
        elif [ ${BASH_REMATCH[1]} == ".js" ]; then
            printf "\33[0;36;44m$PATH : \33[0m\n"
            /usr/bin/sudo -H -u $USER bash -c "$HOME/test_project/bin/jslint -f $PATH"
            echo "-------------------"
        fi
    fi
done

產生亂數字串

This method uses SHA to hash the date, runs through base64, and then outputs the top 32 characters.

test@test:~$ date +%s | sha256sum | base64 | head -c 32 ; echo
MmM1MWFjMDVhYzBkMzk3ODRjM2JiZTQ5

This method used the built-in /dev/urandom feature, and filters out only characters that you would normally use in a password. Then it outputs the top 32.

test@test:~$ < /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c${1:-32};echo;
udqgTdSRa_zWN1m6E_1iKMzCjvqXUUpD

This one uses openssl’s rand function, which may not be installed on your system. Good thing there’s lots of other examples, right?

test@test:~$ openssl rand -base64 32
39iFjCtPB+4zjjsdJJ+FtFjBYPD3GUDRIZtXld5UyRM=

This one works a lot like the other urandom one, but just does the work in reverse. Bash is very powerful!

test@test:~$ tr -cd '[:alnum:]' < /dev/urandom | fold -w30 | head -n1
Rz27w5YH3FF8D4ughqbgHkHIcgds9T

Here’s an even simpler version of the urandom one.

test@test:~$ < /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c6
RA0g58

You can even create a random left-hand password, which would let you type your password with one hand.

test@test:~$ </dev/urandom tr -dc '12345!@#$%qwertQWERTasdfgASDFGzxcvbZXCVB' | head -c8; echo ""
ZBFC@#AB

And here’s the easiest way to make a password from the command line, which works in Linux, Windows with Cygwin, and probably Mac OS X. I’m sure that some people will complain that it’s not as random as some of the other options, but honestly, it’s random enough if you’re going to be using the whole thing.

test@test:~$ date | md5sum
642a1b5de18a71b8585cfe87831d45a5  -

ref: random string

shell scripting and regular expression

echo "$password" | grep -q "[a-z]*[0-9][a-z]*"
if [ $? -eq 0 ] ;then
    echo "match found"
else
    echo "match not found"
fi

match $1

reg=".*(\.php)"
if [[ $PATH =~ $reg  ]]; then
    echo ${BASH_REMATCH[1]}
fi

input : t.php

result : .php

多行取代

sed "N;s/xxx/ccc/g" filename 因為加上 N 參數確實可以多行 matching, 但是當你要取代一個範圍(可能預期 match 到 3~6 行做取代) 卻是沒有辦法的..暈

那我將一個檔案讀取出來處理完再存到另一個檔案總沒問題了吧? 原則上是這樣沒錯..但是有一個小問題是行首的空白會不見.. 原來 bash shell 預設斷行是 space, tabs, new line(\n), 不過我們可以控制斷行的字元 - IFS(nternal Field Separator) 這個變數, 當 shell 內部讀取每一行時一遇到 \n 就中斷, 即可解決此問題, 請看以下程式碼 :

# 將檔案儲存到陣列
c=0
OLD_IFS=$IFS
IFS=$'\n'
while read -r line
do
    bashrcArray[$c]=$line
    c=$(expr $c + 1)
done < bashrc

# 這邊不做取代細節, 直接將陣列儲存成檔案
for i in "${bashrcArray[@]}"
do
    echo $i >> bashrcqq
done
IFS=$OLD_IFS

沒想到 shell script 處理多行取代居然會那麼麻煩