Go Built-in Packages

fmt

輸出固定長度

fmt.Printf("ID : %-10s", id)    // 十個字元的長度, 向左對齊
fmt.Printf("ID : %10s", id)     // 十個字元的長度, 向右對齊
fmt.Printf("ID : %.10s", id)    // 印出頭十個字元
fmt.Printf("%q", []string{"a","b"}) // 印 slice `["a" "b"]`

前面補 0

fmt.Sprintf("%02d:%02d", 5, 3)  // 05:03

接收 command 輸入的值

fmt.Print("Enter a number: ")
var input float64
fmt.Scanf("%f", &input)

印出顏色

fmt.Println("\x1b[31;1mHello, World!\x1b[0m")

最簡單的方法, 不過有個缺點是它會是 string

fmt.Sprintf("%.7f", 123.123456789)                              // 123.1234568

不改變本身的型態

val := 123.123456789
pow10_n := math.Pow10(7)                                        // 1e7 = 10000000
d := math.Trunc(val*pow10_n) / pow10_n                          // 123.1234567

如果已知位數, 可簡化

math.Trunc(123.1234567*1e5) / 1e5                               // 123.12345

不改變本身的型態且4捨5入

val := 123.123456789
pow10_n := math.Pow10(7)
d := math.Trunc((val+0.5/pow10_n)*pow10_n) / pow10_n            // 123.1234568

log

儲存成 file

// open a file
f, err := os.OpenFile("dev.log", os.O_APPEND|os.O_CREATE|os.O_RDWR, 0666)
if err != nil {
    log.Printf("Create log file error: %v", err)
}

// don't forget to close it
defer f.Close()

// assign it to the standard logger
log.SetOutput(f)

// 以下的 log 都會進到 log file 裡
log.Println("test")
log.Println("testr2")
log.Println("testr3")

將 log 內容暫存在 buffer

var buf bytes.Buffer
logger := log.New(&buf, "logger: ", log.Lshortfile)
logger.Print("Hello, log file!")

fmt.Print(&buf)

攔截 log 的內容

var buf bytes.Buffer
logger := log.New(&buf, "", log.Lshortfile)
go func() {
    for {
        if buf.Len() > 0 {
            fmt.Print(&buf)
            buf.Reset()
        }
    }
}()
fmt.Println(0)
logger.Print("Hello, log file!1")
logger.Print("Hello, log file!2")
logger.Print("Hello, log file!3")

dd := make(chan int)
<-dd

寫到 syslog

// Configure logger to write to the syslog. You could do this in init(), too.
logwriter, e := syslog.New(syslog.LOG_NOTICE, "myprog")
if e == nil {
    log.SetOutput(logwriter)
}

// Now from anywhere else in your program, you can use this:
log.Print("Hello Logs!")

/var/log/system.log :

Nov  4 08:19:33 testde-MacBook-Pro myprog[90449]: 2016/11/04 08:19:33 Hello Logs!

net

取得本機上的 Mac

interfaces, err := net.Interfaces()
if err != nil {
    panic("Poor soul, here is what you got: " + err.Error())
}
for _, inter := range interfaces {
    fmt.Println(inter.Name, inter.HardwareAddr)
}

header

resp, _ := http.Get(url)
// &{200 OK 200 HTTP/1.1 1 1 map[Connection:[keep-alive] Server:[nginx] Accept-Ranges:[bytes] Content-Length:[994291] Date:[Sat, 25 Jan 2014 19:13:51 GMT] Etag:["4e28a21e-f2bf3"] Content-Type:[video/x-flv] Last-Modified:[Thu, 21 Jul 2011 22:03:10 GMT]] 0xf8400c2a00 994291 [] false map[] 0xf8400c0240}

Status

resp.Status             // 200 OK
resp.StatusCode         // 200

size

resp.Header.Get("Content-Length")
// 994291

body

source, _ := ioutil.ReadAll(resp.Body)
// source 為 131 239 216 100 96 184 2 221 171 162 131 49 17 33 17 39 152 176 194 18 19 62 40 124 93 230 48 58 9 (..略..)

len(source) 結果會和 header 的 Content-Length 一樣

net/url 解析 URL

s := "postgres://user:pass@host.com:5432/path?k=v#f"
u, err := url.Parse(s)
fmt.Println(u.Scheme)                   // postgres
fmt.Println(u.User)                     // user:pass
fmt.Println(u.User.Username())          // user
p, _ := u.User.Password()
fmt.Println(p)                          // pass
fmt.Println(u.Host)                     // host.com:5432
h := strings.Split(u.Host, ":")
fmt.Println(h[0])                       // host.com
fmt.Println(h[1])                       // 5432
fmt.Println(u.Path)                     // /path
fmt.Println(u.Fragment)                 // f
fmt.Println(u.RawQuery)                 // k=v
m, _ := url.ParseQuery(u.RawQuery)
fmt.Println(m)                          // map[k:[v]]
fmt.Println(m["k"][0])                  // v

u, _ := url.Parse("http://www.test.com/xxx.mp3?foo=bar&foo=baz#this_is_fragment")
fmt.Println("full uri:", u.String())  // full uri: http://www.test.com/xxx.mp3?foo=bar&foo=baz#this_is_fragment
fmt.Println("scheme:", u.Scheme)      // scheme: http
fmt.Println("opaque:", u.Opaque)      // opaque:
fmt.Println("Host:", u.Host)          // Host: www.test.com
fmt.Println("Path", u.Path)           // Path /xxx.mp3
fmt.Println("Fragment", u.Fragment)   // Fragment this_is_fragment
fmt.Println("RawQuery", u.RawQuery)   // RawQuery foo=bar&foo=baz
fmt.Printf("query: %#v\n", u.Query()) // query: url.Values{"foo":[]string{"bar", "baz"}}

url 組 query

u, _ := url.Parse("https://example.com")
u.Path += "/api/v1/user/event"
query := url.Values{}
query.Add("a", "12+3")
query.Add("b", "4&D")
u.RawQuery = query.Encode()

output

example.com
/api/v1/user/event?a=12%2B3&b=4%26D
a=12%2B3&b=4%26D
https://example.com/api/v1/user/event?a=12%2B3&b=4%26D

It Urlencoded automatically.

已存在的 url 再新增 query param

var u *url.URL
u, _ = url.Parse("https://example.com/ccccc/ffff.jpg?a=aa&c=cc")
q := u.Query()
q.Set("q", "golang")
u.RawQuery = q.Encode()
fmt.Println(u.String())

自組 post body 要 escape

Post body 規則像 url, 要 url encode, 否則 + 等等的符號會被當成是 space

body := fmt.Sprintf("p=%s&iv=%s", url.QueryEscape(p_64), url.QueryEscape(iv_64))

net/http Download file

out, err := os.Create("output.txt")
defer out.Close()
resp, err := http.Get("http://example.com/")
defer resp.Body.Close()
n, err := io.Copy(out, resp.Body)

res, _ := http.Head(url)
maps := res.Header

它連完線馬上就會斷了，不需要手動關掉

Get filename from url path

import (
    "net/url"
    "path/filepath"
)

s := "https://talks.golang.org/2012/splash/appenginegophercolor.jpg"
u, _ := url.Parse(s)
fmt.Println(filepath.Base(u.Path))

GET method

import (
  "http"
  "io/ioutil"
  "os"
)

response, _, err := http.Get("http://golang.org/")
if err != nil {
    fmt.Printf("%s", err)
    os.Exit(1)
} else {
    defer response.Body.Close()
    contents, err := ioutil.ReadAll(response.Body)
    if err != nil {
        fmt.Printf("%s", err)
        os.Exit(1)
    }
    fmt.Printf("%s\n", string(contents))
}

ref : https://gist.github.com/ijt/950790

client := http.Client{
  Timeout: time.Second * 5,
}
req, err := http.NewRequest("GET", "http://www.google.com/dd", nil)
req.Close = true        // Note !!  避免發生 POST EOF 問題
req.Header.Set("Content-Type", "text/plain")
req.Header.Add("Content-Type", "text/plain")
resp, err := client.Do(req)
if err != nil {
    return err.Error()
}
defer resp.Body.Close()

if resp.StatusCode != http.StatusOK {
  err = errors.New("Server return non-200 status")
  return
}
contents, _ := ioutil.ReadAll(resp.Body)
fmt.Println(string(contents))
return

post :

http.NewRequest("GET", "http://www.google.com/dd", strings.NewReader("name=cjb"))

Post (form-data / json)

// New request
req, err := http.NewRequest("POST", c.URL, bytes.NewBuffer([]byte(c.PostData)))
req.Close = true
for k, v := range c.Header {
    req.Header.Set(k, v)
}

// 根據不同 Post type 加上對應的 header
switch strings.ToUpper(c.PostType) {
case "FORM":
    req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
case "JSON":
    req.Header.Set("Content-Type", "application/json")
}

// Send request
resp, err = client.Do(req)
if err != nil {
    return
}
defer resp.Body.Close()

Post (file)

// 模擬 form 的一個 選項, 現在是空的
bodyBuf := &bytes.Buffer{}
bodyWriter := multipart.NewWriter(bodyBuf)

// 設定要上傳的檔案到 form 裡
fileWriter, err := bodyWriter.CreateFormFile(c.FileFieldName, c.FileFieldName)  // 一個是 file name, 一個是 field name, 都用 field name 就好了
if err != nil {
    err = errors.New("Failed to create form file")
    return
}

// 支援 file path 及 file bytes
var file *os.File
if c.FilePath != "" {
    // File path
    file, err = os.Open(c.FilePath)
    if err != nil {
        err = errors.New("Failed to open " + c.FilePath)
        return
    }
    defer file.Close()

    if _, err = io.Copy(fileWriter, file); err != nil {
        return
    }
} else {
    // Bytes
    r := bytes.NewReader(c.Bytes)
    if _, err = io.Copy(fileWriter, r); err != nil {
        return
    }
}

// 設定其他 form-data 的欄位跟值
// 它一定要在 bodyWriter.Close() 前, 因為這 close 完畢就是 boundary 的結尾, 有些在實作這功能時看到這個結尾就不會再繼續 parse 接下來 form-data 的欄位了 (Go Echo 上是這樣, 但 php 沒問題)
for key, val := range c.Params {
    if err = bodyWriter.WriteField(key, val); err != nil {
        return
    }
}

bodyWriter.Close()

// New request
req, err := http.NewRequest("POST", c.URL, bodyBuf)
// req.Close = true
for k, v := range c.Header {
    req.Header.Set(k, v)
}
req.Header.Set("Content-Type", bodyWriter.FormDataContentType()) // 取得上傳檔案的 Content-Type 的值, e.g. multipart/form-data; boundary=001e5fc50cf83d32c170bfef235709ba9016b0468c93ad2f108551a9f48c

// Send request
resp, err = client.Do(req)
if err != nil {
    return
}
defer resp.Body.Close()

Http 建立 API

http.HandleFunc("/", Index)
err := http.ListenAndServe(":5555", nil)
if err != nil {
    log.Fatal("ListenAndServe: ", err)
}

func Index(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, string("index"))
    // w.Write([]byte("Time: " + time.Now().Format(time.RFC1123)))
}

也可以改由這樣連線, 先建立 linstener

http.HandleFunc("/", Index)
s := &http.Server{}
l, err := net.Listen("tcp", ":5555")
if err != nil {
    panic(err)
}
panic(s.Serve(l))

ConnState 可以截取 connection 的狀態

總共有這五種狀態 StateNew StateActive StateIdle StateHijacked StateClosed

func main() {
    http.HandleFunc("/", myHandler)
    s := &http.Server{
        Addr:           ":8081",
        ReadTimeout:    10 * time.Second,
        WriteTimeout:   10 * time.Second,
        MaxHeaderBytes: 1 << 20,
        ConnState:      ConnStateListener,
    }
    panic(s.ListenAndServe())
}

func ConnStateListener(c net.Conn, cs http.ConnState) {
    fmt.Printf("CONN STATE: %v, %v\n", cs, c)
    // 如果想要直接結束 connection : c.Close()
}

ref : http://siddontang.com/2015/01/25/stop-server-gracefully/

handle connection (created at 27 Dec 2013)

ln, err := net.Listen("tcp", ":8080")
if err != nil {
    log.Fatal("Starting error", err)
}
conn, err := ln.Accept() // this blocks until connection or error
if err != nil {
    fmt.Println("Create connection failure!")
}
handleConnection(conn)

func handleConnection(conn net.Conn) {
    reader := bufio.NewReader(os.Stdin)
    for {
        fmt.Print("Please enter a message : ")
        str, err := reader.ReadString('\n')
        if err != nil {
            break
        }
        if str != "" {
            str := fmt.Sprintf("HTTP/1.1 200 OK\nContent-Length: %d\nContent-Type: text/html\n\n%s", len(str) - 1 , str)
            conn.Close() // shut down the connection
            break
        }
    }
    bufio.NewReader(os.Stdin).ReadBytes('\n')
}

TCP example

TCP server

func launchTCPserver() {
    // Listen for incoming connections.
    l, err := net.Listen("tcp", ":8081")
    if err != nil {
        fmt.Println("Error listening:", err.Error())
        os.Exit(1)
    }
    // Close the listener when the application closes.
    defer l.Close()
    fmt.Println("Listening on " + CONN_HOST + ":" + CONN_PORT)
    for {
        // Listen for an incoming connection.
        conn, err := l.Accept()
        if err != nil {
            fmt.Println("Error accepting: ", err.Error())
            os.Exit(1)
        }
        // Handle connections in a new goroutine.
        go handleRequest(conn)
    }
}

// Handles incoming requests.
func handleRequest(conn net.Conn) {
    OuterLoop:
        for {
            // will listen for message to process ending in newline (\n)
            message, err := bufio.NewReader(conn).ReadString('\n')
            switch err {
            case io.EOF:
                fmt.Printf("client disconnected: %v\n", err)
                break OuterLoop
            case nil:
            default:
                fmt.Println(err)
                break OuterLoop
            }
            // output message received
            fmt.Print("[server] Message Received:", string(message))
            // Processing
            time.Sleep(1 * time.Second)
            // send new string back to client
            conn.Write([]byte(string(message) + "\n"))
        }
}

TCP client

conn, _ := net.Dial("tcp", "127.0.0.1:8081")
for {
    // read in input from stdin
    reader := bufio.NewReader(os.Stdin)
    fmt.Println("-----")
    fmt.Print("Text to send: ")
    text, _ := reader.ReadString('\n')
    // send to socket
    fmt.Fprintf(conn, text+"\n")
    // listen for reply
    message, _ := bufio.NewReader(conn).ReadString('\n')
    fmt.Print("[Client] Message from server: " + message)
}

reply := make([]byte, 1024)
_, err = conn.Read(reply)
if err != nil {
    println("Write to server failed:", err.Error())
}
fmt.Println(string(reply))

API Custom Handler

type timeHandler struct {
  format string
}

func (th *timeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  tm := time.Now().Format(th.format)
  w.Write([]byte("The time is: " + tm))
}

func main() {
  mux := http.NewServeMux()

  th := &timeHandler{format: time.RFC1123}
  mux.Handle("/time", th)

  log.Println("Listening...")
  http.ListenAndServe(":3000", mux)
}

func (th *timeHandler) CustomName(w http.ResponseWriter, r *http.Request) {}

mux.HandleFunc("/time", th.CustomName)

net/http - x509: certificate signed by unknown authority

Post 到一個 https 的 URL 得到 x509: certificate signed by unknown authority

會發生這個原因主要是 golang 的 client 端會自動地對 server 傳來的 certificate 做驗證，但因為它是由不知名的 CA 簽發的，所以有 error

網路其中解法之一 : To fix this problem, you must add certificates of “COMODO RSA Domain Validation Secure Server CA” to your certififcate served from server, also see full report at https://www.ssllabs.com/ssltest/analyze.html?d=catchchat.catchchatchina.com

另一個解法是不要去驗證它

import ("net/http"; "crypto/tls")

tr := &http.Transport{
    TLSClientConfig: &tls.Config{InsecureSkipVerify : true},
}
client := &http.Client{Transport: tr}
resp, err := client.Get("https://someurl:443/)

更好的解法是對 server 的 certificate 做驗證, 這邊有點超越我的能力了, 請參考這

net/http client request 得到 EOF 原因

Post https://xxxxx.com/wait/: EOF

有可能是這個 request 需要比較久的時間，即使你的 http client 的 timeout 設很長，也會因為 server timeout 太短先把你踢掉，所以你會得到這個 Error

Get lan ip

func GetLocalIP() string {
    addrs, err := net.InterfaceAddrs()
    if err != nil {
        return ""
    }
    for _, address := range addrs {
        // check the address type and if it is not a loopback the display it
        if ipnet, ok := address.(*net.IPNet); ok && !ipnet.IP.IsLoopback() {
            if ipnet.IP.To4() != nil {
                return ipnet.IP.String()
            }
        }
    }
    return ""
}

Proper way to test CIDR membership of an IP 4 or 6 address example

ref: https://www.socketloop.com/tutorials/golang-proper-way-to-test-cidr-membership-of-an-ip-4-or-6-address-example

package main

import (
        "fmt"
        "net"
        "os"
)

func main() {
        // generate a range of IP version 4 addresses from a Classless Inter-Domain Routing address
        ipAddress, ipNet, err := net.ParseCIDR("123.45.67.64/27")

        if err != nil {
                fmt.Println(err)
                os.Exit(1)
        }

        // generate a range of IPv4 addresses from the CIDR address
        var ipAddresses []string

        for ipAddress := ipAddress.Mask(ipNet.Mask); ipNet.Contains(ipAddress); inc(ipAddress) {
                //fmt.Println(ipAddress)
                ipAddresses = append(ipAddresses, ipAddress.String())
        }

        // list out the ipAddresses within range
        for key, ipAddress := range ipAddresses {
                fmt.Printf("[%v] %s\n", key, ipAddress)
        }

        //test for IP version 4 address for membership

        // WRONG WAY!!
        fmt.Println("Contains 123.45.67.69 : ", ipNet.Contains([]byte("123.45.67.69")))

        // CORRECT / PROPER WAY!
        fmt.Println("Contains 123.45.67.69 : ", ipNet.Contains(net.ParseIP("123.45.67.69")))

}

func inc(ip net.IP) {
        for j := len(ip) - 1; j >= 0; j-- {
                ip[j]++
                if ip[j] > 0 {
                        break
                }
        }
}

output :

[0] 123.45.67.64
[1] 123.45.67.65
...略...
[9] 123.45.67.73
...略...
[31] 123.45.67.95
Contains 123.45.67.69 :  false
Contains 123.45.67.69 :  true

io

pipe

pr, pw := io.Pipe()
go func() {
    defer pw.Close()
    for i := 1; i <= 3; i++ {
        _, err := fmt.Fprintf(pw, "Hello %d\n", i)
        if err != nil {
            panic(err)
        }
        time.Sleep(1 * time.Second)
    }
}()

_, err := io.Copy(os.Stdout, pr)
if err != nil {
    panic(err)
}

io/ioutil/bufio

將下載的資料轉成 io.Reader 型態

var source io.Reader
source = resp.Body
buffer := make([]byte, 1024)
for {
    cBytes, _ := source.Read(buffer)
    (...略...)

等同於

buffer := make([]byte, 1024)
for {
    cBytes,_ := resp.Body.Read(buffur)
    (...略...)

Read File

var img64 []byte
img64, _ = ioutil.ReadFile("/home/ubuntu/mygo/src/pushImage/google.png")

Read last line of file (ref: here)

func readLastLine(filename string) {
    var previousOffset int64 = 0

    file, err := os.Open(filename)
    if err != nil {
            panic(err)
    }

    defer file.Close()

    reader := bufio.NewReader(file)

    // we need to calculate the size of the last line for file.ReadAt(offset) to work

    // NOTE : not a very effective solution as we need to read
    // the entire file at least for 1 pass :(

    lastLineSize := 0

    for {
            line, _, err := reader.ReadLine()

            if err == io.EOF {
                    break
            }

            lastLineSize = len(line)
    }

    fileInfo, err := os.Stat(filename)

    // make a buffer size according to the lastLineSize
    buffer := make([]byte, lastLineSize)

    // +1 to compensate for the initial 0 byte of the line
    // otherwise, the initial character of the line will be missing

    // instead of reading the whole file into memory, we just read from certain offset

    offset := fileInfo.Size() - int64(lastLineSize+1)
    numRead, err := file.ReadAt(buffer, offset)

    if previousOffset != offset {

            // print out last line content
            buffer = buffer[:numRead]
            fmt.Printf("%s \n", buffer)

            previousOffset = offset
    }

}

Get last-modified time of file

// Get file list from dir
files, err := ioutil.ReadDir("/tmp")
if err != nil {
    return
}

for _, file := range files {
    // Skip dir
    if file.IsDir() {
        continue
    }

    fmt.Printf("%s %s\n", file.Name(), file.ModTime())
}

Line counter

func lineCounter(r io.Reader) (int, error) {
    buf := make([]byte, 32*1024)
    count := 0
    lineSep := []byte{'\n'}

    for {
        c, err := r.Read(buf)
        count += bytes.Count(buf[:c], lineSep)

        switch {
        case err == io.EOF:
            return count, nil

        case err != nil:
            return count, err
        }
    }
}

file, err := os.Open("/tmp/test.log")
if err != nil {
    return
}
defer file.Close()
count, err := lineCounter(file)

有效率的 monitor file 並取得最新內容 (類似 tail 指令)

先 new File
用兩個 int 變數記錄上一次及偵測到檔案變更的位置 (也就是檔案大小, 用 os.Stat 取得)
使用 fsnotify/fsnotify 偵測變更, 有變更就取最新內容 (用 file.ReadAt 在上一次的位置開始讀取, 讀取的長度用目前的size減上一次size

code:

for {
        select {
        case event := <-watcher.Events:
                if event.Op&fsnotify.Write == fsnotify.Write {
                        // log.Println("modified file:", event.Name)
                        info, err := file.Stat()
                        if err != nil {
                                log.Println("failed to get file size, err: ", err)
                                break
                        }
                        curr_size = info.Size()
                        last_line, err := readLastLine(file, prev_size, curr_size)
                        if err != nil {
                                log.Println("failed to get last line, err: ", err)
                                continue
                        }
                        prev_size = curr_size
                        if last_line != "" {
                                log.Println(last_line)
                        }
                }
        case err := <-watcher.Errors:
                log.Println("error:", err)
                return
        }
}

func readLastLine(file *os.File, prev int64, curr int64) (last_line string, err error) {
        // start reading from the end of the file
        buf := make([]byte, curr-prev)
        n, err := file.ReadAt(buf, prev)
        if err != nil && err != io.EOF {
                return "", err
        }
        return string(buf[:n]), nil
}

io - Read into []byte from another []byte

io.Copy (900MB, 1s)

b2 := bytes.NewBuffer(nil)
_, err = io.Copy(b2, bytes.NewReader(b))
if err != nil {
    panic(err)
}

ioutil.ReadAll (900MB 3s)

b3, err := ioutil.ReadAll(bytes.NewReader(b))
if err != nil {
    panic(err)
}

buffer read (900MB, 3s)

buf := make([]byte, 4*1024)
br := bufio.NewReader(bytes.NewReader(b))
b2 := bytes.NewBuffer(nil)
for {
    n, err := br.Read(buf)
    if err != nil {
        if err == io.EOF {
            break
        }
        panic(err)
    }
    if n == 0 {
        break
    }

    if _, err := b2.Write(buf[:n]); err != nil {
        panic(err)
    }
}

buffer pipe read (900MB, 4s)

pr, pw := io.Pipe()
go func() {
    defer pw.Close()
    buf := make([]byte, 4*1024)
    br := bufio.NewReader(bytes.NewReader(b))
    for {
        n, err := br.Read(buf)
        if err != nil {
            if err == io.EOF {
                break
            }
            panic(err)
        }
        if n == 0 {
            break
        }

        if _, err := pw.Write(buf[:n]); err != nil {
            panic(err)
        }
    }
}()

b2 := bytes.NewBuffer(nil)
_, err = io.Copy(b2, pr)
if err != nil {
    panic(err)
}

string

基本處理

String to Array

strings.Split("a/b/c",  "/")

Array to String (join)

s := []string{"a", "b", "c"}
fmt.Println(strings.Join(s, "/"))

os

外部參數

command : go run t.go dd
t.go : fmt.Println(os.Args[1])

“main.main” 函數並沒有返回值

如果想返回一個出錯信息，可用系統調用強制退出：

os.Exit(1)

# Stop
os.Exit(0)

檔案 / 目錄是否存在

檔案或目錄是否存在

if _, err := os.Stat(filename); os.IsNotExist(err) {
    fmt.Printf("no such file or directory: %s", filename)
    return
}

檔案資訊

func GetFileInfo(path string) (isExistent bool, fileInfo os.FileInfo) {
    fileInfo, err := os.Stat(path)
    if err != nil {
        // no such file or dir
        return false, nil
    }
    if fileInfo.IsDir() {
        // it's a directory
        return false, nil
    }
    // it's a file
    return true, fileInfo
}

目錄是否存在

func DirExists(path string) (bool) {
    fileInfo, err := os.Stat(path)
    if err != nil {
        // no such file or dir
        return false
    }
    if fileInfo.IsDir() {
        // it's a directory
        return true
    }
    // it's a file
    return false
}

檔案是否存在

func FileExists(path string) (bool) {
    fileInfo, err := os.Stat(path)
    if err != nil {
        // no such file or dir
        return false
    }
    if fileInfo.IsDir() {
        // it's a directory
        return false
    }
    // it's a file
    return true
}

目前位置

dir, err := filepath.Abs(filepath.Dir(os.Args[0]))
if err != nil {
        log.Fatal(err)
}
fmt.Println(dir)

列出指定路徑下的檔案或目錄

files, err := ioutil.ReadDir("./")
if err != nil {
    log.Fatal(err)
}
for _, f := range files {
        fmt.Println(f.Name())
}

file - 指定寫入位置

file, _ := os.OpenFile("tt", os.O_RDWR|os.O_APPEND, 0660)
n, err := file.WriteAt([]byte("中Start"), 3)
if err != nil {
    fmt.Println(err.Error())
}
fmt.Println(n)

返回的 n 是寫入的字節大小, 中文字佔 3 btyes, 英文 1 bytes

檔案 tt 寫入後的結果 : ^@^@^@中Start

其中 ^@ 為空字元, 共有 3 組 ^@ 是因為 WriteAt 指定從 3 開始寫入,

但檔案位置是從 0 開始算, 所以會有 3 組 ^@

寫入改成 n, err := file.WriteAt([]byte("678"), 6)

結果為 : ^@^@^@中678rt

註 : ^@ 佔 1byte, 中 佔 3 bytes

Create folder

 import (
   //"fmt"
   "os"
   "path/filepath"
 )

 func main() {
   // create a TestDir directory on current working directory
   os.Mkdir("." + string(filepath.Separator) + "TestDir",0777)
 }

在 windows 下執行可執行檔(.exe)後視窗停留不關閉

引入 :

import (
    "bufio"
    "os"
)

最後一行執行 :

bufio.NewReader(os.Stdin).ReadBytes('\n')

當執行 .exe 檔後視窗就不會關閉了

或用另一種寫法

b := make([]byte, 1)
os.Stdin.Read(b)

不同作業系統取得不同 path separator (slash, 斜線)

string(filepath.Separator)

progress bar

package main
import (
  "fmt"
  "strings"
  "time"
  "os"
)
func main() {
  for i := 0; i < 50; i++ {
    time.Sleep(100 * time.Millisecond)
    h := strings.Repeat("=", i) + strings.Repeat(" ", 49-i)
    fmt.Printf("\r%.0f%%[%s]", float64(i)/49*100, h)
    os.Stdout.Sync()
  }
  fmt.Println("\nAll System Go!")
}

讓字一個一個 print 出來

func main() {
    text := "Hello World!"
    printSlowly(text, 200)

    bufio.NewReader(os.Stdin).ReadBytes('\n')
}

func printSlowly(text string, speed int) {
    text_rune := []rune(text)

    for i := 0; i < len(text_rune); i++ {
        time.Sleep(time.Duration(speed) * time.Millisecond)
        fmt.Printf("\r%s", string(text_rune[0:i+1]))
        os.Stdout.Sync()
    }
    fmt.Print("\n")
}

os/exec

取得執行中的 binary 路徑

[binary path]
os.Args[0]                      # /home/apps/go/src/test/test

[dir path]
filepath.Dir(os.Args[0])        # /home/apps/go/src/test

執行外部指令

output, err := exec.Command("git", "rev-parse", "HEAD").Output()
if err != nil {
    return "unknown"
}
fmt.Println(string(output))

output 最後會含一個 ascii: 10 (hex: 0A) 的換行字元, 它並不是 \n 字元

signal

接收中止訊號(kill signal)

當程式執行後會停在 <-sigc，如果收到 kill 指令則會繼續往下走

sigc := make(chan os.Signal, 1)
signal.Notify(sigc, os.Interrupt, os.Kill)
<-sigc
log.Println("Abort!")

發送 signal

重啟 signal

syscall.Kill(syscall.Getpid(), syscall.SIGHUP)

Kill process signal

syscall.Kill(syscall.Getpid(), syscall.SIGTERM)

What would it be if both of main program and sub-program try to capture the signals at the same time?

Both of them will capture that signal at the same time.

time

Datetime

time.Now()                  // 2016-12-29 17:33:23.784617257 +0800 CST

Format

 time.Now().Format(time.RFC3339)                            // 2017-02-14T06:59:21Z   可以以 string 型態被解到 time.Time 型態
 time.Now().Local().Format("2006-01-02 15:04:05 +0800")     // 2014-06-30 14:23:38 +0800
 time.Now().Format("2006-01-02 15:04:05")                   // 2015-02-03 04:16:54
 time.Now().String()                                        // 2017-01-10 06:51:14.271079336 +0000 UTC
 time.Now().UTC().String()                                  // 轉成 UTC+0

Timestamp

time.Now().Unix()           // 1483004003

Convert timestamp to time

tm := time.Unix(1484032609, 0)

Convert string to time

today := "2017-01-10 16:57:28"
t, err := time.Parse("2006-01-02 15:04:05", today)

Convert string to time in timezone

// 將指定的時間跟 timezone 轉成 time.Time
l, _ := time.LoadLocation("Asia/Taipei")
t, _ := time.ParseInLocation("2006-01-02 15:04:05", "2019-01-02 17:04:43", l)
fmt.Println(t.In(time.UTC).Format("2006-01-02 15:04:05")) // 2019-01-02 09:04:43
fmt.Println(t.In(l).Format("2006-01-02 15:04:05"))        // 2019-01-02 17:04:43

UTC: time.LoadLocation(“UTC”)

Convert string to time in RFC3339

ts, err := time.Parse(time.RFC3339, src)

Convert time to time in timezone

t := time.Now()
loc, err := time.LoadLocation("Asia/Taipei")
fmt.Println(t.String())             // 2017-01-10 08:41:19.833220416 +0000 UTC
fmt.Println(t.In(loc))              // 2017-01-10 16:41:19.833220416 +0800 CST
fmt.Println(t.In(loc).Format("2006-01-02 15:04:05"))

year, month, day, hour, min, sec

year, month, day := t2.Date()
fmt.Printf("Date : [%d]year : [%d]month : [%d]day \n", year, month, day)
// Date : [2017]year : [1]month : [10]day

hr, min, sec := t2.Clock()
fmt.Printf("Clock : [%d]hour : [%d]minutes : [%d] seconds \n", hr, min, sec)
// Clock : [16]hour : [57]minutes : [28] seconds

first/last day of month

y, m, _ := time.Now().Date()
first := time.Date(y, m, 1, 0, 0, 0, 0, loc)
last := first.AddDate(0, 1, 0).Add(-time.Nanosecond)

Parse time 的非預期狀況

timeTest := "20:36"
timezone := "Asia/Tokyo"
loc, err := time.LoadLocation(timezone)
if err != nil {
    log.Fatal(err)
}
t, err := time.ParseInLocation("15:04", timeTest, loc)
if err != nil {
    log.Fatal(err)
}
fmt.Println(t.Format("15:04"))
fmt.Println(t.UTC().Format("15:04"))

在某些主機會是預期結果

20:36
11:36

某些會是:

20:36
11:17

似乎也不是 go 版本不同的問題, 不知道確切發生的原因, 但如果將指定的時間用完整一點可以解決此問題

timeTest := time.Now().Format("2006-01-02") + " " + "20:36"
timezone := "Asia/Tokyo"

loc, err := time.LoadLocation(timezone)
if err != nil {
    log.Fatal(err)
}
t, err := time.ParseInLocation("2006-01-02 15:04", timeTest, loc)
if err != nil {
    log.Fatal(err)
}
fmt.Println(t.Format("15:04"))
fmt.Println(t.UTC().Format("15:04"))

Comapare

After / Before

time1 := "2016-05-15 12:22:00"
time2 := "2017-05-15 12:22:00"
t1, err := time.Parse("2006-01-02 15:04:05", time1)
t2, err := time.Parse("2006-01-02 15:04:05", time2)
if err == nil && t1.Before(t2) {
    fmt.Println("true")
}

延遲

time.Sleep(3 * time.Second)

time.Sleep(500 * time.Millisecond)

speed := 500
time.Sleep(time.Duration(speed) * time.Millisecond)

時間相加／相減

兩時間相減

startTime := time.Now()                     // 2014-01-26 18:17:22.185125 +0800 CST
time.Sleep(3 * time.Second)
endTime := time.Now()                       // 2014-01-26 18:17:25.186307 +0800 CST

var durationTime time.Duration = endTime.Sub(startTime)
fmt.Println(durationTime.String())

加減(日時秒)

timein := time.Now().Add(time.Hour * time.Duration(1))      // Add 1 hour
timein := time.Now().Add(time.Minute * time.Duration(1))    // Add 1 minute
timein := time.Now().Add(time.Second * time.Duration(1))    // Add 1 second

// 減的話一樣是用 Add 但在 Duration 上加上負號
timein := time.Now().Add(time.Minute * time.Duration(-10))


then := time.Now().AddDate(0, 0, 1)     // +1 day
then := time.Now().AddDate(0, 0, -1)    // -1 day
then := time.Now().AddDate(1, 1, 0)     // +1 year and 1 month

timestamp plus 30 seconds

time.Now().Unix() + 30

Week

time.Now().Weekday()            // Tuesday
int(time.Now().Weekday())       // 2 (Day of the week)

相減算時間差

time_start := time.Now()
// do something
log.Printf("elasped time: %s", time.Since(time_start))

3 * time.Second

如果前面的 int 從 string 轉型，是會噴錯的，要改成 :

time.Duration(params["idle_timeout"].(int)) * time.Second

印出到小數的 timestamp/datetime (e.g. `2017-06-28 02:58:46.452 +0000 UTC`)

func main() {
        fmt.Println(unixMilli(time.Unix(0, 123400000)))
        fmt.Println(unixMilli(time.Unix(0, 123500000)))
        m := makeTimestampMilli()
        fmt.Println(m)
        fmt.Println(time.Unix(m/1e3, (m%1e3)*int64(time.Millisecond)/int64(time.Nanosecond)))
}

func unixMilli(t time.Time) int64 {
        return t.Round(time.Millisecond).UnixNano() / (int64(time.Millisecond) / int64(time.Nanosecond))
}

func makeTimestampMilli() int64 {
        return unixMilli(time.Now())
}

Result:

123
124
1498618861845
2017-06-28 03:01:01.845 +0000 UTC

Timestamp with three decimal places

fmt.Sprintf("%f", float64(time.Now().UnixNano())/1000000000) // 1483004003.785

math/rand

產生 0~7 (每一次產生不一樣的變數)

rand.Seed(time.Now().UnixNano())
r := rand.Intn(8)

等於

r := rand.New(rand.NewSource(time.Now().UTC().UnixNano()))
f := r.Intn(8)

產生一串隨機數值 (ex: 7572000213564998818)

r := rand.New(rand.NewSource(time.Now().UTC().UnixNano()))
f := r.Int63()

Unique key

32 bytes HEX

key := make([]byte, 16)       // 16 會產生 32 個字元
_, err := rand.Read(key)
if err != nil {
    log.Fatal(err)
}
mk := fmt.Sprintf("%X", key)  //小寫
mk := fmt.Sprintf("%X", key)  // 大寫
fmt.Println(mk)

generate 32 bytes authentication key

import (
    "crypto/rand"
    "encoding/base64"
    "fmt"
    "log"
)

func GenerateRandomBytes(n int) ([]byte, error) {
    b := make([]byte, n)
    _, err := rand.Read(b)
    // Note that err == nil only if we read len(b) bytes.
    if err != nil {
        return nil, err
    }

    return b, nil
}

func GenerateRandomString(s int) (string, error) {
    b, err := GenerateRandomBytes(s)
    return base64.URLEncoding.EncodeToString(b), err
    // or
    // return hex.EncodeToString(b), err
}
func main() {
    // Example: this will give us a 44 byte, base64 encoded output
    token, err := GenerateRandomString(32)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(token)
}

sync

pool 特性

臨時暫存的 pool (有點像 cache 的感覺)
GC 會把 pool 清空, 也因為如此, 不要用 pool 來當 connection pool (TCP, DB, Redis, etc.)
A Pool is safe for use by multiple goroutines simultaneously.
減少 GC 成本, 提高效能
a dynamically-sized store of temporary output buffers, 會根據壓力下自已變大縮小

pool 是會被GC的

p := &sync.Pool{
    New: func() interface{} {
        var i interface{}
        return i
    },
}
p.Put(1)
p.Put(2)

a := p.Get()
b := p.Get()
p.Put(a)
p.Put(b)
fmt.Println("Before GC:", a, b) // 1,2
runtime.GC()

a = p.Get()
b = p.Get()
p.Put(a)
p.Put(b)
fmt.Println("After GC: ", a, b) // nil, nil

result:

Before GC: 1 2
After GC:  <nil> <nil>

Pool (First In, First out)

var pool sync.Pool
type Item struct {
    Order string
}
v := pool.Get() // You get nothing if pool is empty.
if v == nil {
    v = &Item{Order: "first"}
}
pool.Put(v)                      // Put the first item
pool.Put(&Item{Order: "second"}) // Put the second item
q := pool.Get()                  // You'll get the first item
fmt.Println(q.(*Item).Order)     // "first"
q = pool.Get()                   // You'll get the second item
fmt.Println(q.(*Item).Order)     // "second"

Pool 乎不會比較快 (或許 example 寫的不夠複雜)

// Pool for our struct A
var pool *sync.Pool

// A dummy struct with a member
type A struct {
    Name string
}

// Func to init pool
func initPool() {
    pool = &sync.Pool{
        New: func() interface{} {
            return new(A)
        },
    }
}

// Main func
func main() {
    count := 100000
    fmt.Println("count: ", count)

    // Initializing pool
    initPool()
    for i := 1; i < 5; i++ {
        d := pool.New().(*A)
        d.Name = "Init00" + strconv.Itoa(i)
        // fmt.Printf("%d -> d.Name = %s\n", i, d.Name)
        pool.Put(d)
    }
    var wg sync.WaitGroup
    a := time.Now()
    for i := 0; i < count; i++ {
        wg.Add(1)
        go func(i int) {
            d := pool.Get().(*A)
            if d.Name == "" {
                d.Name = "NewXX" + strconv.Itoa(i)
            } else {
                d.Name = d.Name + "-" + strconv.Itoa(i)
            }
            _ = d
            // fmt.Printf("%d -> d.Name = %s\n", i, d.Name)
            pool.Put(d)
            defer wg.Done()
        }(i)
    }
    wg.Wait()
    fmt.Println("with pool: ", time.Since(a))

    var wg2 sync.WaitGroup
    b := time.Now()
    for i := 0; i < count; i++ {
        wg2.Add(1)
        go func(i int) {
            d := A{Name: "NewXX" + strconv.Itoa(i)}
            _ = d
            // fmt.Printf("%d -> d.Name = %s\n", i, d.Name)
            defer wg2.Done()
        }(i)
    }
    wg2.Wait()
    fmt.Println("without pool: ", time.Since(b))
}

result:

count:  100000
with pool:  150.17869ms
without pool:  40.754878ms

Sync WaitGroup

當同一時間同步做很多事，但希望全部 goroutine 做完事才進行下一步

package main

import (
    "fmt"
    "sync"
)

func main() {
    var wg sync.WaitGroup
    for q := 0; q < 10; q++ {
        wg.Add(1)               # flag
        go func(i int) {
            fmt.Println(i)
            defer wg.Done()     # 通報這個 goroutine 做完了
        }(q)
    }
    wg.Wait()                   # 程式會在這裡等全部 goroutine 做完
}

Mutex 與 RWMutex 差別

Mutex

A global lock.
It only allows one R or W at once.

RWMutex

The lock can be held by an arbitrary number of readers or a single writer.
Readers don’t have to wait for each other. They only have to wait for writers holding the lock.
Preferable for data that is mostly read.
There is no WLock(); instead, use Lock()
RLock() will only be blocked by Lock()

Mutex

a := 0
var wg sync.WaitGroup
for i := 0; i < 10000; i++ {
    wg.Add(1)
    go func() {
        a++
        defer wg.Done()
    }()
}
wg.Wait()
fmt.Printf("a = %d\n", a)     // a = 9222  < 10000 because of race condition

use -race to verify

 ==================
WARNING: DATA RACE
Read at 0x00c420084008 by goroutine 7:
  main.main.func1()
      /tmp/mutex.go:17 +0x3f

Previous write at 0x00c420084008 by goroutine 6:
  main.main.func1()
      /tmp/mutex.go:17 +0x58

Goroutine 7 (running) created at:
  main.main()
      /tmp/mutex.go:15 +0xfb

Goroutine 6 (finished) created at:
  main.main()
      /tmp/mutex.go:15 +0xfb
==================
a = 9978
Found 1 data race(s)
exit status 66

use lock

a := 0
var wg sync.WaitGroup
var l sync.Mutex
for i := 0; i < 10000; i++ {
    wg.Add(1)
    go func() {
        l.Lock()
        a++
        l.Unlock()
        defer wg.Done()
    }()
}
wg.Wait()
fmt.Printf("a = %d\n", a)      // a = 10000

sync/atomic

var atomicCounter uint64
var safeCounter int64
var unsafeCounter int64
var l sync.Mutex

for i := 0; i < 5000; i++ {
    go func() {
        atomic.AddUint64(&atomicCounter, 1)
    }()
    go func() {

        unsafeCounter++
    }()
    go func() {
        l.Lock()
        defer l.Unlock()
        safeCounter++
    }()
}

time.Sleep(time.Second)

fmt.Println("atomic counter:", atomic.LoadUint64(&atomicCounter))
fmt.Println("unsafe counter:", unsafeCounter)
fmt.Println("safe counter:", safeCounter)

output:

atomic counter: 5000
unsafe counter: 4924
safe counter: 5000

sync/once

example

package main

import (
    "fmt"
    "sync"
)

func main() {
    var once sync.Once
    onceBody := func() {
        fmt.Println("Only once")
    }
    done := make(chan bool)
    for i := 0; i < 10; i++ {
        go func() {
            once.Do(onceBody)
            done <- true
        }()
    }
    for i := 0; i < 10; i++ {
        <-done
    }
}

result:

Only once

encoding

組出 SOAP

package main

import "fmt"
import "encoding/xml"

type MyRespEnvelope struct {
    XMLName xml.Name
    Body    Body
}

type Body struct {
    XMLName     xml.Name
    GetResponse completeResponse `xml:"activationPack_completeResponse"`
}

type completeResponse struct {
    XMLName xml.Name `xml:"activationPack_completeResponse"`
    Id      string   `xml:"Id,attr"`
    MyVar   string   `xml:"activationPack_completeResult"`
}

func main() {

    Soap := []byte(`<?xml version="1.0" encoding="UTF-8"?>
                    <soap:Envelope SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/">
                    <soap:Body>
                    <activationPack_completeResponse Id="http://tempuri.org/">
                    <activationPack_completeResult xsi:type="xsd:string">Active</activationPack_completeResult>
                    </activationPack_completeResponse>
                    </soap:Body>
                    </soap:Envelope>`)

    res := &MyRespEnvelope{}
    if err := xml.Unmarshal(Soap, res); err != nil {
        fmt.Println(err.Error())
    }

    var val completeResponse = res.Body.GetResponse
    fmt.Println(val.MyVar)
}

ref : http://play.golang.org/p/957GWzfdvN

Parse xml into struct

Person.xml :

<Person>
    <FullName>Grace R. Emlin</FullName>
    <Company>Example Inc.</Company>
    <Email where="home">
        <Addr>gre@example.com</Addr>
    </Email>
    <Email where='work'>
        <Addr>gre@work.com</Addr>
    </Email>
    <Group>
        <Value>Friends</Value>
        <Value>Squash</Value>
    </Group>
    <City>Hanga Roa</City>
    <State>Easter Island</State>
</Person>

code :

package main

import (
    "encoding/xml"
    "fmt"
    "io/ioutil"
    "os"
)

type Email struct {
    Where string `xml:"where,attr"`
    Addr  string
}

type Address struct {
    City, State string
}

type Result struct {
    XMLName xml.Name `xml:"Person"`
    Name    string   `xml:"FullName"`
    Phone   string
    Email   []Email
    Groups  []string `xml:"Group>Value"`
    Address
}

func main() {

    var v Result
    xmlFile1, err := ioutil.ReadFile("Person.xml")
    if err != nil {
        fmt.Println("Error opening file: ", err)
        return
    }

    err1 := xml.Unmarshal(xmlFile1, &v)
    if err1 != nil {
        fmt.Printf("error: %v", err)
        return
    }
    fmt.Println(v)

    // 用 streaming 形式解析, 較容易處理大數據的 xml
    xmlFile, err := os.Open("Person.xml")
    if err != nil {
        fmt.Println("Error opening file: ", err)
        return
    }
    defer xmlFile.Close()
    decoder := xml.NewDecoder(xmlFile)
    for {
        t, _ := decoder.Token()
        if t == nil {
            break
        }
        switch se := t.(type) {
        case xml.StartElement:
            if se.Name.Local == "Person" {
                var d Result
                decoder.DecodeElement(&d, &se)
                fmt.Println(d)
            }
        }
    }

    fmt.Println(v.XMLName)
    fmt.Printf("Name=%s \n", v.Name)
    fmt.Printf("Where=%s   Addr=%s  \n", v.Email[0].Where, v.Email[0].Addr)
    fmt.Printf("Where=%s   Addr=%s  \n", v.Email[1].Where, v.Email[1].Addr)
    fmt.Printf("Groups : %s,  %s\n", v.Groups[0], v.Groups[1])
    fmt.Printf("City=%s,  State=%s  \n", v.Address.City, v.Address.State)
}

ref : http://www.cnblogs.com/yuanershi/archive/2013/01/29/2881192.html

Parse xml into map - clbanning/mxj

var xml = `
    <Person>
        <FullName>Grace R. Emlin</FullName>
        <Company>Example Inc.</Company>
        <Email where="home">
            <Addr>gre@example.com</Addr>
        </Email>
        <Email where='work'>
            <Addr>gre@work.com</Addr>
        </Email>
        <Group>
            <Value>Friends</Value>
            <Value>Squash</Value>
        </Group>
        <City>Hanga Roa</City>
        <State>Easter Island</State>
    </Person>`
res, err := mxj.NewMapXml([]byte(xml))
if err != nil {
    fmt.Println(err)
}
fmt.Println(res)
fmt.Println(res["Person"].(interface{}).(map[string]interface{})["FullName"])
fmt.Println(res["Person"].(interface{}).(map[string]interface{})["Email"].([]interface{})[1].(map[string]interface{})["Addr"])

result :

map[Person:map[FullName:Grace R. Emlin Company:Example Inc. Email:[map[-where:home Addr:gre@example.com] map[Addr:gre@work.com -where:work]] Group:map[Value:[Friends Squash]] City:Hanga Roa State:Easter Island]]
Grace R. Emlin
gre@work.com

Parse json (struct, map)

方法1 : Json 放進預先定義好的 Struct 裡

type Test struct {
    JsonTag string `json:"field1"` // 利用 Tag mapping 到欄位名稱
    Field2  []struct {
        SubField1 string
        SubField2 string
    }
}

res := &Test{}
if err := json.Unmarshal([]byte(jsonString), res); err != nil {
    fmt.Println(err)
}
fmt.Println(res.JsonTag)
fmt.Println(res.Field2[0].SubField1)
fmt.Println(res.Field2[1].SubField2)

方法2 : 以 map 方式將 json 讀進來

var res2 map[string]interface{}
if err := json.Unmarshal([]byte(jsonString), &res2); err != nil {
    fmt.Println(err)
}
fmt.Println(res2["field1"])
fmt.Println(res2["field2"].([]interface{})[0].(map[string]interface{})["subField1"])
fmt.Println(res2["field2"].([]interface{})[1].(map[string]interface{})["subField2"])

json unmarshal 預設數字讓它是 int64 而不是 float64

json 字串裡面有 "visible_at":1483079819，但 unmarshal 完 int 變成 1.483079819e+09

解決方法 :

var data = `{
    "id": 12423434,
    "Name": "Fernando"
}`
d := json.NewDecoder(strings.NewReader(data))
d.UseNumber()
var x interface{}
if err := d.Decode(&x); err != nil {
    log.Fatal(err)
}
fmt.Printf("decoded to %#v\n", x)

base64 加解密

data := "abc123!?$*&()'-=@~"

sEnc := base64.StdEncoding.EncodeToString([]byte(data))
fmt.Println(sEnc)           # YWJjMTIzIT8kKiYoKSctPUB+

sDec, _ := base64.StdEncoding.DecodeString(sEnc)
fmt.Println(string(sDec))   # abc123!?$*&()'-=@~

POST base64 with form-data

body := fmt.Sprintf("p=%s&iv=%s", url.QueryEscape(crypt.Base64Encode(p_bytes)), url.QueryEscape(crypt.Base64Encode(iv_bytes)))

json.Marshal 不要 escape 某些字元

當欄位裡面有 url 時，json marshal 會自動地將某些字元(e.g. &) escape 為 \\u0026，為了避免此情形改用 :

type Search struct {
        Query string `json:"query"`
}
data := &Search{Query: "http://google.com/?q=stackoverflow&ie=UTF-8"}
buf := new(bytes.Buffer)
enc := json.NewEncoder(buf)
enc.SetEscapeHTML(false)
err = enc.Encode(data)
fmt.Println(string(buf.Bytes()))            // 注意 Encode 會在字尾加上 `\n`

[output]
    {"query":"http://google.com/?q=stackoverflow&ie=UTF-8"}

或用取代的方式將被脫逸的符號還原

b, err := json.Marshal(v)
b = bytes.Replace(b, []byte("\\u003c"), []byte("<"), -1)
b = bytes.Replace(b, []byte("\\u003e"), []byte(">"), -1)
b = bytes.Replace(b, []byte("\\u0026"), []byte("&"), -1)

Cryptography

md5

h := md5.New()
io.WriteString(h, "Hello World!")
fmt.Printf("%x", h.Sum(nil))

hasher := md5.New()
hasher.Write([]byte("test"))
fmt.Println(hex.EncodeToString(hasher.Sum(nil)))

h := md5.Sum([]byte("test"))
fmt.Println(hex.EncodeToString(h[:]))

sha256

方法1

hash := sha256.New()
hash.Write([]byte("test"))
b := hash.Sum(nil)  // byte
fmt.Println(hex.EncodeToString(b[:]))

方法2

b := sha256.Sum256([]byte("test"))
fmt.Println(hex.EncodeToString(b[:]))

RSA (Asymmetric Encryption)

encrypt plaintext with public key and decrypt ciphertext with private key

// Generate a new private key.
privateKey, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
    panic(err)
}

publicKey := &privateKey.PublicKey

plaintext := []byte("Hello, World!")

// Encrypt the plaintext with pulbic key
ciphertext, err := rsa.EncryptOAEP(
    sha256.New(),
    rand.Reader,
    publicKey,
    plaintext,
    nil)
if err != nil {
    panic(err)
}
fmt.Printf("plaintext: %s\n\n", string(plaintext))
fmt.Printf("encrypt plaintext with public key:\n%x\n\n", ciphertext)

// Decrypt the ciphertext with private key
decryptedPlaintext, err := rsa.DecryptOAEP(
    sha256.New(),
    rand.Reader,
    privateKey,
    ciphertext,
    nil)
if err != nil {
    panic(err)
}
fmt.Printf("decrypt ciphertext with private key: %s\n\n", string(decryptedPlaintext))

generate a signature for plaintext with private key and verify signature with public key

// Sign the message using the private key.
hash := sha256.New()
hash.Write(plaintext)
digest := hash.Sum(nil)

signature, err := rsa.SignPKCS1v15(rand.Reader, privateKey, crypto.SHA256, digest)
if err != nil {
    panic(err)
}

fmt.Printf("generate a signature for plaintext with private key:\n%x\n\n", signature)

// Verify the signature using the public key.
err = rsa.VerifyPKCS1v15(publicKey, crypto.SHA256, digest, signature)
if err != nil {
    fmt.Println("Verification failed (with public key)")
} else {
    fmt.Println("Verification successful (with public key)")
}

output:

plaintext: Hello, World!

encrypt plaintext with public key:
5175a5a4f0a560d816000baf8872e7935d00cb45f9d092ca86f741c0315dc56090c01e7baf5ab840c14f06107161a32417253126fadc336e24b2d7ff1af85d3f4ee9a15dc466e6e6dcac996903ed471ed9eccca5a82ffe9d81540c6701d0b161175eb0a9efc5bcdfbcf244ad1a3bfc922fc00298c0beb3883767094abe9924afafdd13d7fcc1a0ad4bee3244a099d7f9fd30544024a2f4349ed5e6b0450e307df35d0c11a57189020d877b0a273492b9bc47d72d11a38e27a53c41f5c7c2b7882f2b77a52151afffecb85d0d6c95758f1cfa77153b8c96ee2a63e92754aa720cebc69be9b071bd085be1697b8e6770905d2202eefd38fa33a2eb2293ffd6722f

decode ciphertext with private key: Hello, World!

generate a signature for plaintext with private key:
7a1bc8da55f0e9dd6ddc73a9480c68914f30bd67af515698c2c0be75b2d5e11655073e95f776be4418ec1782a322cec99f005766c54d94b9d19b70202b8de7b2a20bb435f4a4da6e7a5309d97aa16fa09e86c0e8cb7f51717d0ec398a651134708d770bafd7dc21c69d4f6b3f9efb3e985c5905cb23fcc6fc5f2227c770e5d330fbe175a5cbcf0dd7e599c79ec0d49adbeb5fbb3bc1ad329db40cba5bb03b392640840b4484f8e52b45ad202c1991d997789bbd0a1b95d916e7653faf685ee349e5a80faf4af02fbe18daa971363d2c3c22a11eca52c9205336c5378299d44dea94fb3b948f65cde52d88319cbc6ad45c688e7db97af738a5017ca703e06fbfb

Verification successful (with public key)

AES CBC (Symmetric Encryption)

encrypt

import (
    "crypto/aes"
    "crypto/cipher"
    "crypto/rand"
    "encoding/base64"
    "fmt"
    "io"
)

func main() {
    key := []byte("1234567890123456")
    plaintext := pad([]byte("secret data12345"))

    if len(plaintext)%aes.BlockSize != 0 {
        panic("plaintext is not a multiple of the block size")
    }

    block, err := aes.NewCipher(key)
    if err != nil {
        panic(err)
    }

    ciphertext := make([]byte, aes.BlockSize+len(plaintext))
    iv := ciphertext[:aes.BlockSize]
    if _, err := io.ReadFull(rand.Reader, iv); err != nil {
        panic(err)
    }

    mode := cipher.NewCBCEncrypter(block, iv)
    mode.CryptBlocks(ciphertext[aes.BlockSize:], plaintext)
    fmt.Println(base64.StdEncoding.EncodeToString(iv))              // iv base64: VGnqCX2VQonnNUL/Hlmk1w==
    fmt.Println(base64.StdEncoding.EncodeToString(ciphertext[16:])) // encrypt base64: 5ThQKq6RnAiGsLHU/BWm7A==
}

// plaintext 無法被 aes.Blocksize (16) 要先補位數湊滿
func pad(in []byte) []byte {
    padding := aes.BlockSize - (len(in) % aes.BlockSize)
    if padding == 0 {
        padding = aes.BlockSize
    }
    for i := 0; i < padding; i++ {
        in = append(in, byte(padding))
    }
    return in
}

另種寫法

text := []byte(`secret data12345`)
sharekey := []byte(`1234567890123456`)
block, err := aes.NewCipher(sharekey)
if err != nil {
    panic(errors.New("AESEncrypt Create Cipher error: " + err.Error()))
}

paddingText := pad(text, block.BlockSize())
iv := make([]byte, aes.BlockSize)
if _, err := io.ReadFull(rand.Reader, iv); err != nil {
    panic(err)
}

mode := cipher.NewCBCEncrypter(block, iv)
mode.CryptBlocks(paddingText, paddingText)
fmt.Println(base64.StdEncoding.EncodeToString(iv))
fmt.Println(base64.StdEncoding.EncodeToString(paddingText))

// 補滿 16 位
func pad(src []byte, blocksize int) []byte {
    pdSize := blocksize - (len(src) % blocksize)
    padBytes := bytes.Repeat([]byte{0x00}, pdSize)
    src = append(src, padBytes...)
    return src
}

The IV can be saved (or can even be “exposed”/transmitted publicly); it is no ‘secret’. The key should be kept private. So you could save/transmit your data like <iv_here>;<encrypted_data_here>.

decrypt

import (
    "crypto/aes"
    "crypto/cipher"
    "encoding/base64"
    "fmt"
)

func main() {
    key := []byte("1234567890123456")
    iv, _ := base64.StdEncoding.DecodeString("7/JbVL/5cMvqf9sslD6qdQ==")                // iv base64
    ciphertext, _ := base64.StdEncoding.DecodeString("6AFagV3iLEAqbBuSqvL19Q==")        // encrypt base64

    block, _ := aes.NewCipher(key)
    if len(ciphertext) < aes.BlockSize {
        panic("ciphertext too short")
    }
    if len(ciphertext)%aes.BlockSize != 0 {
        panic("ciphertext is not a multiple of the block size")
    }
    aes := cipher.NewCBCDecrypter(block, iv)
    aes.CryptBlocks(ciphertext, ciphertext)
    fmt.Printf("%s\n", ciphertext)
}

String 相同, 同長度及 byte 不同 `invalid character '\x00' after top-level value`

可能是 AES 加解密後發生的情況，可能在過程中被塞入 padding, 導致解出來後的 byte 數量不一樣

加密前 :

2561953d9ec389715498d44b0c150fec
len = 32
[50 53 54 49 57 53 51 100 57 101 99 51 56 57 55 49 53 52 57 56 100 52 52 98 48 99 49 53 48 102 101 99]

加密後 :

len = 48
[50 53 54 49 57 53 51 100 57 101 99 51 56 57 55 49 53 52 57 56 100 52 52 98 48 99 49 53 48 102 101 99 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

ASCII 對應

50 => 2
99 => c
0 => 空字元（Null）

解決方法: 將多餘的 \x00 刪除

b = bytes.Trim(b, "\x00")

ref: 類似問題

UUID (crypto/rand)

// 一般
uuid := make([]byte, 16)
io.ReadFull(rand.Reader, uuid)

// 64 encode
uuid := Gen()
ret := base64.URLEncoding.EncodeToString(uuid)
ret = strings.Replace(ret, "=", "", -1)

import uuid "github.com/satori/go.uuid"
uuid.NewV4()

defer

關於 defer 如何運作

package main

import (
    "fmt"
)

func main() {
    defer Function3()
    Function1()
}

func Function1() {
    fmt.Println("Function1 開始")
    defer Function2()
    fmt.Println("Function1 結束")
}

func Function2() {
    fmt.Println("Function 2")
}

func Function3() {
    fmt.Println("Function 3")
}


執行結果 :

Function1 開始
Function1 結束
Function 2
Function 3

ref : http://ithelp.ithome.com.tw/question/10153473

error

擲出自定義錯誤訊息

err = errors.New("fail")

比對 error

在另一個 package 用全域變數定義

package fruits

var NoMorePumpkins = errors.New("No more pumpkins")

就可以比對了

package shop

if err == fruits.NoMorePumpkins {
     ...
}

runtime

detect OS

switch runtime.GOOS {
    case "windows":
    case "darwin":       //Mac OS
    case "linux":
    default :
}

記憶體使用量

var m0 runtime.MemStats
runtime.ReadMemStats(&m0)
fmt.Printf("Memory: %.2f mb", float64(m0.Sys)/1024/1024)

Cpu Usage (沒用到 runtime package)

func getCPUSample() (idle, total uint64) {
    contents, err := ioutil.ReadFile("/proc/stat")
    if err != nil {
        return
    }
    lines := strings.Split(string(contents), "\n")
    for _, line := range lines {
        fields := strings.Fields(line)
        if fields[0] == "cpu" {
            numFields := len(fields)
            for i := 1; i < numFields; i++ {
                val, err := strconv.ParseUint(fields[i], 10, 64)
                if err != nil {
                    fmt.Println("Error: ", i, fields[i], err)
                }
                total += val // tally up all the numbers to get total ticks
                if i == 4 {  // idle is the 5th field in the cpu line
                    idle = val
                }
            }
            return
        }
    }
    return
}

func main() {
    idle0, total0 := getCPUSample()
    time.Sleep(3 * time.Second)
    idle1, total1 := getCPUSample()

    idleTicks := float64(idle1 - idle0)
    totalTicks := float64(total1 - total0)
    cpuUsage := 100 * (totalTicks - idleTicks) / totalTicks

    fmt.Printf("CPU usage is %f%% [busy: %f, total: %f]\n", cpuUsage, totalTicks-idleTicks, totalTicks)
}

channel

把 Channel 當成 Queue 使用

它是先進先出，如果要做一個 worker 很好用，直接宣告它的大小是多少，但塞值不需要管它的 index，一直丟就對了

var worker_ch = make(chan string, 10000)

// 用 Goroutine 跑，一個 Worker，它會一直 wait，直到 worker_ch 有值
for {
    file_name := <-worker_ch
    DoSomething(file_name)
}

// 另一個 Goroutine，假設它是 API 負責塞值給 worker_ch
func NonBlcok(ctx *fasthttp.RequestCtx, _ fasthttprouter.Params) {
    file_name := string(ctx.FormValue("filename"))
    worker_ch <- file_name
}

用 channel & select 實作 long lived, concurrent safe pools

// Pool holds Clients.
type Pool struct {
    pool   chan *Client
}

// NewPool creates a new pool of Clients.
func NewPool(max int) *Pool {
    return &Pool{
        pool:   make(chan *Client, max),
    }
}

// Borrow a Client from the pool.
func (p *Pool) Borrow() *Client {
    var c *Client
    select {
    case c = <-p.pool:
    default:
        c = newClient()
    }
    return c
}

// Return returns a Client to the pool.
func (p *Pool) Return(c *Client) {
    select {
    case p.pool <- c:
    default:
        // let it go, let it go...
    }
}

reflect

Call dynamic func by name

func Call(m map[string]interface{}, name string, params ...interface{}) (result []reflect.Value, err error) {
    f := reflect.ValueOf(m[name])
    if len(params) != f.Type().NumIn() {
        err = errors.New("The number of params is not adapted.")
        return
    }
    in := make([]reflect.Value, len(params))
    for k, param := range params {
        in[k] = reflect.ValueOf(param)
    }
    result = f.Call(in)
    return
}

用法 :

// 宣告
funcs := map[string]interface{}{
    "qq": curl.Get,  // index 隨便取, 後面的是 func name  e.g. func QQ() 就填  QQ, 如果是 curl pacakge 的 Get func 就填 curl.Get
}
Call(funcs, "qq", params)  // 如果有參數就傳入 params

Call dynamic struct.function by name (string)

import "fmt"
import "reflect"

type T struct{}

func (t *T) Test() string {
    fmt.Println("test...")
    return "ok"
}

func main() {
    var t T
    res := reflect.ValueOf(&t).MethodByName("Test").Call([]reflect.Value{})
    fmt.Println(res[0]) // ok
}

res 是一個陣列，即使你的 func 只回傳一個參數，它也是放在陣列裡

Call dynamic function by interface

value := reflect.ValueOf(jb)                // jb can be a struct which implemented an interface
method := value.MethodByName("Done")        // func name
if method.IsValid() {
    if method.IsValid() {
        method.Call([]reflect.Value{})
    }
}

Good example of call dynamic function by either value or pointer

type Test struct {
    Start string
}

// value receiver
func (t Test) Finish() string {
    return t.Start + "finish"
}

// pointer receiver
func (t *Test) Another() string {
    return t.Start + "another"
}

func CallMethod(i interface{}, methodName string) interface{} {
    var ptr reflect.Value
    var value reflect.Value
    var finalMethod reflect.Value

    value = reflect.ValueOf(i)

    // if we start with a pointer, we need to get value pointed to
    // if we start with a value, we need to get a pointer to that value
    if value.Type().Kind() == reflect.Ptr {
        ptr = value
        value = ptr.Elem()
    } else {
        ptr = reflect.New(reflect.TypeOf(i))
        temp := ptr.Elem()
        temp.Set(value)
    }

    // check for method on value
    method := value.MethodByName(methodName)
    if method.IsValid() {
        finalMethod = method
    }
    // check for method on pointer
    method = ptr.MethodByName(methodName)
    if method.IsValid() {
        finalMethod = method
    }

    if (finalMethod.IsValid()) {
        return finalMethod.Call([]reflect.Value{})[0].Interface()
    }

    // return or panic, method not found of either type
    return ""
}

func main() {
    i := Test{Start: "start"}
    j := Test{Start: "start2"}

    fmt.Println(CallMethod(i, "Finish"))
    fmt.Println(CallMethod(&i, "Finish"))
    fmt.Println(CallMethod(i, "Another"))
    fmt.Println(CallMethod(&i, "Another"))
    fmt.Println(CallMethod(j, "Finish"))
    fmt.Println(CallMethod(&j, "Finish"))
    fmt.Println(CallMethod(j, "Another"))
    fmt.Println(CallMethod(&j, "Another"))
}

Result

startfinish
startfinish
startanother
startanother
start2finish
start2finish
start2another
start2another

ref: https://stackoverflow.com/questions/14116840/dynamically-call-method-on-interface-regardless-of-receiver-type

flag

auto flag

config := struct {
    Name    string        `flag:"queue,queue name"`
}{
    Name:    "",
}
autoflags.Define(&config)
flag.Parse()

image

Get image’s width and height

import (
    "fmt"
    "image"
    _ "image/jpeg"
    "io/ioutil"
    "os"
    "path/filepath"
)

const dir_to_scan string = "/tmp/images"

func main() {
    files, _ := ioutil.ReadDir(dir_to_scan)
    for _, imgFile := range files {

        if reader, err := os.Open(filepath.Join(dir_to_scan, imgFile.Name())); err == nil {
            defer reader.Close()
            im, _, err := image.DecodeConfig(reader)
            if err != nil {
                fmt.Fprintf(os.Stderr, "%s: %v\n", imgFile.Name(), err)
                continue
            }
            fmt.Printf("%s %d %d\n", imgFile.Name(), im.Width, im.Height)
        } else {
            fmt.Println("Impossible to open the file:", err)
        }
    }
}

image/jpg Convert jpg to png

file, err := os.Open("test.jpg")
if err != nil {
    log.Fatal("File error")
}

img, err := jpeg.Decode(file)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}

out, err := os.Create("test.png")
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
err = png.Encode(out, img)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
fmt.Println("Done!")

image/jpg Convert jpg to bmp

// Convert jpg to bmp
img_file, err := os.Open("test.jpg")
if err != nil {
    log.Fatal("File error")
}
defer img_file.Close()

img, err := jpeg.Decode(img_file)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
out_file, err := os.Create("test.bmp")
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
err = bmp.Encode(out_file, img)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
defer out_file.Close()

image/jpg Convert bmp to jpg

// Convert jpg to bmp
img_file, err := os.Open("test.bmp")
if err != nil {
    log.Fatal("File error")
}
defer img_file.Close()

img, err := bmp.Decode(img_file)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
out_file, err := os.Create("test.jpg")
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}

options := &jpeg.Options{Quality: 50}
err = jpeg.Encode(out_file, img, options)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}
defer out_file.Close()

convert an image to a black and white image

import (
    "fmt"
    "image"
    "image/draw"
    "image/jpeg"
    "os"
)

func main() {
    file, err := os.Create("test-result.jpg")
    if err != nil {
        fmt.Println(err)
    }
    defer file.Close()

    file1, err := os.Open("test.jpg")
    if err != nil {
        fmt.Println(err)
    }
    defer file1.Close()
    img, _ := jpeg.Decode(file1)

    jpg := image.NewGray(img.Bounds()) //NewGray

    draw.Draw(jpg, jpg.Bounds(), img, img.Bounds().Min, draw.Src)

    jpeg.Encode(file, jpg, nil)

}

Draw a line on an image

import (
    "fmt"
    "image"
    "image/color"
    "image/draw"
    "image/jpeg"
    "os"
)

func main() {
    file, err := os.Create("test2.jpg")
    if err != nil {
        fmt.Println(err)
    }
    defer file.Close()

    file1, err := os.Open("test.jpg")
    if err != nil {
        fmt.Println(err)
    }
    defer file1.Close()
    img, _ := jpeg.Decode(file1)

    // Create an image with size same as the original image
    jpg := image.NewRGBA(img.Bounds())    // or you can specify the size: image.NewRGBA(image.Rect(0, 0, 1080, 1080))

    // Put the original image into to the image that we just created
    draw.Draw(jpg, jpg.Bounds(), img, img.Bounds().Min, draw.Over)

    // Draw a red dot at (2, 3)
    for x := 1; x < 100; x++ {
        y := x
        jpg.Set(x, y, color.RGBA{255, 0, 0, 255})
    }

    // jpeg.Encode(file, jpg, nil)
    jpeg.Encode(file, jpg, &jpeg.Options{Quality: 100})
}

unsafe

看變數佔多數記憶體 (bytes)

a1 := "xxxdflasjdfl;daskjfdsalfkjasdlfjas"
a2 := int64(66666)
a3 := int32(5555)
a4 := float32(2222.4)
fmt.Println(unsafe.Sizeof(a1))  // 16
fmt.Println(unsafe.Sizeof(a2))  // 8
fmt.Println(unsafe.Sizeof(a3))  // 4
fmt.Println(unsafe.Sizeof(a4))  // 4

utf8

Count size of UTF8 string

utf8.RuneCountInString("世界")    // 2

read unicode characters

s := "Hello, 世界!"

for i := 0; i < len(s); {
    char, width := utf8.DecodeRuneInString(s[i:])
    fmt.Printf("Character at byte index %d: %c (Unicode: %U) %d\n", i, char, char, width)
    i += width // move forward by the number of bytes read
}

reflect

Implementation of generics

type Animal interface{}

type Body struct {
    FeetCount int64
}

type People struct {
    Name string
    Body
}

type Cat struct {
    Name string
    Body
}

func main() {
    var h People
    var c Cat
    setName(&h, "People") // Must be passed by pointer
    setFeetCount(&h, 2)
    fmt.Println(h)

    setName(&c, "Cat")
    setFeetCount(&c, 4)
    fmt.Println(c)
}

// Set string
func setName(a Animal, name string) {
    o := reflect.ValueOf(a).Elem().FieldByName("Name")
    if o.CanSet() {
        reflect.ValueOf(a).Elem().FieldByName("Name").SetString(name)
        fmt.Println(reflect.ValueOf(a).Elem().FieldByName("Name").String())
    }
}

// Set struct
func setFeetCount(a Animal, c int64) {
    o := reflect.ValueOf(a).Elem().FieldByName("Body")
    if o.CanSet() {
        o.FieldByName("FeetCount").SetInt(c)
    }
}

output:

People
{People {2}}
Cat
{Cat {4}}

SQL

(last updated at 2016-12-21)

Connect

conn, err = sql.Open("mysql", "root:password@tcp(127.0.0.1:3306)/db_name")
或
conn, err = sql.Open("mysql", "root:password@/db_name")
if err != nil {
    os.Exit(1)
}
err = conn.Ping()
if err != nil {
    os.Exit(1)
}
defer db.Close()

Prepare (prevent sql injection)

有兩種寫法，第一種是用 Query；第二種是先 Prepare 再 Query 或 Exec

SELECT with Query (with Prepare)

rows, err := conn.Query("SELECT name FROM users WHERE age=?", req.FormValue("age)
defer rows.Close()
if rows.Next() {
    var name string
    err = rows.Scan(&name)
} else {
    // no data
}

SELECT with Prepare

stmt, err := conn.Prepare("SELECT name, address FROM user WHERE age = ? AND name = ?")
defer stmt.Close()
rows, err := stmt.Query(27, "test") // replace the params
defer rows.Close()
for rows.Next() {
    var name, address string
    err = rows.Scan(&name, &address)
    fmt.Println(name, address)
}

使用 Query 要注意 :

// this is safe
conn.Query("SELECT name FROM users WHERE age=?", req.FormValue("age)

// this allows sql injection.
conn.Query("SELECT name FROM users WHERE age=" + req.FormValue("age"))

UPDATE with Prepare

stmt, err := conn.Prepare("UPDATE user SET name = ?, age = ? WHERE id = ?")
defer stmt.Close()
res, err := stmt.Exec("test", 27, 123)   // replace the params
num, err := res.RowsAffected()          // 判斷是否有成功影響欄位的值
fmt.Println(num)
// where 不到，num 是 0
// where 到，但資料沒變也是 0
// where 到，但資料有變是 1

CRUD 操作

SELECT 取第一筆

var str string
err = conn.QueryRow("SELECT id FROM user").Scan(&str)

SELECT 多筆 (fetch every single row)

var id int
var name string
rows, err := conn.Query("SELECT id, name FROM user")
defer rows.Close()
for rows.Next() {
    err := rows.Scan(&id, &name)    // do check
    fmt.Println(id, name)
}
err = rows.Err()        // do check

Query Row 取第一筆

var name string
err = db.QueryRow("SELECT name FROM users WHERE id = ?", 1).Scan(&name)

stmt, err := db.Prepare("SELECT name FROM users WHERE id = ?")
var name string
err = stmt.QueryRow(1).Scan(&name)

INSERT (TODO)

stmt, err := db.Prepare("INSERT ... ")
res, err := stmt.Exec("A", "B")
id, err := res.LastInsertId()

UPDATE (TODO)

stmt, err = db.Prepare("UPDATE ...")
res, err = stmt.Exec("A", id)
affect, err := res.RowsAffected()

DELETE (TODO)

stmt, err = db.Prepare("DELETE ...")
res, err = stmt.Exec(id)
affect, err = res.RowsAffected()

unix

調整系統設定最大讀寫檔案數量 (for currency)

unix 預設一個 process 最多可開 1024 個檔案，也就是能出去的 network currency 數量是 1024

import "golang.org/x/sys/unix"

var rLimit unix.Rlimit
var max_rlimit uint64 = 50000
var cur_rlimit uint64 = 50000
rLimit.Max = max_rlimit
rLimit.Cur = cur_rlimit
_ = unix.Setrlimit(unix.RLIMIT_NOFILE, &rLimit)

context

It’s not for killing a goroutine,
It is for ending earlier if you don’t wanna wait any longer.
An example to show you how to use WithCancel, WithTimeout and WithDeadline.
An example to show you how to do nested goroutines with their own context to interact with each other.

Code:

ctxTimeout := 3
ctx2Timeout := 5
ctx3Timeout := 7

start := time.Now()
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
defer cancel()
go func() {
    ctx2, cancel2 := context.WithTimeout(ctx, time.Duration(ctx2Timeout)*time.Second)
    go func() {
        ctx3, cancel3 := context.WithDeadline(context.Background(), time.Now().Add(time.Duration(ctx3Timeout)*time.Second))
        go func() {
            log.Printf("ctx3 waiting... timeout: %ds\n", ctx3Timeout)
            select {
            case <-ctx3.Done():
                log.Printf("ctx3 done, spent %fs (its own context)\n", time.Now().Sub(start).Seconds())
                return
            }
        }()
        log.Printf("ctx2 waiting... timeout: %ds\n", ctx2Timeout)
        select {
        case <-ctx2.Done():
            log.Printf("ctx2 done, spent %fs (ctx2 inherits ctx)\n", time.Now().Sub(start).Seconds())
            return
        }
        cancel3()
    }()
    log.Printf("ctx  waiting... timeout: %ds\n", ctxTimeout)
    select {
    case <-ctx.Done():
        log.Printf("ctx  done, spent %fs\n", time.Now().Sub(start).Seconds())
        return
    }
    cancel2()
}()

select {
case <-time.After(time.Duration(ctxTimeout) * time.Second):
    cancel()
}
time.Sleep(10 * time.Second)

Result:

2019/08/28 22:04:58 ctx  waiting... timeout: 3s
2019/08/28 22:04:58 ctx3 waiting... timeout: 7s
2019/08/28 22:04:58 ctx2 waiting... timeout: 5s
2019/08/28 22:05:01 ctx2 done, spent 3.003540s (ctx2 inherits ctx)
2019/08/28 22:05:01 ctx  done, spent 3.003625s
2019/08/28 22:05:05 ctx3 done, spent 7.005422s (its own context)

template

template 來源可以是字串也可以是一個檔案(html or text) 透過 html/template 或 text/template 將內容取代

取代的 HTML template 為字串

package main

import (
    "html/template"
    "log"
    "net/http"
)

const tmpl = `
<html>
    <head>
        <title>{{.Title}}</title>
    </head>
    <body>
        {{.Body}}
    </body>
</html>`

func tHandler(w http.ResponseWriter, r *http.Request) {
    t := template.Must(template.New("ex").Parse(tmpl))
    v := map[string]interface{}{
        "Title": "Test <b>World</b>",
        "Body":  template.HTML("Hello <b>World</b>"),
    }
    t.Execute(w, v)
}

func main() {
    http.HandleFunc("/", tHandler)
    err := http.ListenAndServe(":9090", nil)
    if err != nil {
        log.Fatal("ListenAndServe: ", err)
    }
}

瀏覽器得到 server 輸出的內容 :

<html>
    <head>
        <title>Test &lt;b&gt;World&lt;/b&gt;</title>
    </head>
    <body>
        Hello <b>World</b>
    </body>
</html>

可以看到 title 跟 body 的結果截然不同, title 的 html 標籤會被轉義, 而 body 沒有是因為我加上了 template.HTML() 不要轉義 HTML 標籤, 我覺得預設轉義設計是很好的, 防止沒注意而產生了 XSS 漏洞, rails 也是預設轉義的, 但 php 什麼時候才要改!!!??? (怒

當改成 t.Execute(os.Stdout, v), 表示是系統的標準輸出, 所以只會在 terminal 上看到輸出, 瀏覽器則不會有任何內容

取代的 HTML template 為檔案

將 header 與 footer 做為固定的 template, 並傳入動態內容 :

func tHandler(w http.ResponseWriter, r *http.Request) {
    t, _ := template.ParseFiles("header.tmpl", "body.html", "footer.tmpl")
    var data = map[string] interface{}{
        "content" : "Do you copy?",
    }
    t.ExecuteTemplate(w, "body", data)
    t.Execute(w, nil)
}

header.tmpl :

{{define "header"}}
<html>
<head>
    <title>Video downloader</title>
</head>
<body>
{{end}}

body.html :

{{define "body"}}
{{template "header"}}
<h1 id="go">Golang Web Works!</h1>
{{.content}}
{{template "footer"}}
{{end}}

footer.tmpl :

{{define "footer"}}
</body>
</html>
{{end}}

最後在瀏覽器顯示的結果是 :

<html>
<head>
    <title>Video downloader</title>
</head>
<body>

<h1 id="go">Golang Web Works!</h1>
Do you copy?

</body>
</html>

副檔名不需要一定是 .html .tmpl, 你可以自訂任何你喜歡的 template.ParseFiles 只負責引入檔案而已, 而真正負責要呈獻出來的內容及 parse 是 t.ExecuteTemplate

For loop template

有時候我們會重覆使用同一個 template 連續印出10次, 作法如下

作法 1, 用 `range` 直接在 template 跑 10 次

view/nameList.tmpl :

{{define "nameList"}}
Name list :
{{ range $i, $name := .nameList}}
{{$i}} -> {{$name}}
{{end}}
{{end}}

func main :

var tmp bytes.Buffer
var data = map[string]interface{}{}
t, _ := template.ParseFiles(
    "view/nameList.tmpl",
)
data["nameList"] = []string{"Jack", "Bob"}
t.ExecuteTemplate(&tmp, "nameList", data)
fmt.Println(tmp.String())

作法2, 先跑 10 次的 template, 再將生成後的 html 傳到 body 再輸出

建立要重覆的 urlItem.tmpl :

{{define "urlItem"}}
<li id="url-{{.num}}">{{.num}}</li>
{{end}}

body.html :

{{define "body"}}
<ul class="list-group">
    {{.urlItem}}
</ul>
{{end}}

完整程式碼 :

// Show view
var tmplPath string = "view/template/"
var indexPath string = "view/index/"
t, _ := template.ParseFiles(
    tmplPath + "header.tmpl",
    indexPath + "body.html",
    tmplPath + "index/urlItem.tmpl",
    tmplPath + "footer.tmpl",
)

// For loop url item
var tmplBuf bytes.Buffer                            // 建立 buffer, 等等 for loop 的 template 都存>進來
var nums = map[string] interface{}{}                // 建立 data interface{}
for num := 1; num <= 10; num++ {
    nums["num"] = num
    t.ExecuteTemplate(&tmplBuf, "urlItem", nums)    // 將數字帶進 template
}
data["urlItem"] = template.HTML(tmplBuf.String())   // 將 for loop template 的結果存進 urlItem, 並>且不要 escape HTML Tag
t.ExecuteTemplate(w, "body", data)
t.Execute(w, nil)

ref : http://play.golang.org/p/Uw8l3M7Qvg https://groups.google.com/forum/#!topic/golang-nuts/8L4eDkr5Q84 http://blog.xcai.net/golang/templates

取代的 TEXT template 為檔案

// 讀取資料夾底下的 template (不需要指定檔名)
var emailBodyTemplates = template.Must(template.ParseGlob("app/template/email/en/*"))     // 資料夾底下有很多 .txt 的 template 檔案
var tmp bytes.Buffer
err = emailBodyTemplates.ExecuteTemplate(&tmp, templateName, replacement)
if err != nil {
    return
}

// 讀取檔案的 template (需要指定檔名)
var emailLayoutTemplates = template.Must(template.ParseFiles("app/template/layout/email.html"))
err = emailLayoutTemplates.Execute(&tmp, replacement)

不要脫逸 `'` `"` 等 HTML 符號

用 template.HTML 包起來

improt "html/template"

var t = template.Must(template.New("ex").Parse(`{{.name}} pen. {{.name2}} pen.`))
data := map[string]interface{}{
    "name":  template.HTML("Tom's"),
    "name2": "Tom's",
}
t.ExecuteTemplate(os.Stdout, "ex", data)

result

Tom's pen. Tom&#39;s pen.

' -> '

解法2: improt 改成 text/template, 就不需要用 template.HTML 防止脫逸了

fmt

輸出固定長度

前面補 0

接收 command 輸入的值

印出顏色

最簡單的方法, 不過有個缺點是它會是 string

不改變本身的型態

如果已知位數, 可簡化

不改變本身的型態且4捨5入

log

儲存成 file

將 log 內容暫存在 buffer

攔截 log 的內容

寫到 syslog

net

取得本機上的 Mac

header / status / etc.

net/url 解析 URL

url 組 query

已存在的 url 再新增 query param

自組 post body 要 escape

net/http Download file

只取得 response 的 Header

Get filename from url path

GET method

Get 另一種寫法，指定 header

Post (form-data / json)

Post (file)

Http 建立 API

ConnState 可以截取 connection 的狀態

handle connection (created at 27 Dec 2013)

TCP example

API Custom Handler

net/http - x509: certificate signed by unknown authority

net/http client request 得到 EOF 原因

Get lan ip

Proper way to test CIDR membership of an IP 4 or 6 address example

io

io/ioutil/bufio

將下載的資料轉成 io.Reader 型態

Read File

Read last line of file (ref: here)

Get last-modified time of file

Line counter

有效率的 monitor file 並取得最新內容 (類似 tail 指令)

io - Read into []byte from another []byte

string

基本處理

os

外部參數

“main.main” 函數並沒有返回值

檔案 / 目錄 是否存在

file - 指定寫入位置

Create folder

在 windows 下執行可執行檔(.exe)後視窗停留不關閉

不同作業系統取得不同 path separator (slash, 斜線)

progress bar

讓字一個一個 print 出來

os/exec

取得執行中的 binary 路徑

執行外部指令

signal

接收中止訊號(kill signal)

發送 signal

What would it be if both of main program and sub-program try to capture the signals at the same time?

time

Parse time 的非預期狀況

Comapare

延遲

時間相加／相減

3 * time.Second

印出到小數的 timestamp/datetime (e.g. 2017-06-28 02:58:46.452 +0000 UTC)

math/rand

產生 0~7 (每一次產生不一樣的變數)

產生一串隨機數值 (ex: 7572000213564998818)

Unique key

generate 32 bytes authentication key

sync

pool 特性

pool 是會被GC的

檔案 / 目錄是否存在

印出到小數的 timestamp/datetime (e.g. `2017-06-28 02:58:46.452 +0000 UTC`)

String 相同, 同長度及 byte 不同 `invalid character '\x00' after top-level value`

作法 1, 用 `range` 直接在 template 跑 10 次