Software engineering notes

Network HTTP

HTTP/1 vs HTTP/2 vs HTTP/3

ref:

Http Status

HTTP status codes are grouped into five classes

Http Haader

keep-alive

Connection: Keep-Alive is used to reuse the same TCP connection to send and receive multiple HTTP requests/responses, instead of opening a new connection for each request/response pair

TCP keep-alive and HTTP keep-alive serve different purposes

Etag

第一次 : browser 對 server 發出請求, server 回應 200 ok, 並多加上 header[‘ETag’] = body 以 md5 編碼

browser 會 cache response 及儲存 Etag

第二次 : browser 對 server 發出請求並多帶 headers[‘If-None-match’] = 上面 ETag 的值, server 再算出 ETag 是否值為一樣, 一樣的話會返回 304 Not Modified

browser 收到 304 會從 cache 拿之前 cache 的結果

HTTP Methods

Restful common endpoints

Index       GET     /users
Show        GET     /users/1
New         GET     /users/new
Create      POST    /users
Edit        GET     /users/1/edit
Update      PUT     /users/1
Detele      DELETE  /users/1

GET

GET 是我們最常使用也是最簡單的，它帶資料的方式就是把 query string 放在網址後面 :

example.com?f1=v1&f2=v2

GET 沒有 Body, 整個 Http 封包會像這樣 :

GET /api/test?f1=v1&f2=v2 HTTP/1.1
Host: example.com
Content-Type: application/x-www-form-urlencoded
Cache-Control: no-cache
Postman-Token: 0f99b8e9-0322-952e-a67c-76b498c51903

POST form

當要背景傳送資料時要使用 POST，也就是我們常在用的表單，它的實現原理是在 Header 加上 Content-Type: application/x-www-form-urlencoded

而 Body 就是 query string :

f1=v1&f2=v2

整個 Http 封包會像這樣 :

POST /api/test?f1=f1&f2=v2 HTTP/1.1
Host: example.com
Content-Type: application/x-www-form-urlencoded
Cache-Control: no-cache
Postman-Token: 3f036f1b-dfec-1269-41b7-bbf6781cbac4

f1=v1&f2=v2

POST 可同時支援 GET 及 POST 的參數

POST with file

如果要傳送檔案，就不是用原本的 Content-Type: application/x-www-form-urlencoded，要改用 Content-Type: multipart/form-data

雖然說 Content-Type 使用後者，但仍然一樣可以傳送 form data

你的 params 會被拆解成一塊一塊的，整個 Http 封包會像這樣 :

POST /api/test/ HTTP/1.1
Host: example.com
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Cache-Control: no-cache
Postman-Token: b953aac1-b8d5-f1e4-36ac-2672b3a028fc

------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="image"; filename=""
Content-Type:

data:application/octet-stream;base64,/9j/4AAQSkZJRgABAQEASABIAA....
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="f1"

v1
------WebKitFormBoundary7MA4YWxkTrZu0gW--

PUT vs PATCH

PUT 相當於是 delete + insert, 是對整個資源進行更新

PATCH 是只更新部份的資源

Does browser support PATCH, PUT and DELETE?

不支援, HTML 只定義了 GET/POST,

所以 HTML Form 是沒有支援 PUT/DELETE 的

但 XmlHttpRequest (也就是Ajax) 有定義 GET/POST/PUT/PATCH/DELETE/HEAD/OPTIONS

workaround: 在 HTML Form 裡偷藏 _method 參數, 定義不支援的 method, 送到 server 端再判斷

Terms of URL

http://example.com/public/index.html#post

http://www.example.com:8888/public/index.html?key=value#hashtag

HTTPS

TLS

What are the steps to apply TLS for your domain?

  1. Generate a private-public key pair
    • The public key will be shared as part of the digital certificate
  2. Create a Certificate Signing Request (CSR).
    • CSR content
      • include public key, domain, and other details
      • they aren’t encrypted by private key
    • Generate a unique signature
      • Generate a hash of the CSR content using a cryptographic hash function
      • Sign the hash using the private key
    • Create the CSR file
      • Encode the CSR content and signature using base64
      • Create a CSR file by wrapping Base64-encoded text with header and footer lines.
      • CSR is typically in Privacy-Enhanced Mail (PEM) format.
      • Command to generate CSR file openssl req -new -key private_key.pem -out my_csr.pem
    • Signature is the only data to be encrypted by private key. Public key and domain aren’t.
  3. Submit CSR to a Certificate Authority (CA)
  4. CA verifies your CSR file for Domain and organization validation
    • The CA extracts the Base64-encoded content of the CSR (between header and footer)
    • Convert it back into binary data that contains the actual CSR information in a standardized format, such as PKCS#10 or X.509.
    • The CA parses the binary data to extract the public key, domain, signature and other information.
    • The CA can just read public key and domain without needing your private key.
    • The CA uses the included public key to verify the signature of the CSR.
    • This step ensures that the CSR was indeed created using the corresponding private key and that the data within the CSR has not been tampered with.
    • Since public-private key pairs are mathematically linked, the CA can perform this verification without access to your private key.
  5. Get a digital certificate from CA
    • Once CA successfully verifies your CSR, it will issue you a digital certificate.
    • the certificate file(s) in a format like PEM or PFX/PKCS#12 includes:
      • Your server certificate: for your domain, containing the public key and other verified information
      • Intermediate certificates: to maintain security and limit the risk associated with the compromise of a root certificate
      • Root certificate (optional): Some CAs might provide their root certificate, but it’s often unnecessary, as most web servers and clients already have trusted root certificates pre-installed
  6. Install the digital certificate
    • Install the server certificate
      • Save the server certificate, private key, and (if provided) intermediate certificate(s) to a secure location on your server.
    • Install intermediate certificates (if provided)
      • Concatenate the intermediate certificate(s) to the end of your server certificate file. cat server.crt intermediate.crt > combined.crt
  7. Configure the web server
    • Taking Nginx as an example
      • listen 443 ssl;
      • ssl_certificate_key /path/to/private.key
      • ssl_certificate /path/to/combined.crt
      • ssl_protocols TLSv1.2 TLSv1.3;
      • ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305';
      • ssl_prefer_server_ciphers on;

CSR file looks like below

-----BEGIN CERTIFICATE REQUEST-----
MIIC2jCCAcICAQAwgZExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UE
BwwJU2FuIEpvc2UxEDAOBgNVBAoMB0FDTUUgSW5jMRQwEgYDVQQLDAtFbmdpbmVl
cmluZzEUMBIGA1UEAwwLZXhhbXBsZS5jb20xHDAaBgkqhkiG9w0BCQEWDXN1cHBv
cnRAZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDU
...
I+9M5Q5cej5qY7lJlgvKx6kLca6fDDU6B5CJGOyTeX9gToh1JUX8W+RJvRZlQ2Fw
-----END CERTIFICATE REQUEST-----

How HTTPS works?

  1. DNS resolution: the browser first performs a DNS lookup to find the IP address associated with the domain name.
  2. TCP handshake: The browser establishes a TCP connection with the web server at the IP address obtained during DNS resolution.
  3. SSL/TLS handshake: Once the TCP connection is established, the browser initiates the SSL/TLS handshake to securely exchange cryptographic parameters and establish a secure connection.
    1. ClientHello: The client initiates the handshake by sending a ClientHello message, which includes the supported TLS protocol versions, a list of cipher suites, a ClientRandom number, and any additional extensions (such as SNI).
    2. ServerHello: The server responds with a ServerHello message containing the selected TLS protocol version, the chosen cipher suite, and a ServerRandom number.
    3. Server Certificate: The server sends its digital certificate containing its public key and identity information to the client. This allows the client to authenticate the server and encrypt the data exchanged during the handshake.
    4. ServerHelloDone: The server sends a ServerHelloDone message to signal the end of its part of the handshake negotiation.
    5. ClientKeyExchange: The client generates a PreMasterSecret, encrypts it with the server’s public key (from the server’s certificate), and sends the encrypted PreMasterSecret in a ClientKeyExchange message.
      • PreMasterSecret, which is a random value that both the client and server will use to compute a shared secret called the Master Secret.
      • This ensures that only the server can decrypt the PreMasterSecret, as it possesses the corresponding private key.
      • The server receives the ClientKeyExchange message and decrypts the PreMasterSecret using its private key.
    6. (Both client and server now can generate the master secret by pre-master secret, client random and server random
      • master secret = PRF(PreMasterSecret, ClientRandom + ServerRandom)
    7. ChangeCipherSpec: Both the client and server send a ChangeCipherSpec message to indicate that subsequent messages will be encrypted using the negotiated cipher suite and keys derived from the PreMasterSecret, ClientRandom, and ServerRandom.
    8. Finished: Both the client and server send an encrypted Finished message containing a hash of the complete handshake process to verify its integrity.
  4. Secure data exchange: After the SSL/TLS handshake is complete, the client and server can exchange application data over the encrypted SSL/TLS connection. All messages are encrypted and decrypted using symmetric encryption keys derived from the PreMasterSecret and the ClientRandom and ServerRandom values.
    • Both the client and server now have the same PreMasterSecret.
    • The Master Secret is then used to derive symmetric encryption keys, also known as session keys, for data encryption and integrity checks.
  5. Connection closure: When the client and server have finished exchanging data, they send close_notify alerts to each other to signal the end of the secure connection. The underlying TCP connection is then closed through the standard TCP connection termination process.

The SSL/TLS handshake process:

Client                                                                                    Server
  |                                                                                         |
  | ----------- ClientHello (Supported versions, cipher suites, ClientRandom) ------------> |
  |                                                                                         |
  | <---------- ServerHello (Selected version, cipher suite, ServerRandom) ---------------- |
  | <---------- Server Certificate (Server's public key, identity information) ------------ |
  | <---------- ServerHelloDone ----------------------------------------------------------- |
  |                                                                                         |
  | ----------- ClientKeyExchange (Encrypted PreMasterSecret with server's public key) ---> |
  | ----------- ChangeCipherSpec (Switch to negotiated cipher) ---------------------------> |
  | ----------- Finished (Encrypted handshake hash) --------------------------------------> |
  |                                                                                         |
  | <---------- ChangeCipherSpec (Switch to negotiated cipher) ---------------------------- |
  | <---------- Finished (Encrypted handshake hash) --------------------------------------- |
  |                                                                                         |
  | <---------- (Encrypted data exchange using negotiated cipher suite) ------------------> |