Update inter-cluster instance move design with HMAC signatures

This also adds a large piece of pseudo code for explanatory purposes.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

doc/design-2.2.rst

@@ -234,13 +234,19 @@ may accept unverified certificates. The generated certificate should
 only be valid for the time necessary to move the instance.
 
 For additional protection of the instance data, the two clusters can
-verify the certificates exchanged via the third party by signing them
-using HMAC with a key shared among the involved clusters. If the third
-party does not know this secret, it can't forge the certificates and
-redirect the data. Unless disabled by a new cluster parameter, verifying
-the HMAC must be mandatory. The HMAC will be prepended to the
-certificate and only covers the certificate (from ``-----BEGIN
-CERTIFICATE-----`` to ``-----END CERTIFICATE-----``).
+verify the certificates and destination information exchanged via the
+third party by checking an HMAC signature using a key shared among the
+involved clusters. By default this secret key will be a random string
+unique to the cluster, generated by running SHA1 over 20 bytes read from
+``/dev/urandom`` and the administrator must synchronize the secrets
+between clusters before instances can be moved. If the third party does
+not know the secret, it can't forge the certificates or redirect the
+data. Unless disabled by a new cluster parameter, verifying the HMAC
+signatures must be mandatory. The HMAC signature for X509 certificates
+will be prepended to the certificate similar to an RFC822 header and
+only covers the certificate (from ``-----BEGIN CERTIFICATE-----`` to
+``-----END CERTIFICATE-----``). The header name will be
+``X-Ganeti-Signature``.
 
 On the web, the destination cluster would be equivalent to an HTTPS
 server requiring verifiable client certificates. The browser would be
@@ -270,19 +276,61 @@ Workflow
 
 #. Third party tells source cluster to shut down instance, asks for the
    instance specification and for the public part of an encryption key
+
+   - Instance information can already be retrieved using an existing API
+     (``OpQueryInstanceData``).
+   - An RSA encryption key and a corresponding self-signed X509
+     certificate is generated using the "openssl" command. This key will
+     be used to encrypt the data sent to the destination cluster.
+
+     - Private keys never leave the cluster.
+     - The public part (the X509 certificate) is signed using HMAC with
+       salting and a secret shared between Ganeti clusters.
+
 #. Third party tells destination cluster to create an instance with the
    same specifications as on source cluster and to prepare for an
    instance move with the key received from the source cluster and
    receives the public part of the destination's encryption key
+
+   - The current API to create instances (``OpCreateInstance``) will be
+     extended to support an import from a remote cluster.
+   - A valid, unexpired X509 certificate signed with the destination
+     cluster's secret will be required. By verifying the signature, we
+     know the third party didn't modify the certificate.
+
+     - The private keys never leave their cluster, hence the third party
+       can not decrypt or intercept the instance's data by modifying the
+       IP address or port sent by the destination cluster.
+
+   - The destination cluster generates another key and certificate,
+     signs and sends it to the third party, who will have to pass it to
+     the API for exporting an instance (``OpExportInstance``). This
+     certificate is used to ensure we're sending the disk data to the
+     correct destination cluster.
+   - Once a disk can be imported, the API sends the destination
+     information (IP address and TCP port) together with an HMAC
+     signature to the third party.
+
 #. Third party hands public part of the destination's encryption key
    together with all necessary information to source cluster and tells
    it to start the move
+
+   - The existing API for exporting instances (``OpExportInstance``)
+     will be extended to export instances to remote clusters.
+
 #. Source cluster connects to destination cluster for each disk and
    transfers its data using the instance OS definition's export and
    import scripts
+
+   - Before starting, the source cluster must verify the HMAC signature
+     of the certificate and destination information (IP address and TCP
+     port).
+   - When connecting to the remote machine, strong certificate checks
+     must be employed.
+
 #. Due to the asynchronous nature of the whole process, the destination
    cluster checks whether all disks have been transferred every time
-   after transfering a single disk; if so, it destroys the encryption
+   after transferring a single disk; if so, it destroys the encryption
    key
 #. After sending all disks, the source cluster destroys its key
 #. Destination cluster runs OS definition's rename script to adjust
@@ -291,6 +339,147 @@ Workflow
    by the third party
 #. Source cluster removes the instance if requested
 
+Instance move in pseudo code
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. highlight:: python
+
+The following pseudo code describes a script moving instances between
+clusters and what happens on both clusters.
+
+#. Script is started, gets the instance name and destination cluster::
+
+    (instance_name, dest_cluster_name) = sys.argv[1:]
+
+    # Get destination cluster object
+    dest_cluster = db.FindCluster(dest_cluster_name)
+
+    # Use database to find source cluster
+    src_cluster = db.FindClusterByInstance(instance_name)
+
+#. Script tells source cluster to stop instance::
+
+    # Stop instance
+    src_cluster.StopInstance(instance_name)
+
+    # Get instance specification (memory, disk, etc.)
+    inst_spec = src_cluster.GetInstanceInfo(instance_name)
+
+    (src_key_name, src_cert) = src_cluster.CreateX509Certificate()
+
+#. ``CreateX509Certificate`` on source cluster::
+
+    key_file = mkstemp()
+    cert_file = "%s.cert" % key_file
+    RunCmd(["/usr/bin/openssl", "req", "-new",
+             "-newkey", "rsa:1024", "-days", "1",
+             "-nodes", "-x509", "-batch",
+             "-keyout", key_file, "-out", cert_file])
+
+    plain_cert = utils.ReadFile(cert_file)
+
+    # HMAC sign using secret key, this adds a "X-Ganeti-Signature"
+    # header to the beginning of the certificate
+    signed_cert = utils.SignX509Certificate(plain_cert,
+      utils.ReadFile(constants.X509_SIGNKEY_FILE))
+
+    # The certificate now looks like the following:
+    #
+    #   X-Ganeti-Signature: $1234$28676f0516c6ab68062b[…]
+    #   -----BEGIN CERTIFICATE-----
+    #   MIICsDCCAhmgAwIBAgI[…]
+    #   -----END CERTIFICATE-----
+
+    # Return name of key file and signed certificate in PEM format
+    return (os.path.basename(key_file), signed_cert)
+
+#. Script creates instance on destination cluster and waits for move to
+   finish::
+
+    dest_cluster.CreateInstance(mode=constants.REMOTE_IMPORT,
+                                spec=inst_spec,
+                                source_cert=src_cert)
+
+    # Wait until destination cluster gives us its certificate
+    dest_cert = None
+    disk_info = []
+    while not (dest_cert and len(disk_info) < len(inst_spec.disks)):
+      tmp = dest_cluster.WaitOutput()
+      if tmp is Certificate:
+        dest_cert = tmp
+      elif tmp is DiskInfo:
+        # DiskInfo contains destination address and port
+        disk_info[tmp.index] = tmp
+
+    # Tell source cluster to export disks
+    for disk in disk_info:
+      src_cluster.ExportDisk(instance_name, disk=disk,
+                             key_name=src_key_name,
+                             dest_cert=dest_cert)
+
+    print ("Instance %s sucessfully moved to %s" %
+           (instance_name, dest_cluster.name))
+
+#. ``CreateInstance`` on destination cluster::
+
+    # …
+
+    if mode == constants.REMOTE_IMPORT:
+      # Make sure certificate was not modified since it was generated by
+      # source cluster (which must use the same secret)
+      if (not utils.VerifySignedX509Cert(source_cert,
+            utils.ReadFile(constants.X509_SIGNKEY_FILE))):
+        raise Error("Certificate not signed with this cluster's secret")
+
+      if utils.CheckExpiredX509Cert(source_cert):
+        raise Error("X509 certificate is expired")
+
+      source_cert_file = utils.WriteTempFile(source_cert)
+
+      # See above for X509 certificate generation and signing
+      (key_name, signed_cert) = CreateSignedX509Certificate()
+
+      SendToClient("x509-cert", signed_cert)
+
+      for disk in instance.disks:
+        # Start socat
+        RunCmd(("socat"
+                " OPENSSL-LISTEN:%s,…,key=%s,cert=%s,cafile=%s,verify=1"
+                " stdout > /dev/disk…") %
+               port, GetRsaKeyPath(key_name, private=True),
+               GetRsaKeyPath(key_name, private=False), src_cert_file)
+        SendToClient("send-disk-to", disk, ip_address, port)
+
+      DestroyX509Cert(key_name)
+
+      RunRenameScript(instance_name)
+
+#. ``ExportDisk`` on source cluster::
+
+    # Make sure certificate was not modified since it was generated by
+    # destination cluster (which must use the same secret)
+    if (not utils.VerifySignedX509Cert(cert_pem,
+          utils.ReadFile(constants.X509_SIGNKEY_FILE))):
+      raise Error("Certificate not signed with this cluster's secret")
+
+    if utils.CheckExpiredX509Cert(cert_pem):
+      raise Error("X509 certificate is expired")
+
+    dest_cert_file = utils.WriteTempFile(cert_pem)
+
+    # Start socat
+    RunCmd(("socat stdin"
+            " OPENSSL:%s:%s,…,key=%s,cert=%s,cafile=%s,verify=1"
+            " < /dev/disk…") %
+           disk.host, disk.port,
+           GetRsaKeyPath(key_name, private=True),
+           GetRsaKeyPath(key_name, private=False), dest_cert_file)
+
+    if instance.all_disks_done:
+      DestroyX509Cert(key_name)
+
+.. highlight:: text
+
 Miscellaneous notes
 ^^^^^^^^^^^^^^^^^^^