Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>

Add shared file storage handling during cluster initialization.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch introduces core file storage support, consisting of the following:

A configure-time switch for enabling/disabling shared file storage
support and controlling the shared file storage location:
--with-shared-file-storage-dir=.  Shared file storage configuration is then
available as _autoconf.ENABLE_SHARED_FILE_STORAGE and
_autoconf.SHARED_FILE_STORAGE_DIR and there is a cluster-wide ssconf
key named "shared_file_storage_dir" for changing the file location.

A new disk template named "sharedfile" (DT_SHARED_FILE), using
ganeti.bdev.FileStorage.

Auxiliary functions in lib/config.py to handle shared file storage.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
[iustin@google.com: small style fixes]
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is analogue to the existing check for a responsive node daemon.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

At least ganeti-confd was not started. It got started a few minutes
later by ganeti-watcher. Also move one pylint disable to the effective
line.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

- Code formatting
- Use ShellQuote for one argument
- Remove variables no longer used after commit 9294514d

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Adeodato Simo <dato@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is reverting the revert and adaption to fix the issue which caused
the revert

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This reverts commit fd7f5b41.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

If the configuration file doesn't denote this node as master, we prevent
startup. This would have detected our previous race condition more
easily, hence we add it as a permanent check.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This fixes a recently diagnosed race condition between master failover
and the watcher.

Currently, the master failover first stops the master daemon, checks
that the IP is no longer reachable, and then distributes the updated
configuration. Between the stop and the distribution, it can happen that
the watcher starts the master daemon on the old node again, since ssconf
still points the master to it (and all nodes vote so).

In even more weird cases, the master daemon starts and before it manages
to open the configuration file, it is updated, which means the master
will respond to QueryClusterInfo with another node as the real master.

This patch reorders the actions during master failover:

- first, we redistribute a fixed config; this means the old master will
  refuse to update its own config file and ssconf, and that most jobs
  that change state will fail to finish
- we then immediately kill it; after this step, the watcher will be
  unable to start it, since the master will refuse startup
- and only then we check for IP reachability, etc.

I've tested the new version against concurrent launch of the watcher;
while my tests are not very exhaustive, two things can happen: watcher
see the daemons as dead, and tries to restart them, which also fail; or
it simply get an error while reading from the master daemon. Both these
should be OK.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This is the first step for the support of wiping block devices prior
to creation of the instance.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Please note that if the pid file is broken or missing we'll not catch
the process (if any is running) and it's up to the user to fix this state

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patch also ensures that the initial configuration has all the
needed UUIDs and that they are unique, by using a
TemporaryReservationManager inside InitConfit to generate them.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

This patches changes the StartMaster method to consult the cluster
primary ip version when deciding whether to use arping or ndisc6 after
activating the master ip.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

We expose the ip_version (4, 6) to the external interface and internally
we convert it to ip_family (AF_INET=2, AF_INET6=10). This makes the code
more concise as all functions deal with family rather than version.

This patch does not yet expose this parameter via gnt-cluster, but only uses
the constant IP4_VERSION. This will be enabled in a future patch.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch enables IPv6 name resolution by using socket.getaddrinfo
instead of socket.gethostbyname_ex.

It renames the HostInfo class to Hostname and unifies its use throughout
the code. This is achieved by using static calls where no object is
needed and removes some obsolete code.

For now, we just resolve to IPv4 addresses, but this will change once it
is needed.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This change is needed as we will need to read the primary ip version
cluster parameter before we start the node daemon. The reason is that we
need to know in advance if we bind to the IPv4 or IPv6 any address.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch unifies the netutils functions dealing with IP addresses to
three classes:
- IPAddress: Common IP address functionality
- IPv4Address: IPv4 specific functionality
- IPv6address: IPv6-specific functionality

Furthermore it adds methods to check whether an address is a loopback
address, replacing the .startswith("127") for IPv4 and adding IPv6
support.

It also provides the basis for future IPv6 address handling. Methods to
convert IP strings to their corresponding interger values will allow to
canonicalize IPv6 addresses.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Due to the current switch of the RPC client to PycURL, a bug with newer
versions of libcurl surfaced. When the 'Subject' or 'Issuer' of
'server.pem' were empty, SSL handshake failed.

This patch changes the certificate generation functions such that they
always use "ganeti.example.com" as commonName (CN) for 'Subject' and
'Issuer'.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch moves network utility functions to a dedicated module.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add a cluster parameter to hold the iallocator that will be used by default
when required and no alternative (manually-specified iallocator or
manually-specified node(s)) is given.

Signed-off-by: Apollon Oikonomopoulos <apollon@noc.grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

I have seen some very seldom errors where (it seems) the address is
still live for a short while after removing it from the old master, thus
the new master will fail in startup/adding its own IP address.

To prevent against this, we add a delay/retry before we proceed, if the
IP is still reachable.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>
(cherry picked from commit 425f0f54)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Balazs Lecz <leczb@google.com>

I have seen some very seldom errors where (it seems) the address is
still live for a short while after removing it from the old master, thus
the new master will fail in startup/adding its own IP address.

To prevent against this, we add a delay/retry before we proceed, if the
IP is still reachable.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Signed-off-by: Luca Bigliardi <shammash@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This patch introduces functions to check for valid IPv4 and IPv6
addresses and converts IsValidIP() to return True if it is either a IPv4
or a IPv6 address.

For now we do not change the functional behavior and replace IsValidIP
with IsValidIP4. This might change in the future.

Signed-off-by: Manuel Franceschini <livewire@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This is necessary to allow cfgupgrade to work on a non-standard directory.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: Balazs Lecz <leczb@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

There are a few issues with disabled file storage:
- cluster initialization is broken by default, as it uses the 'no'
  setting which is not a valid path
- some other parts of the code require the file storage dir to be a
  valid path; we workaround by skipping such code paths when it is
  disabled

A side effect is that we abstract the storage type checks into a
separate function and add validation in RepairNodeStorage (previously a
luxi client which didn't use cli.py and submitted an invalid type would
get "storage units of type 'foo' can not be repaired").

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

Commit 43575108 added bootstrap.GenerateclusterCrypto and commit
7506a7f1 changed cfgupgrade to use it. However, this lost the
functionality of upgrading in non-default DATA_DIR.

To fix this, we enhance bootstrap.GenerateclusterCrypto to accept custom
file paths for the three files it modifies. If more files will be needed
in the future, we could just pass in modified DATA_DIR, but for now it
suffices.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This will be used to conditionally enable the watcher node maintenance
feature.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Information exchanged between different clusters via untrusted
third parties (e.g. for remote instance import/export) must be
signed with a secret shared between all involved clusters to
ensure the third party doesn't modify the information.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Currently, the ganeti-confd's HMAC key is called “cluster HMAC key” or
simply “HMAC key” everywhere. With the implementation of inter-cluster
instance moves, another HMAC key will be introduced for signing critical
data. They can not be the same, so this patch clarifies the purpose of the
“cluster HMAC key” by renaming it. The actual file name is not changed.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>