Acquire corresponding instance locks if conflicts_check is True.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Support:
 - gnt-network add --tags.
 - gnt-network list-tags/add-tags/remove-tags/.
 - gnt-network list -o +tags

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

If a nic has a network field then encapsulate a network object in
netinfo slot for every rpc. This is needed to pass network info to
scripts managing nics (kvm-vif-bridge).

Introduce _BuildNetworkEnv()

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Modify existing GenerateMAC so that it takes network as an argument.

Substitute _GenerateOneMAC with a decorator _GenerateMACPrefix
that chooses the prefix (network's or cluster's) and wraps
_GenerateMACSuffix that generates the three remaining bytes.

Check mac prefix validity when adding/modifing a network.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Support:
 - GetNetwork(s)
 - CreateNetwork
 - ConnectNetwork
 - DisconnectNetwork
 - RemoveNetwork

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

gnt-network is used to manipulate and handle networks that
currently provides the following operations:

 * Add a new network:
   gnt-network add --network=1.2.3.0/28 --gateway=1.2.3.1
                   --add-reserved-ips=1.2.3.4,1.2.3.5 testnet

 * Remove an existing network:
   gnt-network remove testnet

 * Modify an existing network:
   gnt-network modify --gateway=1.2.3.6
                      --network-type=private
                      --network6=2001:648::/64
                      --gateway6=none testnet
                      --add-reserved-ips=1.2.3.10,1.2.3.10,
                      --remove-reserved-ips=1.2.3.20
                      testnet

 * Connect an existing network to a nodegroup:
   gnt-network connect testnet default bridged br100
   gnt-network connect testnet <nodegroup> <mode> <link>
   (pass all for <nodegroup> to connect to all nodegroups)

 * Disconnect an existing network from a nodegroup:
   gnt-network disconnect testnet <nodegroup>
   (pass all for <nodegroup> to disconnect from all nodegroups)

 * List available networks:
   gnt-network list

 * Show network info:
   gnt-network info [testnet]

Introduce new option NOCONFLICTSCHECK_OPT for not checking for
conflicting IPs. Using this might cause data inconsistency.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Add --no-conflicts-check option.
Modify instance query methods to return nic's network info.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Specifically:
 * export a new environment variable for the gnt-os-interface
 * export a new environment variable for KVM specific scripts
 * ovf support for networks

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Keep backwards compatibility just like LUInstanceCreate() and
support network parameter for NIC object.

Add ec_id argument in Update() to be able to commit temporary
IPs that were previously reserved.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Release any IPs held the instance before actually removing.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Implement backend support, to export the IP pool management
functionality to the clients. When the new NIC parameter 'network'
is given, the ippool management system is triggered.
If a NIC belongs to a network, it inherits the netparams
(mode, link) as its nicparams. If a network is requested, then
no 'mode' or 'link' should be specified. Backwards compatibility
is preserved, if 'network' parameter is omitted.

Examples for IP pool management system:

--net 0:network=net1
  for an IPless NIC in a network

--net 0:ip=pool,network=net1
  for automatically generated IP inside the network

--net 0:ip=1.2.3.4,network=net1
  for specific IP inside the network

--net 0:network=none
  for IPless NIC with default nicparams

Examples for traditional assignment:

--net 0:ip=1.2.3.4,link=br100,mode=bridged
--net 0:ip=none,link=rt2,mode=routed
--net 0

Check for conflicting IPs to ensure IP uniqueness inside nodegroups.

IAllocator should be extended to become network aware. If an instance
has a NIC inside a network, then IAllocator must pick a node that its
nodegroup is already connected with that network.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Use TemporaryReservationManager for IP handling to ensure
config data consistency.

Implement:
 - methods for reserving/releasing IPs
 - _UnlockedCommitTemporaryIps()
 - CheckIPInNodeGroup() used for conflicting IPs
 - GetGroupNetParams() used for inheriting netparams.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

LUNetworkConnect() connects a network to a nodegroup and defines
its netparams (mode and link). Specifically:
 * Check if network already exists
 * Check if netparams are valid
 * Check if already connected to the same nodegroup
 * Produce errors if any conflicting IPs found
 * Update the 'networks' slot in NodeGroup config object
   with the network UUID and netparams requested. E.g.
   nodegroup_uuid {
     networks {
       network_uuid: {mode: 'bridged', link: 'br100'}
     }
   }

LUNetworkDisconnect() unmaps a network from a nodegroup:
 * Produce errors if an instance has a NIC with IP inside the network
 * Remove corresponding entry in 'networks' slot of NodeGroup object

Check for conflicting ips can be avoided with corresponding option.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Support modifying all network parameters except for network (ip range)
Cannot modify gateway and reserved ips at the same time.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Summarily list all existing networks
Supply detailed info for every existing network
 - List used/free IPs
 - List instances with NICs assigned to the corresponding network
 - List NIC index and IP for the above instances

Implement complementary config methods for retrieving networks.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Implement LUs for corresponding opcodes:
 * LUNetworkAdd:
   - Check for IP validity
   - Reserves all necessary IPs
   - Create new Network config object
 * LUNetworkRemove:
   - Checks if connected to any nodegroup
   - Remove a Network config object

Implement basic config methods:
 * LookupNetwork()
   - Given the network name return the network UUID
 * AddNetwork()
   - Add a new network to the config
 * RemoveNetwork()
   - Remove a network from the config

Add new locking level: LEVEL_NETWORK

Add various useful config methods for retrieving network info.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Add new library module lib/network.py.
Introduce new class: AddressPool.

AddressPool implements all operations needed for managing IPs
inside the IP pool.

Given a Network config object (nobj), the class:

 * initializes the corresponding IP pool object via
   network.AddressPool.InitializeNetwork(nobj)
 * obtains the corresponding IP pool object via
   network.AddressPool(nobj)
 * manipulates IPs inside the pool

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Config objects:
 * Introduce L{Network} with
  - IPv4 network field (mandatory)
  - IPv4 gateway, IPv6 (network/gateway), mac prefix, type (optional)
 * Modify existing config objects to support networks:
  - Add new slot 'network' to L{NIC} config object
  - Add new slot 'networks' to L{NodeGroup} config object

Opcodes:
 * Introduce new opcodes for networks
  - add/remove/modify/query/connect/disconnect.
 * In InstanceCreate/InstanceSetParams add conflicts_check option

Constants:
 * INIC_PARAM 'INIC_NETWORK'
 * NIC_IP_POOL for automaticaly obtain an IP from a pool
 * NETWORK_TYPE_PUBLIC/PRIVATE for network types

Checking of network_type handled by the opcode parameter validation.
Introduce _CheckCIDR*Notation() functions for network parameters validation.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Currently, hail fails with FailDisk when trying to add an instance
of type: 'file', 'sharedfile' and 'rbd'.

This is due to a "0" or None value in the corresponding dict inside
_ComputeDiskSize, which results in a "O" or non Int value of the
exported 'disk_space_total' parameter. This in turn makes hail fail,
when trying to process the value:

 - with "Unable to read Int" if value is None (file)
 - with FailDisk if value is 0 (sharedfile, rbd)

The latter happens because the 0 value doesn't match the instance's
IPolicy, since it is lower than the minimum disk size.

The second problem still exists when using adoption with 'plain'
and 'blockdev' template and will be addressed in another commit.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Due to an oversight, the pause/resume sync RPC calls in _WipeDisks
lack the verification of the overall RPC status, and directly iterate
over the payload. The code actually doing the wipe does verify
correctly the results. This can result in jobs failing with a hard to
diagnose:

OpExecError ['NoneType' object is not iterable]

instead of proper "RPC failed" message.

This patch adds a hard check on the pause call, but for the resume
call it just logs a warning if the RPC failed; the rationale being
that if we can't contact the node for pausing the sync, it's likely
wiping will fail too, but after the wipe has been done, we can
continue.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This breaks the command, as optparse considers that an error.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

It seems that 'explicitely' is wrong, and that the right form is
'explicitly'. This is just fixing the typo plus adjusting affected
paragraphs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

In case LUClusterVerifyGroup is run on a group which doesn't contain
the master node, the following could happen:

- master node is selected due to the explicit check
- if the order of nodes in the 'absent_nodes' list is such that the
  master node is the first in it, then we'll select (again) the master
  node
- passing duplicate nodes to RPC calls will break due to RPC
  internals; this should be fixed separately, but in the meantime we
  just refrain from passing such duplicates

This patch should not change the semantics of the code, since it
wasn't guaranteed even before that we find a vm_capable node.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>

Commit 904b3bfe tried to fix the deletion of custom ndparams from
group, but instead broke both modification and deletion: because we
run ForceDictType on self.op.ndparams instead of the updated
new_ndparams, we can neither delete nor set properly spindle_count
(since it won't be coerced to int).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This generalises commit 4304964a to cdroms too, since they have
somewhat the same logic. We just abstract the needs_boot_flag into a
separate variable, and then reuse it in the cdrom section.

Note that the logic of what 'if=' type to pass to KVM was very
convoluted, and (I think) incorrect; I went and cleaned it to be more
consistent.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This addresses issue 230: passing two methods of booting to KVM can,
depending on the KVM version, confuse it.

Note that commit 4304964a introduced a partial fix for this (but only
for disks, and keyed on KVM versions). However, it didn't fix cdrom
booting, which still fails with the same error, so let's fix it more
generically; we still leave the per-disk check since that is about
-boot c versus -drive …,boot=on rather than two boot methods.

Patch is based on the one submitted by Vladimir Mencl, many thanks!

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

On node groups we don't have the std field. However, the InstancePolicy
object always verifies that the std value is within a given range. As we
fill it up with defaults if not set (as it happens to be on node groups)
and the min value is higher than the default std value (taken from
constants.py) we fail.

We overcome this situation by simply let the function know if we want to
verify the std value at all. If we don't want to verify std, we just set
it to a compliant value (min_v) and continue.

We also slightly adapt the error message provided, as we don't have std
values on groups.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Sorry, I forgot that with GenerateTable this needs to be handled
manually. Fixed now and tested in both ways.

(But to be honest, this should be abstracted in GenerateTable, instead
of the 'if' test in all its callers.)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

This uses confd to query the DRBD minors, which is very special; no
other command currently does so.

Since the backend is only implemented in the Haskell version of confd,
we have checks that 1) confd is enable, and 2) hs confd is also
enabled. If by mistake people do manage to query Python confd, the
error message will be clean:

  Query gave non-ok status '2': not implemented

So nothing breaks in an "ugly" way.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

This is not implemented currently.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

It seems this was not updated since the move to static minors…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>

Commit cce46164 fixed upgrading from other 2.6 versions, but
accidentally broke upgrading from 2.5 (disk.params was left as None,
which breaks FillDict). The simplest way to handle params is to always
set them to an empty dict (disregarding what they currently contain).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

While looking at the testability of this piece of code, I found another
consistency problem. We have two if branches instead of one, with
elif's.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This inconsistency was found during rebalancing. Hbal failed because,
Ganeti couldn't load the opcode. After digging through the cause, an
inconsistency with the "args" field in the LUXI protocol was triggered
by the TemplateHaskell side where it's done uniformed.

For SubmitJob and SubmitManyJobs we treat args as one argument,
containing the job definition. In every other LUXI call args is actually
a list of arguments. This patch fixes this consistency.

This change is NOT backwards compatible.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

This bug was found during disk parameter debugging. While looking at the
config some values seem present on the disk parameters, but that's not
expected behaviour. This patch fixes this, and also fixes the "broken"
configs automatically upon masterd restart.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Agata Murawska <agatamurawska@google.com>