Check if extstorage's log dir exists on the host machine, before
providing a logfile name. We need to be sure the dir exists,
otherwise RunCmd will fail ugly when passed a logfile that cannot
be opened.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>

During attach the logfile is `None'. If the attach script
fails and we try to Tailfile() its logfile, TailFile
breaks with:

TypeError: coercing to Unicode: need string or buffer, NoneType found

Thus, we don't show the logfile, if ExtStorage's attach script fails.
Just throw the appropriate error.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>

Commit 5fce6a89 changed RpcRunner._InstDict to add the disk parameters
on all encoded instances. It didn't remove a special case in
“_InstDictOspDp”. Update and fix unittests as well.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: René Nussbaumer <rn@google.com>

We call _GatherAndLinkBlockDevs during the process, which in turn
calls _RecursiveFindBD. This needs disk parameters to work.

See also commit b8291e00.

This was reported by Ansgar and Damien.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

When locks are copied “locking.ALL_SET” must be handled separately
(ALL_SET has the value None). Reported by Constantinos Venetsanopoulos
who saw failover for RDB-based instances not working.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is only needed in case base64 and bitarray.tostring() is used for
saving reservations. If bitarray.to01() was used then we would not have
this restriction.

Signed-off-by: Dimitris Aragriorgs <dimara@grnet.gr>

There was an issue with the recent ipolicy introduction which lead to a
bug in gnt-debug iallocator. It was not providing the spindle_use field
and therefore it wont let you create a valid iallocator request.

Signed-off-by: René Nussbaumer <rn@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

To help developing Ganeti on newer distributions, let's try to fix
pylint warnings/errors. I'm using pylint from current Debian wheezy:
pylint 0.25.1, astng 0.23.1, common 0.58.0, and we have 3 things that
needs fixing.

First, a really wide "except", with the silencing in the wrong
place. I'm not sure why this doesn't have "except Exception", so let's
add it. However, pylint still complains about "Catching too general
exception", even though we do want to catch both system and our
exception, so let's add a silence for W0703. It's true that we
shouldn't catch KeyboardInterrupt and friends, but that should be
cleaned up on the master branch.

Second, pylint complains about "redefining name builtin tuple",
because we do some pattern matching in the except blocks in
netutils. This seems to be a false positive, but let's clean the code
around this.

And finally, type inference again goes bad, so let's silence E1103
with its "boolean doesn't have 'get' method".

After this, I can run "make lint", and by extension "make
commit-check" on Debian Wheezy, yay! We might be able to bump our
required pylint versions to something not ancient…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Pylint version:

  pylint 0.25.1,
  astng 0.23.1, common 0.58.0

crashes when passing the fully-qualified decorator name with:

  File "/usr/lib/pymodules/python2.7/pylint/checkers/base.py", line 161, in visit_function
    if not redefined_by_decorator(node):
  File "/usr/lib/pymodules/python2.7/pylint/checkers/base.py", line 116, in redefined_by_decorator
    decorator.expr.name == node.name):
AttributeError: 'Getattr' object has no attribute 'name'

I found out that simply using a shortened name will 'fix' this issue,
so let's do this to allow running newer pylint versions.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Signed-off-by: Dimitris Aragriorgs <dimara@grnet.gr>

Signed-off-by: Dimitris Aragriorgs <dimara@grnet.gr>

Introduce _GenerateKVMBlockDevicesOptions()

In case hotplug is enabled per instance then the runtime file
contains one more field: block_devices. In this case kvm_cmd
is extended with block device options during _ExecuteKVMRuntime().

If no hotplug is enabled then runtime file has block_devices info
encapsulated in kvm_cmd and block device options are exteded
during _GenerateKVMRuntime.

Signed-off-by: Dimitris Aragriorgs <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Add a new client called 'gnt-storage'.
The client interacts with the ExtStorage interface, similarly to
the way gnt-os interacts with the OS interface.

For now, only two commands are supported: 'info' and 'diagnose'.

'diagnose' calculates the node status of each provider on each node,
similarly to gnt-os diagnose. Furthermore, for every provider, it
calculates it's nodegroup validity for each nodegroup. This is done
inside the LU and not the client (marked as 'TODO' for the  global
validity of gnt-os diagnose).

In the future, gnt-storage can be used to manage storage pools,
or even be extended to diagnose other storage types supported by
Ganeti, such as lvm, drbd (INT_MIRROR) or rbd (EXT_MIRROR).

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>

Add support for passing parameters to the ext template (ext-params).
Take advantage of disk-params, that don't seem to make much sense in
this template (ExtStorage Providers are not predefined and we don't
know their needs) and use them to pass the ext-params dynamically to
the template.

ext-params are correlated with gnt-os-interface's os-params.
All ext-params are exported to the ExtStorage Provider through it's
environment, with variables prefixed with 'EXTP_' (similarly to the
OS interface's 'OSP_' params).

ext-params are passed through the --disk option. If the disk template
is of type `ext' during instance add, then any additional options that
are not in IDISK_PARAMS given to --disk are considered ext-params
e.g.:

 gnt-instance add -t ext --disk=0:size=2G,param1=value1,param2=value2

Finally, we introduce a new IDISK_PARAM called IDISK_PROVIDER, that is
mandatory for template `ext' and is used to select the desired
ExtStorage Provider. This parameter is not valid for other template
types.

The IDISK_PROVIDER parameter becomes the first element of the
disk's unique_id tuple e.g.:

 unique_id = ('sample_provider1', 'UUID.ext.diskX')

Example selecting different ExtStorage Providers for each disk and
passing different ext-params to them:

 -t ext --disk=0:size=2G,provider=sample_provider1,param1=value1
        --disk=1:size=3G,provider=sample_provider2,param2=value2

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>

With this commit we introduce the External Storage Interface
to Ganeti, abbreviated: ExtStorage Interface.

The ExtStorage Interface provides Ganeti with the ability to interact
with externally connected shared storage pools, visible by all
VM-capable nodes. This means that Ganeti is able to handle VM disks
that reside inside a NAS/SAN or any distributed block storage provider.

The ExtStorage Interface provides a clear API, heavily inspired by the
gnt-os-interface API, that can be used by storage vendors or sysadmins
to write simple ExtStorage Providers (correlated to gnt-os-interface's
OS Definitions). Those Providers will glue externally attached shared
storage with Ganeti, without the need of preprovisioned block devices
on Ganeti VM-capable nodes as confined be the current `blockdev' disk
template.

To do so, we implement a new disk template called `ext' (of type
DTS_EXT_MIRROR) that passes control to externally provided scripts
(the ExtStorage Provider) for the template's basic functions:

 create / attach / detach / remove / grow

The scripts reside under ES_SEARCH_PATH (correlated to OS_SEARCH_PATH)
and only one ExtStorage Provider is supported called `ext'.

The disk's logical id is the tuple ('ext', UUID.ext.diskX), where UUID
is generated as in disk template `plain' and X is the disk's index.

Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>

Add --hotplug option.
Only used by OpInstanceCreate()/OpInstanceSetParams.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Hotplugging is done by functions invoked by ApplyContainerMods().  In
order hotplugging to take place a --hotplug option must be passed.
Hotplug can enabled for a VM if --hotplug is passed during
gnt-instance add (hotplug_info will be added).

In order initial devices of a VM to be hotpluggable hotplug_info must be
early available. So _GetPCIInfo() gets either lu.hotplug_info in case of
LUInstanceCreate() or invokes lu.cfg.GetPCIInfo() in case of
LUInstanceSetParams(). To return a pci slot into the pool
lu.cfg.UpdatePCIInfo() must be invoked.

If no --hotplug is passed during gnt-instance modify --net add the nic
won't get a pci value and will be placed automaticaly in a free pci slot
after reboot. This means that is not going to be hotpluggable for the
rest of the instance's life.

To keep pci pool consistent we do not let a device that has been
hotplugged (i.e. has pci value) to be removed without --hotplug.

Before hotplugging a Disk it must be assembled. During
LuInstanceSetParams() newly created disks are not added to the instance
so _ExpandCheckDisks() in _AssembleInstanceDisks() will fail. So we make
this check optional only for this case.

In order to remove a disk (with blockdev_remove) it must be shutdown. So
after unplugging the disk _ShutdownDiskInstanceDisks() must be invoked.

Everything else is just like nic hotplugging.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Introduce new RPCs that eventually invoke hypervisor specific
hotplug functions.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

KVM >= 1.0 is needed.

Introduce new methods: HotAddNic/HotDelNic

Both qemu monitor interface for hotplugging. If a device has
hotplug info (idx, pci) then:
 - it gets named after it (e.g. virtio-net-pci.%d % idx ).
 - it gets placed to the pci slot (hex(pci)) of bus 0.
If no hotplug info exist or kvm version is old then no hotplug takes
place and modifications will take place after reboot:
 - devices will get traditionaly named
 - and placed automaticaly in empty pci slots

NIC hotplug:
 - open a tap and get its file descriptor.
 - pass fd with SCM rights (using python-fdsend) via monitor socket
 - create netdev and device with proper qemu interface

Disk hotplug:
 - create drive with drive%d % idx name
 - create device virtio-blk-pci.%d % idx

In order to migrate a VM that has devices hotplugged, an identical
VM should be booted with -incoming. To this end every time a hotplug
takes place runtime file must be updated.

For disk hotplug support we removeds disk from kvm_cmd in
runtime file add added them separately just like nics. So code building
disk related options has been moved out from _GenerateKVMRuntime()
and added to _ExecuteKVMRuntime().

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Introduce new HotplugInfo object as part of Instance object.

hotplug_info { 'nics': 0, 'disks': 0, 'pci_pool': [16, 17, 18] }

nics/disks is an index used for device identification and naming.
Everytime a new device (disk/nic) is hotadded these values get incremented.

pci_pool is a list of pci slots where a device can be placed. Every
time a device gets hot-plugged/unplugged a pci slot gets popped/appended.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Support:
 - gnt-network add --tags.
 - gnt-network list-tags/add-tags/remove-tags/.
 - gnt-network list -o +tags

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

If a nic has a network field then encapsulate a network object in
netinfo slot for every rpc. This is needed to pass network info to
scripts managing nics (kvm-vif-bridge).

Introduce _BuildNetworkEnv()

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Modify existing GenerateMAC so that it takes network as an argument.

Substitute _GenerateOneMAC with a decorator _GenerateMACPrefix
that chooses the prefix (network's or cluster's) and wraps
_GenerateMACSuffix that generates the three remaining bytes.

Check mac prefix validity when adding/modifing a network.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Support:
 - GetNetwork(s)
 - CreateNetwork
 - ConnectNetwork
 - DisconnectNetwork
 - RemoveNetwork

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

gnt-network is used to manipulate and handle networks that
currently provides the following operations:

 * Add a new network:
   gnt-network add --network=1.2.3.0/28 --gateway=1.2.3.1
                   --add-reserved-ips=1.2.3.4,1.2.3.5 testnet

 * Remove an existing network:
   gnt-network remove testnet

 * Modify an existing network:
   gnt-network modify --gateway=1.2.3.6
                      --network-type=private
                      --network6=2001:648::/64
                      --gateway6=none testnet
                      --add-reserved-ips=1.2.3.10,1.2.3.10,
                      --remove-reserved-ips=1.2.3.20
                      testnet

 * Connect an existing network to a nodegroup:
   gnt-network connect testnet default bridged br100
   gnt-network connect testnet <nodegroup> <mode> <link>
   (pass all for <nodegroup> to connect to all nodegroups)

 * Disconnect an existing network from a nodegroup:
   gnt-network disconnect testnet <nodegroup>
   (pass all for <nodegroup> to disconnect from all nodegroups)

 * List available networks:
   gnt-network list

 * Show network info:
   gnt-network info [testnet]

Introduce new option NOCONFLICTSCHECK_OPT for not checking for
conflicting IPs. Using this might cause data inconsistency.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Add --no-conflicts-check option.
Modify instance query methods to return nic's network info.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Specifically:
 * export a new environment variable for the gnt-os-interface
 * export a new environment variable for KVM specific scripts
 * ovf support for networks

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Keep backwards compatibility just like LUInstanceCreate() and
support network parameter for NIC object.

Add ec_id argument in Update() to be able to commit temporary
IPs that were previously reserved.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Release any IPs held the instance before actually removing.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Implement backend support, to export the IP pool management
functionality to the clients. When the new NIC parameter 'network'
is given, the ippool management system is triggered.
If a NIC belongs to a network, it inherits the netparams
(mode, link) as its nicparams. If a network is requested, then
no 'mode' or 'link' should be specified. Backwards compatibility
is preserved, if 'network' parameter is omitted.

Examples for IP pool management system:

--net 0:network=net1
  for an IPless NIC in a network

--net 0:ip=pool,network=net1
  for automatically generated IP inside the network

--net 0:ip=1.2.3.4,network=net1
  for specific IP inside the network

--net 0:network=none
  for IPless NIC with default nicparams

Examples for traditional assignment:

--net 0:ip=1.2.3.4,link=br100,mode=bridged
--net 0:ip=none,link=rt2,mode=routed
--net 0

Check for conflicting IPs to ensure IP uniqueness inside nodegroups.

IAllocator should be extended to become network aware. If an instance
has a NIC inside a network, then IAllocator must pick a node that its
nodegroup is already connected with that network.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Use TemporaryReservationManager for IP handling to ensure
config data consistency.

Implement:
 - methods for reserving/releasing IPs
 - _UnlockedCommitTemporaryIps()
 - CheckIPInNodeGroup() used for conflicting IPs
 - GetGroupNetParams() used for inheriting netparams.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

LUNetworkConnect() connects a network to a nodegroup and defines
its netparams (mode and link). Specifically:
 * Check if network already exists
 * Check if netparams are valid
 * Check if already connected to the same nodegroup
 * Produce errors if any conflicting IPs found
 * Update the 'networks' slot in NodeGroup config object
   with the network UUID and netparams requested. E.g.
   nodegroup_uuid {
     networks {
       network_uuid: {mode: 'bridged', link: 'br100'}
     }
   }

LUNetworkDisconnect() unmaps a network from a nodegroup:
 * Produce errors if an instance has a NIC with IP inside the network
 * Remove corresponding entry in 'networks' slot of NodeGroup object

Check for conflicting ips can be avoided with corresponding option.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Support modifying all network parameters except for network (ip range)
Cannot modify gateway and reserved ips at the same time.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Summarily list all existing networks
Supply detailed info for every existing network
 - List used/free IPs
 - List instances with NICs assigned to the corresponding network
 - List NIC index and IP for the above instances

Implement complementary config methods for retrieving networks.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Implement LUs for corresponding opcodes:
 * LUNetworkAdd:
   - Check for IP validity
   - Reserves all necessary IPs
   - Create new Network config object
 * LUNetworkRemove:
   - Checks if connected to any nodegroup
   - Remove a Network config object

Implement basic config methods:
 * LookupNetwork()
   - Given the network name return the network UUID
 * AddNetwork()
   - Add a new network to the config
 * RemoveNetwork()
   - Remove a network from the config

Add new locking level: LEVEL_NETWORK

Add various useful config methods for retrieving network info.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>