This will show smaller/better error messages: full node dumps are no
longer included.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The node resource locks were not set correctly on instance import.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch changes the luxi submit job calls to use wrapped opcodes,
and therefore it changes Hbal to submit actual meta opcodes. For
nicety, hbal also submits a comment now, showing who generated the
job.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This fixes:

  Got job IDs JobId {fromJobId = 1052613}

And restores to:

  Got job IDs 1052624

Other improvements could be done here, of course.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

If TIsLength is applied to a non-container item, it will fail (type
error) due to invalid application of len(). Since this can happen on
user-supplied data, we add an explicit TList/TTuple check (the TTuple
test is a new one).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds the "meta" opcode type and the common op
params. Compatibility tests with Python are changed to pass Meta
opcodes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This was deprecated via commit 88609f00, “Switch Luxi TH code from
simple to custom fields”.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the opcode and luxi "showJSON" functions generate directly
a JSValue; in contrast, the object (single-constructor) types generate
a 'toDict' function, and then `showJSON = makeObj . toDict`. This is
useful, as the 'dict' form can be manipulated if needed.

This patch changes the opcode and luxi types to behave the same; we
generate a dict, and then (since this differs between opcodes and
luxi) generate showJSON either as `makeObj . toDict` (for opcodes), or
(for luxi) `showJSON . map sn . toDict`.

The change is needed for MetaOpCode implementation.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This is reused in more than just a few places, so adding it makes the
signatures much nicer.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This will go into a separate type; this patch adds the needed
underlying types and parameters.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This mirrors the positive one, and will be needed for relative job
IDs.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Currently, the job ID is a simple type alias. This is suboptimal, as
it means we can't use a custom JSON (or Arbitrary) instance for it.

The patch changes it into a newtype, and then a) simplifies some
deserialisation code and b) changes some more fields to this new type
(rather than plain 'Int').

We also move the JobId to types, since it will be needed in opcodes as
well.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

For now, we don't need a pending job status type as well, so we'll
delay adding that until later.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

As opposed to the existing test, which tests the type/serialisation of
fields, this one simply tests the equivalence of the list of fields
for each opcode.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 213076fe (“Fix locking in networks”) changed Python OpNetworkAdd
without corresponding Haskell definition changes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Due to manual conversion, a few fields were missing from the
conversion, but as they were optional our type equivalence checking
didn't detect this.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The original htools opcodes were minimalistic and not 1:1 equivalent
with the Python ones. Let's add all missing fields and, since we
changed the order, switch to more readable record syntax for building
the opcodes.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch changes THH to export a new function which defines all
fields of a given OP_ID. Not very clean, since for an invalid OP_ID we
return empty list, but since it will only be used in tests it should
be good enough.

The generated code looks as follows:

    allOpFields :: String -> [String]
    allOpFields "OP_TEST_DELAY"
                  = ["duration", "on_master", "on_nodes", "repeat"]
    allOpFields "OP_INSTANCE_REPLACE_DISKS"
                  = ["disks", "early_release", "iallocator", "ignore_ipolicy",
                     "instance_name", "mode", "remote_node"]
    …
    allOpFields _ = []

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch changes a bit the handling of custom fields. Since in
general we use custom fields to aggregate multiple entries in the JSON
object into a safer data-type, we should also have a way to declare
which extra entries this field covers (so that in the future we can
say what are all the JSON keys for an object).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Just some cleanup:

- don't run hlint over hpc-htools, since it's just a symlink to
  htools.hs (so we'd get duplicate warnings)
- build rpc-test in make really-all, via HS_ALL_PROGS variable

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Since hlint doesn't check style issues but rather code only issues,
let's add a simple Makefile check for too-long lines, and fix the
single bad case we have currently.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Indentation within rules is fixed to always be 2 spaces (as opposed to
mixed tabs/2 spaces/4 spaces). Additionally, the check-dirs rule,
which was more complex than necessary, has been simplified: moved
auto4mte.cache to DIRCHECK_EXCLUDE, and simplify the find call and the
exit condition (no need for if test …; then exit 0; else exit 1; fi).

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The current uniqueness algorithm (generate random node names, suffix
them with node index) is actually wrong: a node named "21" at index 5
will end up with the same name as a node named "2" at position 15.

To fix this, we also add a character from a different "set" ("-"), so
that such mixups can't happen again, and also add an explicit check
for it.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

All resources are acquired and opportunistic instance creations would
fail. Also add a TODO.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Opportunistic locks are not yet used. This patch changes
LUInstanceCreate to acquire the node allocation lock to avoid conflicts
with other opcodes acquiring many node locks.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

See code for an explanatory comment. The lock is released as soon as
possible.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

LUNetworkConnect/Disconnect looks up a nodegroup's instances for
conflicting IPs. To do so, use GetNodeGroupInstances() and
GetMulitInstanceInfo().

Additionally, check if the correct locks were acquired.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

The network management code needs to verify a MAC address prefix.
Instead of (ab)using NormalizeAndValidateMac, clean code should be used.
Unit tests for NormalizeAndValidateMac are updated and new ones for
NormalizeAndValidateThreeOctetMacPrefix are added.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Instead of having two different lists of directories, now there is only one.

Signed-off-by: Michele Tartara <mtartara@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This code is going to be used by a new utility for setting up the node
daemon. Unit tests are updated/added.

Additionally, the certificate and key stored in “server.pem” are
verified, too.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Similar to “share_locks”, a new dictionary containing booleans for each
locking level is added to “cmdlib.LogicalUnit”. Logical units wanting to
make use of opportunistic locks will be able to configure this
dictionary accordingly.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Just forwarding the parameter, nothing more.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This patch adds a new parameter to “LockSet.acquire” named
“opportunistic”. When enabled the lockset will try to acquire as many
locks as possible, but it won't wait for them (with the exception of the
lockset-internal lock in case the whole set is acquired). This is
implemented by using a timeout of 0 seconds when acquiring individual
locks. Commit 03c5291c made such acquisitions significantly cheaper.

The most complicated code included in this patch is probably the helper
function used to determine which mode to use and which timeout functions
are needed.

Full unit tests are included for the new and changed code
(“locking.py”'s overall coverage is at 97%).

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Configuring a node daemon on a newly added node will need all ssconf
values.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Commit 224ff0f7 modified the node SSH setup to use the system's SSH
client. Before that Paramiko was used. It's not entirely clear whehter
the latter ever supported IPv6 properly, but with this patch
“bootstrap.RunNodeSetupCmd” is changed to use it if configured. The code
for doing so is taken from “bootstrap.SetupNodeDaemon”, which is going
to use “bootstrap.RunNodeSetupCmd”.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Part of the code used for running “prepare-node-join” can be re-used for
running a tool to configure the node daemon.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

A new utility for configuring the node daemon will support a dry-run
mode. This patch adds the necessary functionality to
“ssconf.SimpleStore” and provides comprehensive tests for
“SimpleStore.WriteFiles”. To enable the latter, a testing-only parameter
specifying the lockfile was added to the class.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

The new utility for configuring the node daemon will have to check
whether it received valid ssconf names.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Add the design document for opportunistic locking.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

This function can be applied to any type of class "Element" but its
internal variables used to refer to a list of tuples containing these
elements and their names as "nodes". Moreover it built a list of name to
indices tuples referring to it as "na" (which is not very clear).

Thus nodes becomes name_element, and na name_idx.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>