1. 08 Jul, 2008 20 commits
    • Guido Trotter's avatar
      LogicalUnit: add ExpandNames function · d465bdc8
      Guido Trotter authored
      New concurrent LUs will need to call ExpandNames so that any names
      passed in by the user are canonicalized, and can be used by hooks,
      locking and other parts of the code. This was done in CheckPrereq
      before, but it's now splitted out, as it's needed for locking, which in
      turn CheckPrereq needs. Old LUs can be converted gradually.
      
      Reviewed-by: iustinp
      d465bdc8
    • Guido Trotter's avatar
      Processor: Move LU execution to its own method · 36c381d7
      Guido Trotter authored
      This makes the try...finally code simplier, and helps adding a more
      complex locking structure before the actual execution. It also fixes a
      concurrency bug caused by the fact that write_count was read before
      acquiring the BGL, and thus spurious config update hooks run could have
      been triggered. This doesn't solve the issue of running config update
      hooks for concurrent LUs.
      
      Reviewed-by: iustinp
      36c381d7
    • Guido Trotter's avatar
      Add a new LockSet unittest · 2e1d6d96
      Guido Trotter authored
      This test checks the LockSet behaviour when an empty list is passed.
      The current behaviour is expected, but since this is a corner case,
      we're safer to keep it under a check, and if we need a different one
      monitor that everything is as we expect it to be.
      
      Reviewed-by: imsnah
      2e1d6d96
    • Michael Hanselmann's avatar
      constants: Add job and opcode status strings · 5f33b613
      Michael Hanselmann authored
      Reviewed-by: iustinp
      5f33b613
    • Michael Hanselmann's avatar
      workerpool: Don't notify if there was no task · b3558df1
      Michael Hanselmann authored
      Workers have to notify their pool if they finished a task to make
      the WorkerPool.Quiesce function work. This is done in the finally:
      clause to notify even in case of an exception. However, before
      we notified on each run, even if there was no task, thereby creating
      some sort of an endless loop of notifications. In a future patch
      we should split the single condition object into several to
      produce less spurious notifications.
      
      While we're at this, this patch also adds two new functions to
      BaseWorker to query whether it's currently running a task and then
      uses one of these functions in the WorkerPool instead of querying
      the internal variable directly.
      
      Reviewed-by: iustinp
      b3558df1
    • Iustin Pop's avatar
      Create all SUB_RUN_DIRS in ganeti-noded · 195c7f91
      Iustin Pop authored
      Rather than just creating BDEV_CACHE_DIR we loop through the
      SUB_RUN_DIRS list and create all its childs.
      
      Reviewed-by: iustinp
      195c7f91
    • Iustin Pop's avatar
      Add a top level RUN_GANETI_DIR constant · 75afaefc
      Iustin Pop authored
      This patch creates a base RUN_GANETI_DIR and then moves the other run
      dir constants to use that (even if just setting BDEV_CACHE_DIR as equal
      to it, rather than putting it deeper, for now).
      
      Also we create a constant list of all the subdirs we need in RUN_DIR to
      work properly, which we'll use when creating them in ganeti-noded.
      
      Reviewed-by: iustinp
      75afaefc
    • Iustin Pop's avatar
      symlinks: Add DISK_LINKS_DIR constant · bf94c0f5
      Iustin Pop authored
      The DISK_LINKS_DIR points to the RUN_DIR/ganeti/instance-disks
      directory, which will contain symlinks to the instances' disks. These
      provide a stable name accross all nodes for them, and permit
      live-migration to happen.
      
      Unfortunately RUN_DIR/ganeti/instance-disks happens to be below ganeti
      1.2's BDEV_CACHE_DIR, which will we need to address at some point
      (possibly in 2.0).
      
      Reviewed-by: iustinp
      bf94c0f5
    • Michael Hanselmann's avatar
      luxi: Use serializer module instead of simplejson · fad50141
      Michael Hanselmann authored
      Reviewed-by: iustinp
      fad50141
    • Michael Hanselmann's avatar
      serializer.DumpJson: Control indentation by parameter · 071448fb
      Michael Hanselmann authored
      If the simplejson module supports indentation, it's always used. There
      are cases where we might not want to use it or enable it only for
      debugging purposes, such as in RPC.
      
      Reviewed-by: iustinp
      071448fb
    • Guido Trotter's avatar
      Add a missing import to cmdlib · 6048c986
      Guido Trotter authored
      cmdlib uses some constants from locking (ie. locking levels) but doesn't
      import it. This patch fixes the issue.
      
      Reviewed-by: iustinp
      6048c986
    • Guido Trotter's avatar
      Fix an error accessing the cfg · f64c9de6
      Guido Trotter authored
      Since the context is passed to LogicalUnit, rather than the cfg, we can
      only access the cfg as self.cfg, self.context.cfg, or context.cfg (in
      the constructor). cfg is not valid anymore.
      
      Reviewed-by: iustinp
      f64c9de6
    • Guido Trotter's avatar
      Add and remove instance/node locks · a2fd9afc
      Guido Trotter authored
      Whenever we add an instance or node to the cluster (i.e. to the config
      and whenever we remove them we should add/remove locks as well). In the
      future we may want to optimize this so that the configwriter does it, or
      it's handled at the context level, but till we're adding/removing
      instances and nodes with the BGL held it doesn't matter too much.
      
      Reviewed-by: iustinp
      a2fd9afc
    • Guido Trotter's avatar
      Pass context to LUs · 77b657a3
      Guido Trotter authored
      Rather than passing a ConfigWriter to the LUs we'll pass the whole
      context, from which a ConfigWriter can be extracted, but we can also
      access the GanetiLockManager. This also fixes the places where a FakeLU
      is created.
      
      Reviewed-by: iustinp
      77b657a3
    • Guido Trotter's avatar
      mocks: create a FakeContext object · f4555db2
      Guido Trotter authored
      This will be passed to FakeLUs
      
      Reviewed-by: iustinp
      f4555db2
    • Guido Trotter's avatar
      Fix a typo in LUTestDelay docstring · 0b097284
      Guido Trotter authored
      Reviewed-by: iustinp
      0b097284
    • Guido Trotter's avatar
      Locking: remove LEVEL_CONFIG lockset · 08a6c581
      Guido Trotter authored
      Since the ConfigWriter now handles its own locking it's not necessary to
      have a specific level for the config in the Locking Manager anymore.
      This patch thus removes it, and all the unittest calls that used it, or
      depended on it being present.
      
      Reviewed-by: iustinp
      08a6c581
    • Guido Trotter's avatar
      ConfigWriter: synchronize access · f78ede4e
      Guido Trotter authored
      Since we share the ConfigWriter we need somehow to make sure that
      accessing it is properly synchronized. We'll do it using the
      locking.ssynchronized decorator and a module-private shared lock.
      
      This patch also renames a few functions, which were called inside the
      ConfigWriter, to a private version _UnlockedFunctionName, and exports
      the synchronized public ones. The internal callers, which are already
      synchronized, are then changed to use the _Unlocked version, to prevent
      double locking.
      
      Reviewed-by: iustinp
      f78ede4e
    • Guido Trotter's avatar
      Locking: add ssynchronized decorator · 42a999d1
      Guido Trotter authored
      This patch creates a new decorator function ssynchronized in the locking
      library, which takes as input a SharedLock, and synchronizes access to
      the decorated functions using it. The usual SharedLock semantics apply,
      so it's possible to call more than one synchronized function at the same
      time, when the lock is acquired in shared mode, and still protect
      against exclusive access.
      
      The patch also adds a few unit test to check the basic decorator's
      functionality, and to provide an example on how to use it.
      
      Reviewed-by: iustinp
      42a999d1
    • Guido Trotter's avatar
      ConfigWriter: remove _ReleaseLock · 5b4cd1b0
      Guido Trotter authored
      Remove empty function _ReleaseLock and all its calls. Since we only
      have one configwriter per cluster the locking needs to cover all the
      data in the object, and not just the file contents. Locking in
      ConfigWriter will be handled using the ganeti locking library.
      
      Reviewed-by: iustinp
      5b4cd1b0
  2. 04 Jul, 2008 2 commits
    • Iustin Pop's avatar
      Fix some issues with the watcher · 26517d45
      Iustin Pop authored
      This patch fixes two bugs:
        - the state file is not saved because we use the method for checking
          for udpated data
        - in two places 'Error' was used instead of 'Exception', which breaks
          error handling
      
      Additionally:
        - the unused 're' import has been removed
        - a variable named 'id' which collides with a builtin function has
          been renamed
      
      Note that comparing the serialized forms might create false negatives
      (due to the dicts being reordered) but that will just cause an extra
      write of the file, which is sub-optimal but harmless.
      
      Reviewed-by: ultrotter
      26517d45
    • Michael Hanselmann's avatar
      Add generic worker pool implementation · 76094e37
      Michael Hanselmann authored
      Reviewed-by: ultrotter
      76094e37
  3. 03 Jul, 2008 3 commits
    • Iustin Pop's avatar
      Reuse the luxi client in cli.SubmitOpCode · 2cb687ad
      Iustin Pop authored
      By a mistake, we don't reuse the luxi client. As such, we open and close
      the connection at each poll cycle and spam the server logs.
      
      Reviewed-by: ultrotter
      2cb687ad
    • Iustin Pop's avatar
      Add custom logging setup for daemons · 3b316acb
      Iustin Pop authored
      It's better for daemons if:
        - they log only to one log file
        - the log level is included
        - for debug runs, the filename/line number is included
      
      This patch moves the custom formatter from the watcher to the logging
      module and generalizes it; then it changes the master daemon to use this
      function instead of the generic logging (which might be deprecated
      anyway in the future).
      
      Reviewed-by: imsnah
      3b316acb
    • Iustin Pop's avatar
      Remove custom locking code from gnt-instance · fbe5b278
      Iustin Pop authored
      The gnt-instance script doesn't run in the same process anymore, so we
      can't and don't have to unlock.
      
      Reviewed-by: ultrotter
      fbe5b278
  4. 02 Jul, 2008 2 commits
  5. 01 Jul, 2008 10 commits
    • Guido Trotter's avatar
      Context: s/GLM/glm/ · 984f7c32
      Guido Trotter authored
      Make the GanetiLockManager instance of GanetiContext lowercase
      
      Reviewed-by: imsnah
      984f7c32
    • Michael Hanselmann's avatar
      Set locale when using docbook programs · 8ab8e969
      Michael Hanselmann authored
      At least docbook2man inserts a date formatted using the current
      locale into its output.
      
      Reviewed-by: iustinp
      8ab8e969
    • Iustin Pop's avatar
      Update .gitignore · 60053f31
      Iustin Pop authored
      Reviwed-by: imsnah
      60053f31
    • Iustin Pop's avatar
      Add a FirstFree function to utils.py · 7b4126b7
      Iustin Pop authored
      This function will return the first unused integer based on a list of
      used integers (e.g. [0, 1, 3] will return 2).
      
      Reviewed-by: imsnah
      7b4126b7
    • Guido Trotter's avatar
      Increase the thread size to 5 · a478cd7e
      Guido Trotter authored
      Now that we use the locking library to make sure running opcodes cannot
      step on each other toes we can have a bigger thread size, and
      potentially process many opcodes in a parallel manner.
      
      Reviewed-by: iustinp
      a478cd7e
    • Guido Trotter's avatar
      Processor: acquire the BGL for LUs requiring it · 04864530
      Guido Trotter authored
      If a LU required the BGL (all LUs do, right now, by default) we'll
      acquire it in the Processor before starting them. For LUs that don't
      we'll still acquire it, but in a shared fashion, so that they cannot run
      together with LUs that do.
      
      We'll also note down whether we own the BGL exclusively, and if we don't
      and we try to chain a LU that does, we'll fail.
      
      More work will need to be done, of course, to convert LUs not to require
      the BGL, but this basic infrastructure should guarantee the coexistance
      of the old and new world for the time being.
      
      Reviewed-by: iustinp
      04864530
    • Guido Trotter's avatar
      Processor: pass context in and use it. · 1c901d13
      Guido Trotter authored
      The processor used to create a new ConfigWriter when it was initialized.
      We now have one in the context, so we'll just recycle it. First of all
      we'll pass the context in when creating a new Processor object, then
      we'll just use context.cfg, which is granted to be initialized, wherever
      we used self.cfg, and stop checking whether the config is already
      initialized or not.
      
      In the future the Processor will be able to use the context also to
      acquire the BGL for LUs that require it, and to push the context down to
      LUs that don't in order for them to manage their own locking.
      
      Reviewed-by: iustinp
      1c901d13
    • Guido Trotter's avatar
      Add REQ_BGL LogicalUnit run requirement · 7e55040e
      Guido Trotter authored
      When logical units have REQ_BGL set (it is currently the default) they
      need to be the only ganeti operation run on the cluster, and we'll
      guarantee it at the master daemon level. Currently only one thread is
      running at a time, so this requirement is never broken.
      
      Reviewed-by: iustinp
      7e55040e
    • Guido Trotter's avatar
      Burnin doesn't need a Processor · 2da55f7c
      Guido Trotter authored
      In 2.0 burnin submits job to the master daemon, so it doesn't need to
      create an internal Processor anymore. Even if the processor is not used
      anywhere in the burnin code it was still initialized as a leftover of
      how burnin used to work. Fixing this.
      
      Reviewed-by: iustinp
      2da55f7c
    • Iustin Pop's avatar
      Implement “gnt-job list -o +...” · 7a5d3bbd
      Iustin Pop authored
      This adds the same “-o +...” functionality in gnt-job as in the node and
      instance scripts.
      
      Reviewed-by: imsnah
      7a5d3bbd
  6. 30 Jun, 2008 3 commits
    • Guido Trotter's avatar
      Fix sstore handling in Processor · c6868e1d
      Guido Trotter authored
      - no need to keep the sstore as an object member, remove it
      - don't reinitialize sstore only if self.cfg is None
          This is not an issue, as the Processor is recycled for every opcode,
          but in general we know that (a) we might need a different type of
          sstore for different opcodes and (b) initializating them is cheap
      - recreate sstore when chaining opcodes
          Without this fix chaining an opcode which requires a writable sstore
          to one which doesn't would fail. This doesn't happen today, but it's
          better to fix it anyway
      
      These changes are possible because nowadays all opcodes already require
      a working cluster/configuration.
      
      Reviewed-by: iustinp
      c6868e1d
    • Guido Trotter's avatar
      Remove duplicate code in hooks unittests · d2525573
      Guido Trotter authored
      All the tests there used to creare a cfg, a sstore, an opcode and a LU.
      Put all the duplicate code in the setUp function.
      
      Reviewed-by: iustinp
      d2525573
    • Guido Trotter's avatar
      ganeti-masterd: init and distribute common context · 39dcf2ef
      Guido Trotter authored
      This patch creates a new GanetiContext class, which is used to hold
      context common to all ganeti worker threads. As for the
      GanetiLockingManager class it is paramount that there is only one such
      class throughout the execution of Ganeti, so the class checks for that,
      and also forbids its own modification after it's been initialized. The
      context for now contains a ConfigWriter and a GanetiLockingManager and
      is created by the daemon and propagated to PoolWorker(s) and
      JobRunner(s).
      
      Reviewed-by: iustinp
      39dcf2ef