1. 04 Jan, 2010 1 commit
  2. 25 Nov, 2009 1 commit
  3. 05 Nov, 2009 1 commit
    • Michael Hanselmann's avatar
      Add new “daemon-util” script to start/stop Ganeti daemons · f154a7a3
      Michael Hanselmann authored
      
      
      Until now, Ganeti started and stopped its own daemons using custom functions.
      To start, the daemon was just executed and then sent the appropriate signals to
      stop it again. Init scripts would have to pay attention to the PID file and
      other things.
      
      With this patch, a new script is added (“daemon-util”, installed in
      $prefix/lib/ganeti/), centralizing the starting and stopping of daemons. The
      provided example init script is adjusted to use this new script. Ganeti's code
      no longer calls its own init script.
      Signed-off-by: default avatarMichael Hanselmann <hansmi@google.com>
      Reviewed-by: default avatarGuido Trotter <ultrotter@google.com>
      f154a7a3
  4. 18 Sep, 2009 1 commit
  5. 26 Aug, 2009 1 commit
  6. 24 Jul, 2009 1 commit
  7. 25 May, 2009 2 commits
    • Iustin Pop's avatar
      watcher: automatically restart noded/rapi · c4f0219c
      Iustin Pop authored
      
      
      This patch makes the watcher automatically restart the node and rapi
      daemons, if they are not running (as per the PID file).
      
      This is not an exhaustive test; a better one would be TCP connect to the
      port, and an even better one a simple protocol ping (e.g. get / for rapi
      and a rpc_call_alive for noded), but since we don't know how they've
      been started we can't implement it today. rapi would need to write the
      SSL/port to a file, and noded something similar, so that we know how to
      connect.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
      c4f0219c
    • Iustin Pop's avatar
      watcher: handle full and drained queue cases · 24edc6d4
      Iustin Pop authored
      
      
      Currently the watcher is broken when the queue is full, thus not
      fulfilling its job as a queue cleaner. It also doesn't handle nicely the
      queue drained status.
      
      This patch does a few changes:
        - first archive jobs, and only after submit jobs; this fixes the case
          where the queue is already full and there are jobs suited for
          archiving (but not the case where the jobs all too young to be
          archived)
        - handle nicely the job queue full and drained cases—instead of
          tracebacks, log such cases nicely
        - reverse the initial value and special cases for update_file; we now
          whitelist instead of blacklist cases, since we have much more
          blacklist cases than vice versa, and we set the flag to True only
          after the run is successful
      
      The last change, especially, is a significant one: now errors during the
      watcher run will not update the status file, and thus they won't be lost
      again in the logs.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      Reviewed-by: default avatarMichael Hanselmann <hansmi@google.com>
      24edc6d4
  8. 20 May, 2009 1 commit
  9. 19 May, 2009 1 commit
    • Iustin Pop's avatar
      watcher: try to restart the master if down · 7dfb83c2
      Iustin Pop authored
      
      
      Bugs in either our code or in associated libraries can bring the master daemon
      down, and this (due to the 2.0 architecture) stops all work on the cluster.
      
      Since the watcher already does periodic checks on the cluster, we modify
      it to try to start the master automatically in case of failures to
      connect. This will be tried only once per cycle.
      
      Also, in this case, we modify the code so that the watcher status file
      is not updated - its timestamp will reflect thus the time of last
      successful connection to the master.
      
      Side note: the except errors.ConfigurationError part could be cleaned
      up, since in 2.0 we don't usually get that directly, and if we do it's
      an error and we shouldn't touch the file anyway; but that is not a rc5
      change.
      Signed-off-by: default avatarIustin Pop <iustin@google.com>
      7dfb83c2
  10. 06 Apr, 2009 2 commits
    • Iustin Pop's avatar
      Fix the output of watcher on non-master nodes · 2c404217
      Iustin Pop authored
      Currently the watcher spews errors message on non-master nodes. This
      cleans it up.
      
      Reviewed-by: imsnah
      2c404217
    • Iustin Pop's avatar
      Change the watcher to use jobs instead of queries · 6dfcc47b
      Iustin Pop authored
      As per the mailing list discussion, this patch changes the watcher to
      use a single job (two opcodes) for getting the cluster state (node list
      and instance list); it will then compute the needed actions based on
      this data.
      
      The patch also archives this job and the verify-disks job.
      
      Reviewed-by: imsnah
      6dfcc47b
  11. 09 Mar, 2009 1 commit
    • Iustin Pop's avatar
      watcher: fix startup sequence locking the master · cc962d58
      Iustin Pop authored
      Currently, the watcher startup sequence does:
        - open a luxi client
        - get the instance list
        - get the node boot ids
        - open and lock the status file, and:
          - archive jobs
          - restart the down instances
          - check disks
      
      This, of course, can lead to problems when a node is (genuinely or not)
      locked for more than (watcher interval * maximum query clients) time. At
      that time, the master is completely unresponsive until the node is
      unlocked and all the watchers exit with error due to the state file
      being locked by the first instance.
      
      This patch reworks the startup sequence to first open/lock the status
      file, and only then open a luxi client. This should prevent the above
      case.
      
      Reviewed-by: ultrotter
      cc962d58
  12. 24 Feb, 2009 1 commit
    • Iustin Pop's avatar
      Remove the extra_args parameter in instance start · 07813a9e
      Iustin Pop authored
      This patch removes the extra_args parameter and instead switches the
      instance to the HV_KERNEL_ARGS hypervisor option.
      
      This is a big change, but it's a needed cleanup, this extra parameter on
      all RPC calls is not generic and we also need to have a persistent value
      here.
      
      Reviewed-by: imsnah
      07813a9e
  13. 16 Feb, 2009 2 commits
    • Iustin Pop's avatar
      watcher: fix checking of boot IDs · 3448aa22
      Iustin Pop authored
      The recent change (commit 2151) to the watcher to make it handle offline
      nodes also saves the offline attribute to the state file, but this is
      not needed and also breaks the checking of the boot ID. This patch
      simply removes it, restoring the correct behaviour.
      
      Reviewed-by: imsnah
      3448aa22
    • Iustin Pop's avatar
      watcher: autoarchive old jobs · f07521e5
      Iustin Pop authored
      This patch adds auto-archiving of jobs older than 6 hours to the
      watcher.
      
      Reviewed-by: imsnah
      f07521e5
  14. 04 Feb, 2009 1 commit
    • Iustin Pop's avatar
      Implement lockless query operations · ec79568d
      Iustin Pop authored
      This patch adds the framework for, and enables lockless OpQueryInstances. This
      means that instances will be shown in ERROR_up or ERROR_down state, even though
      this is not an error (but just an in-progress job).
      
      The framework is implemented as follows:
        - the OpQueryInstances, OpQueryNodes and OpQueryExports opcodes take
          an additional “use_locking” flag which will denote whether to lock
          or not; this patch only implements this for LUQueryInstances
        - the luxi query functions take an additional argument use_locking
          which is passed to the master daemon, and then passed to the above
          opcodes
        - cli.py export a new SYNC_OPT command line options which implement
          setting this flag to true
        - except for gnt-instance list, which uses this option, and for
          name-only queries (e.g. QueryNodes(fields=["names"])), all other
          callers are setting this flag to True
        - RAPI also sets the flag to True
      
      The patch was tested with a continuous (0.2s sleep in-between)
      gnt-instance list during a burnin, and no problems were observed.
      
      Reviewed-by: ultrotter
      ec79568d
  15. 13 Jan, 2009 1 commit
  16. 11 Dec, 2008 1 commit
    • Iustin Pop's avatar
      Fix epydoc format warnings · c41eea6e
      Iustin Pop authored
      This patch should fix all outstanding epydoc parsing errors; as such, we
      switch epydoc into verbose mode so that any new errors will be visible.
      
      Reviewed-by: imsnah
      c41eea6e
  17. 05 Dec, 2008 1 commit
    • Iustin Pop's avatar
      watcher: handle offline nodes better · cbfc4681
      Iustin Pop authored
      This patch changes the LUQueryInstances to show a different state for
      offline nodes and also modifies the watcher to understand the offline
      state in its checks.
      
      Reviewed-by: ultrotter
      cbfc4681
  18. 20 Oct, 2008 1 commit
    • Iustin Pop's avatar
      Remove the logger.py module · 82d9caef
      Iustin Pop authored
      Since now we use only one function from the logger module
      (SetupLogging), we move it to utils.py (which is already imported by all
      users of this function), and we remove the module.
      
      Reviewed-by: imsnah
      82d9caef
  19. 01 Oct, 2008 4 commits
    • Michael Hanselmann's avatar
      Convert ganeti-watcher · 2859b87b
      Michael Hanselmann authored
      Use RPC calls instead of ssconf.
      
      Reviewed-by: iustinp
      2859b87b
    • Iustin Pop's avatar
      Fix the watcher with down nodes · 37b77b18
      Iustin Pop authored
      The watcher didn't handle the down nodes, fix this by ignoring (in
      secondary node reboot checks) any node that doesn't return a boot id.
      
      Reviewed-by: imsnah
      37b77b18
    • Iustin Pop's avatar
      Fix the watcher not restarting instance bug · b7309a0d
      Iustin Pop authored
      The watcher was using conflicting attributes of the instance:
        - it queried the admin_/oper_state, which are booleans
        - but it compared those to the status (which is a text field)
      
      The code was changed to query the aggregated 'status' field, as that
      will also return indication of node problems, and we can use this only
      one field for all decisions. We still ask for the admin_state field as
      that is needed for the activate disks check (in secondary node restart).
      
      The patch also touches the watcher in some other parts:
        - log exceptions nicer
        - convert a method to @staticmethod
        - remove unused imports
      
      Reviewed-by: imsnah
      b7309a0d
    • Iustin Pop's avatar
      Remove last use of utils.RunCmd from the watcher · 5188ab37
      Iustin Pop authored
      The watcher has one last use of ganeti commands as opposed to sending
      requests via luxi. The patch changes this to use the cli functions.
      
      The patch also has two other changes:
        - fix the docstring for OpVerifyDisks (found out while converting
          this)
        - enable stderr logging on the watcher when “-d” is passes
      
      Reviewed-by: imsnah
      5188ab37
  20. 07 Aug, 2008 1 commit
  21. 30 Jul, 2008 1 commit
    • Iustin Pop's avatar
      Unify SetupDaemon/SetupLogging · 59f187eb
      Iustin Pop authored
      The 'old-style' info, error, debug logs do not make much sense. This
      patch unifies the SetupLogging and SetupDaemon functions. As a result,
      all the commands logs to a 'commands.log' file.
      
      The patch also changes the log setup to keep going if there's an error
      in setting up the file logging but we're logging to stderr.
      
      Also, burnin now logs to its own file (burnin.log).
      
      Reviewed-by: ultrotter
      59f187eb
  22. 10 Jul, 2008 1 commit
  23. 04 Jul, 2008 1 commit
    • Iustin Pop's avatar
      Fix some issues with the watcher · 26517d45
      Iustin Pop authored
      This patch fixes two bugs:
        - the state file is not saved because we use the method for checking
          for udpated data
        - in two places 'Error' was used instead of 'Exception', which breaks
          error handling
      
      Additionally:
        - the unused 're' import has been removed
        - a variable named 'id' which collides with a builtin function has
          been renamed
      
      Note that comparing the serialized forms might create false negatives
      (due to the dicts being reordered) but that will just cause an extra
      write of the file, which is sub-optimal but harmless.
      
      Reviewed-by: ultrotter
      26517d45
  24. 03 Jul, 2008 1 commit
    • Iustin Pop's avatar
      Add custom logging setup for daemons · 3b316acb
      Iustin Pop authored
      It's better for daemons if:
        - they log only to one log file
        - the log level is included
        - for debug runs, the filename/line number is included
      
      This patch moves the custom formatter from the watcher to the logging
      module and generalizes it; then it changes the master daemon to use this
      function instead of the generic logging (which might be deprecated
      anyway in the future).
      
      Reviewed-by: imsnah
      3b316acb
  25. 18 Jun, 2008 8 commits
  26. 13 May, 2008 2 commits
    • Iustin Pop's avatar
      Watcher: do not activate disks for started instances · eee1fa2d
      Iustin Pop authored
      Currently the watcher runs first the instance startup and then the
      boot-id method of disk reactivation. However, irrelevant of the fact
      that a node has rebooted or not, if we just started an instance, there's
      no need for its disks to be activated again, since the start instance
      has done that (if it is at all possible).
      
      The patch modifies the watcher to remember all started instances and not
      run activate-disks for them.
      
      Reviewed-by: ultrotter
      eee1fa2d
    • Iustin Pop's avatar
      Watcher: do not activate disks for admin_down · 0c0f834d
      Iustin Pop authored
      Currently the watcher does activate disks (via bootid mechanisms) even
      for admin_down instances.  This patch logs and skips over these
      instances.
      
      Reviewed-by: ultrotter
      0c0f834d