diff --git a/.gitignore b/.gitignore
index 99658f06399c1cd16684d2f9d9392b9536e428aa..76be0df09bbbbc8c1a39f40653b0fde95f99dbc8 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,6 +8,9 @@ bin/
share/
build/
include/
+!snf-cyclades-app/synnefo/ui/static/snf/extra/noVNC/include/
+_sass/
+.sass-cache/
*.pt.py
*.installed.cfg
*.sqlite
@@ -20,7 +23,7 @@ selenium-server-standalone-2.0b2.jar
.settings/
settings.d/*-local.conf
*.egg-info
-dist
+#dist
_build
.coverage
@@ -31,6 +34,7 @@ pithos/data
*/synnefo/versions/*.py
*/pithos/*/version.py
!*/synnefo/versions/__init__.py
+snf-admin-app/synnefo_admin/version.py
snf-astakos-app/astakos/version.py
snf-tools/synnefo_tools/version.py
snf-quotaholder-app/quotaholder_django/version.py
@@ -41,5 +45,8 @@ astakosclient/astakosclient/version.py
snf-django-lib/snf_django/version.py
snf-branding/synnefo_branding/version.py
snf-deploy/snfdeploy/version.py
+# Temp dirs generated by snf-deploy keygen
+snf-deploy/files/root/.ssh
+snf-deploy/files/root/ddns
*.egg
*.tar.gz
diff --git a/COPYRIGHT b/COPYRIGHT
index 31b631fea2591d9f5f20c492a920c4be01b22077..94a9ed024d3859793618152ea559a168bbcbb5e2 100644
--- a/COPYRIGHT
+++ b/COPYRIGHT
@@ -1,32 +1,674 @@
-Copyright (C) 2010-2014 GRNET S.A. All rights reserved.
-
-Redistribution and use in source and binary forms, with or
-without modification, are permitted provided that the following
-conditions are met:
-
- 1. Redistributions of source code must retain the above
- copyright notice, this list of conditions and the following
- disclaimer.
-
- 2. Redistributions in binary form must reproduce the above
- copyright notice, this list of conditions and the following
- disclaimer in the documentation and/or other materials
- provided with the distribution.
-
-THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A. OR
-CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-POSSIBILITY OF SUCH DAMAGE.
-
-The views and conclusions contained in the software and
-documentation are those of the authors and should not be
-interpreted as representing official policies, either expressed
-or implied, of GRNET S.A.
+ GNU GENERAL PUBLIC LICENSE
+ Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc.
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+ Preamble
+
+ The GNU General Public License is a free, copyleft license for
+software and other kinds of works.
+
+ The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works. By contrast,
+the GNU General Public License is intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users. We, the Free Software Foundation, use the
+GNU General Public License for most of our software; it applies also to
+any other work released this way by its authors. You can apply it to
+your programs, too.
+
+ When we speak of free software, we are referring to freedom, not
+price. Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+ To protect your rights, we need to prevent others from denying you
+these rights or asking you to surrender the rights. Therefore, you have
+certain responsibilities if you distribute copies of the software, or if
+you modify it: responsibilities to respect the freedom of others.
+
+ For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must pass on to the recipients the same
+freedoms that you received. You must make sure that they, too, receive
+or can get the source code. And you must show them these terms so they
+know their rights.
+
+ Developers that use the GNU GPL protect your rights with two steps:
+(1) assert copyright on the software, and (2) offer you this License
+giving you legal permission to copy, distribute and/or modify it.
+
+ For the developers' and authors' protection, the GPL clearly explains
+that there is no warranty for this free software. For both users' and
+authors' sake, the GPL requires that modified versions be marked as
+changed, so that their problems will not be attributed erroneously to
+authors of previous versions.
+
+ Some devices are designed to deny users access to install or run
+modified versions of the software inside them, although the manufacturer
+can do so. This is fundamentally incompatible with the aim of
+protecting users' freedom to change the software. The systematic
+pattern of such abuse occurs in the area of products for individuals to
+use, which is precisely where it is most unacceptable. Therefore, we
+have designed this version of the GPL to prohibit the practice for those
+products. If such problems arise substantially in other domains, we
+stand ready to extend this provision to those domains in future versions
+of the GPL, as needed to protect the freedom of users.
+
+ Finally, every program is threatened constantly by software patents.
+States should not allow patents to restrict development and use of
+software on general-purpose computers, but in those that do, we wish to
+avoid the special danger that patents applied to a free program could
+make it effectively proprietary. To prevent this, the GPL assures that
+patents cannot be used to render the program non-free.
+
+ The precise terms and conditions for copying, distribution and
+modification follow.
+
+ TERMS AND CONDITIONS
+
+ 0. Definitions.
+
+ "This License" refers to version 3 of the GNU General Public License.
+
+ "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+ "The Program" refers to any copyrightable work licensed under this
+License. Each licensee is addressed as "you". "Licensees" and
+"recipients" may be individuals or organizations.
+
+ To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy. The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+ A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+ To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy. Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+ To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies. Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+ An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License. If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+ 1. Source Code.
+
+ The "source code" for a work means the preferred form of the work
+for making modifications to it. "Object code" means any non-source
+form of a work.
+
+ A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+ The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form. A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+ The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities. However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work. For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+ The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+ The Corresponding Source for a work in source code form is that
+same work.
+
+ 2. Basic Permissions.
+
+ All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met. This License explicitly affirms your unlimited
+permission to run the unmodified Program. The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work. This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+ You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force. You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright. Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+ Conveying under any other circumstances is permitted solely under
+the conditions stated below. Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+ 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+ No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+ When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+ 4. Conveying Verbatim Copies.
+
+ You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+ You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+ 5. Conveying Modified Source Versions.
+
+ You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+ a) The work must carry prominent notices stating that you modified
+ it, and giving a relevant date.
+
+ b) The work must carry prominent notices stating that it is
+ released under this License and any conditions added under section
+ 7. This requirement modifies the requirement in section 4 to
+ "keep intact all notices".
+
+ c) You must license the entire work, as a whole, under this
+ License to anyone who comes into possession of a copy. This
+ License will therefore apply, along with any applicable section 7
+ additional terms, to the whole of the work, and all its parts,
+ regardless of how they are packaged. This License gives no
+ permission to license the work in any other way, but it does not
+ invalidate such permission if you have separately received it.
+
+ d) If the work has interactive user interfaces, each must display
+ Appropriate Legal Notices; however, if the Program has interactive
+ interfaces that do not display Appropriate Legal Notices, your
+ work need not make them do so.
+
+ A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit. Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+ 6. Conveying Non-Source Forms.
+
+ You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+ a) Convey the object code in, or embodied in, a physical product
+ (including a physical distribution medium), accompanied by the
+ Corresponding Source fixed on a durable physical medium
+ customarily used for software interchange.
+
+ b) Convey the object code in, or embodied in, a physical product
+ (including a physical distribution medium), accompanied by a
+ written offer, valid for at least three years and valid for as
+ long as you offer spare parts or customer support for that product
+ model, to give anyone who possesses the object code either (1) a
+ copy of the Corresponding Source for all the software in the
+ product that is covered by this License, on a durable physical
+ medium customarily used for software interchange, for a price no
+ more than your reasonable cost of physically performing this
+ conveying of source, or (2) access to copy the
+ Corresponding Source from a network server at no charge.
+
+ c) Convey individual copies of the object code with a copy of the
+ written offer to provide the Corresponding Source. This
+ alternative is allowed only occasionally and noncommercially, and
+ only if you received the object code with such an offer, in accord
+ with subsection 6b.
+
+ d) Convey the object code by offering access from a designated
+ place (gratis or for a charge), and offer equivalent access to the
+ Corresponding Source in the same way through the same place at no
+ further charge. You need not require recipients to copy the
+ Corresponding Source along with the object code. If the place to
+ copy the object code is a network server, the Corresponding Source
+ may be on a different server (operated by you or a third party)
+ that supports equivalent copying facilities, provided you maintain
+ clear directions next to the object code saying where to find the
+ Corresponding Source. Regardless of what server hosts the
+ Corresponding Source, you remain obligated to ensure that it is
+ available for as long as needed to satisfy these requirements.
+
+ e) Convey the object code using peer-to-peer transmission, provided
+ you inform other peers where the object code and Corresponding
+ Source of the work are being offered to the general public at no
+ charge under subsection 6d.
+
+ A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+ A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling. In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage. For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product. A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+ "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source. The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+ If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information. But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+ The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed. Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+ Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+ 7. Additional Terms.
+
+ "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law. If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+ When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it. (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.) You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+ Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+ a) Disclaiming warranty or limiting liability differently from the
+ terms of sections 15 and 16 of this License; or
+
+ b) Requiring preservation of specified reasonable legal notices or
+ author attributions in that material or in the Appropriate Legal
+ Notices displayed by works containing it; or
+
+ c) Prohibiting misrepresentation of the origin of that material, or
+ requiring that modified versions of such material be marked in
+ reasonable ways as different from the original version; or
+
+ d) Limiting the use for publicity purposes of names of licensors or
+ authors of the material; or
+
+ e) Declining to grant rights under trademark law for use of some
+ trade names, trademarks, or service marks; or
+
+ f) Requiring indemnification of licensors and authors of that
+ material by anyone who conveys the material (or modified versions of
+ it) with contractual assumptions of liability to the recipient, for
+ any liability that these contractual assumptions directly impose on
+ those licensors and authors.
+
+ All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10. If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term. If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+ If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+ Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+ 8. Termination.
+
+ You may not propagate or modify a covered work except as expressly
+provided under this License. Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+ However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+ Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+ Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License. If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+ 9. Acceptance Not Required for Having Copies.
+
+ You are not required to accept this License in order to receive or
+run a copy of the Program. Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance. However,
+nothing other than this License grants you permission to propagate or
+modify any covered work. These actions infringe copyright if you do
+not accept this License. Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+ 10. Automatic Licensing of Downstream Recipients.
+
+ Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License. You are not responsible
+for enforcing compliance by third parties with this License.
+
+ An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations. If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+ You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License. For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+ 11. Patents.
+
+ A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based. The
+work thus licensed is called the contributor's "contributor version".
+
+ A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version. For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+ Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+ In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement). To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+ If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients. "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+ If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+ A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License. You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+ Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+ 12. No Surrender of Others' Freedom.
+
+ If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License. If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all. For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+ 13. Use with the GNU Affero General Public License.
+
+ Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU Affero General Public License into a single
+combined work, and to convey the resulting work. The terms of this
+License will continue to apply to the part which is the covered work,
+but the special requirements of the GNU Affero General Public License,
+section 13, concerning interaction through a network will apply to the
+combination as such.
+
+ 14. Revised Versions of this License.
+
+ The Free Software Foundation may publish revised and/or new versions of
+the GNU General Public License from time to time. Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+ Each version is given a distinguishing version number. If the
+Program specifies that a certain numbered version of the GNU General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation. If the Program does not specify a version number of the
+GNU General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+ If the Program specifies that a proxy can decide which future
+versions of the GNU General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+ Later license versions may give you additional or different
+permissions. However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+ 15. Disclaimer of Warranty.
+
+ THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+ 16. Limitation of Liability.
+
+ IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+ 17. Interpretation of Sections 15 and 16.
+
+ If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+ END OF TERMS AND CONDITIONS
+
+ How to Apply These Terms to Your New Programs
+
+ If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+ To do so, attach the following notices to the program. It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+
+ Copyright (C)
+
+ This program is free software: you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published by
+ the Free Software Foundation, either version 3 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program. If not, see .
+
+Also add information on how to contact you by electronic and paper mail.
+
+ If the program does terminal interaction, make it output a short
+notice like this when it starts in an interactive mode:
+
+ Copyright (C)
+ This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+ This is free software, and you are welcome to redistribute it
+ under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License. Of course, your program's commands
+might be different; for a GUI interface, you would use an "about box".
+
+ You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU GPL, see
+.
+
+ The GNU General Public License does not permit incorporating your program
+into proprietary programs. If your program is a subroutine library, you
+may consider it more useful to permit linking proprietary applications with
+the library. If this is what you want to do, use the GNU Lesser General
+Public License instead of this License. But first, please read
+.
diff --git a/Changelog b/Changelog
index dc5d36969bcdb9450a4935776860a3721c807afc..2e7bd85d29547178197ddb7fe06e66bec9f03654 100644
--- a/Changelog
+++ b/Changelog
@@ -6,6 +6,257 @@ Unified Changelog file for Synnefo versions >= 0.13
Since v0.13 most of the Synnefo components have been merged into a single
repository and have aligned versions.
+.. _Changelog-0.16:
+
+v0.16
+=====
+
+Released: Tue Nov 11 10:44:57 EET 2014
+
+Synnefo-wide
+------------
+
+* Replace accumulative projects with pool projects:
+
+ * Projects are now viewed as a source of finite resources. A member can
+ reserve a part of these resources up to a specified limit.
+
+ * Base quota are now offered through a special purpose user-specific system
+ project, identified with the same UUID as the user.
+
+ * Each actual resource (Cyclades VM, network, floating IP and Pithos
+ container) is now also associated with a project besides the owner.
+
+ * In resource creation, project defaults to the user-specific system
+ project, if not specified otherwise. It is also possible to change the
+ project assignment of an existing resource.
+
+ * All existing resources have been assigned to the respective
+ user-specific system projects.
+
+* Logging mechanism for Synnefo management commands
+
+ * Log all stdout and stderr output of every invocation of snf-manage,
+ on unique filenames under a given directory.
+
+ * Add a new setting in the snf-common package, namely 'LOG_DIR', which
+ specifies the directory to be used by Synnefo components to write
+ their log files.
+
+* Rename argument names of Synnefo management commands
+
+ * All Synnefo management commands now use 'user/flavor/image/network'
+ as argument names, instead of 'userid/user-id/owner/flavor-id/image
+ -id/network-id', for consistency across commands.
+
+Astakos
+-------
+
+* Decouple projects from applications:
+
+ * Support project creation (by the system) and modification (by a
+ privileged user) without the need to submit/approve an application.
+
+ * View applications as modifications. When a project is uninitialized
+ (e.g. an application for a new project is pending), no further
+ modification is allowed.
+
+ * Applications are removed from the API. A project's last application is
+ only accessible as part of the project details.
+
+ * Decouple project state from application state; they can be combined by
+ an API client, if needed.
+
+* Changes concerning quota and pool projects:
+
+ * A project must provide limits for all registered resources. On project
+ activation, resources missing are automatically completed using a
+ skeleton.
+
+ * Field `uplimit' of registed resources is exposed as `system_default' and
+ provide the skeleton for user-specific system projects. A new field
+ `project_default' is introduce to act as a skeleton for conventional
+ projects.
+
+ * The quotaholder now also records project quota besides user quota. The
+ two types of holders are distinguished with a prefix: `user:' and
+ `project:'.
+
+ * The quota API is extended to make project quota available.
+
+ * Removed setting `ASTAKOS_PROJECTS_VISIBLE'; we now always display
+ projects in Astakos menu.
+
+* Projects can be set `private', making it accessible only to its owner and
+ members.
+
+* Admin users API
+
+ * Extend astakos identity API to support user management functionality.
+ * Admin API related settings
+ * ``ADMIN_API_ENABLED``
+ Whether or not the admin api endpoints will be enabled.
+ * ``ADMIN_API_PERMITTED_GROUPS``
+ A list of group names for which users which belong to any of
+ them be granted full access to the admin API endpoints.
+
+
+* Updated projects integration in UI
+
+ * Display both max per member/total in group quota in project views.
+ * Enhanced usage view to display individual resource usage details for
+ each of the projects the user is associated with.
+ * Display project usage statistics in project details view.
+ * Display system projects in admin project list view.
+ * Handle display of infinite resource quota values.
+
+* Explicitly specify the lists in which every notification will be sent.
+
+
+Cyclades
+--------
+
+* Introduce Volumes and Snapshots:
+
+ * Implement 'cyclades_volumes' service, containing the /volumes, /types,
+ and /snapshots API endpoints under '/volumes/v2.0'.
+ * Implement `snf-manage volume-{create, list, show, modify, inspect, remove}'
+ management commands for handling of volumes.
+ * Implement `snf-manage snapshot-{create, list, show, modify}` management
+ commands for handling of snapshots.
+ * Implement 'volume-type-{create, list, modify, show, remove}` management
+ commands for handling of volume types.
+ * Implement reconciliation for stale snapshots.
+ * Add new settings:
+
+ * GANETI_MAX_DISKS_PER_INSTANCE: Maximum number of disks that each Ganeti
+ instance can have.
+ * CYCLADES_VOLUME_MAX_SIZE: The maximum allowed size (in GB) for Cyclades
+ volumes.
+ * CYCLADES_SNAPSHOTS_ENABLED: Enable/Disable the snapshots feature
+ altogether at the API level.
+
+
+* Integrate Cyclades resources with Astakos projects.
+
+ * Extend API calls that create resources with the 'project' attribute
+ in order to assign resources to the specified project.
+ * Implement API calls for reassigning resources to a new project.
+ * Export the project that a resource belongs in the 'tenant_id' API
+ attribute.
+
+* Add support for snf-vncauthproxy 1.6 (configurable VNC console types).
+* Change the ``CYCLADES_VNCAUTHPROXY_OPTS`` setting to a list of dictionaries
+ and support configurable vncauthproxy proxy addresses (added in
+ snf-vncauthproxy-1.6).
+* Add support for more types of VM console:
+
+ * Modify the current 'console' action to support VNC WebSockets (requires
+ snf-vncauthproxy=1.6).
+ * Add support for the three OpenStack Compute console actions
+ (VNC, RDP, Spice). RDP and Spice are currently not implemented.
+
+* Update Cyclades to work with Pithos with integrated Archipelago v2
+ mapfiles.
+* Include functionality for checking the status of the Cyclades update path,
+ which includes Ganeti, AMQP, snf-ganeti-eventd and snf-dispatcher. This
+ functionality is implemented as part of the snf-dispatcher and can be used
+ by passing the '--status-check' option.
+* Add setting `CYCLADES_VM_MAX_METADATA` to limit the maximum number of
+ metadata per vm.
+* Add setting `CYCLADES_VOLUME_MAX_METADATA` to limit the maximum number of
+ metadata per volume.
+* Use common -u/--user option to specify the UUID or email of the user, for
+ all management commands.
+* Store basic information about images that have been used to create servers,
+ in order to preserve this information even if images are deleted from
+ Pithos.
+* Make 'snf-ganeti-eventd' tolerate failures when processing Ganeti jobs. The
+ daemon will not crash but continue to run in order to process jobs that can
+ be processed.
+* Update 'backend-list' command to not count the free IPs from networks that
+ are drained.
+* Fix the'network-inspect' command to not contain externally reserved IPs
+ in th number of available IPs.
+* Add `GANETI_DISKS_WAIT_FOR_SYNC` setting to decide whether Ganeti will
+ wait for the disk mirror to sync (`--no-wait-for-sync` Ganeti option).
+* Fix mishandling of `MAX_CIDR_BLOCK` setting. Allowed CIDR sizes
+ changed from (MAX_CIDR_BLOCK, 29) to [MAX_CIDR_BLOCK, 29].
+* Fix various minor bugs.
+* Remove stale '--public' and '--user' options from 'image-show' and
+ 'snapshot-show' management commands.
+
+Cyclades UI
+-----------
+
+* In sync with the updated astakos projects API
+
+ * Include a project select widget within all resource create views, to let user
+ decide the project that the created resources will be assigned to.
+ * Display resource assigned project in resource list views.
+ * Let user reassign resource project.
+
+* Volumes API integration.
+
+ * Introduce the Disks list/create views.
+ * Display machine attached disks in icon/single machine views.
+ * Use ``CYCLADES_VOLUME_MAX_SIZE`` setting to determine the
+ maximum allowed disk size.
+
+* Integrate volume snapshots
+
+ * Include available snapshots in disk/vm create wizards.
+ * Disk snaphot create actions.
+ * The ui snapshoting functionality (snapshot actions and listing)
+ can be enabled/disabled using the introduced ``UI_SNAPSHOTS_ENABLED``
+ setting.
+
+* Replace TigerVNC Java client with an HTML5 Websocket-based client (noVNC)
+
+* Other UI fixes
+
+ * Disabled suspended vm actions
+ * Custom error message for 413 api error response codes
+ * Enable resize actions from both info/actions subviews for active
+ machines
+ * Update images collection each time machine create wizard opens
+ * Fixed a couple of title truncate in several views
+ * Handle display of infinite resource quota values.
+ * Improve network create wizard. Support for custom gateway.
+ * Common font styling for all resource titles
+
+Pithos
+------
+
+* Backend modifications to enable/support snapshot creation.
+* Backend and API modifications to reflect the modifications in the resource
+ allocation mechanism via the accumulative projects with pool projects.
+* Change default disposition type: If no disposition-type is
+ specifically requested or an invalid value is passed, the disposition type
+ is set to 'inline'.
+
+Admin
+-----
+
+* Introduce Administrator Dashboard, which provides the following:
+
+ * Graphic access to the details of various Synnefo entities (users, VMs,
+ Projects).
+ * Intuitive filtering.
+ * Multiple actions and notifications.
+ * Charts and statistics generation.
+
+* Define the Admin node URL and path with the ``ADMIN_BASE_URL`` setting
+
+ * The ``ADMIN_BASE_URL`` setting should be adjusted in every node that Admin
+ is installed.
+
+Tools
+-----
+
+* Extend snf-burnin to include testing of snapshots.
+
+
.. _Changelog-0.15.2:
v0.15.2
@@ -29,6 +280,7 @@ Cyclades UI
.. _Changelog-0.15.1:
+
v0.15.1
=======
@@ -49,7 +301,6 @@ Astakos
* Fix Authenticate Identity API call with trailing slash, which used to fail
with 405 (Method not allowed)
-
Cyclades
--------
diff --git a/NEWS b/NEWS
index a3a803ef2824650b905145e95a455f16b8f5183e..a11d3c45754eb36ff13ab095b7fa37ed6590d929 100644
--- a/NEWS
+++ b/NEWS
@@ -5,6 +5,88 @@ Unified NEWS file for Synnefo versions >= 0.13
Since v0.13 all Synnefo components have been merged into a single repository.
+.. _NEWS-0.16:
+
+v0.16
+=====
+
+Released: Tue Nov 11 10:44:57 EET 2014
+
+The Synnefo 0.16 release brings major new features and significant bug fixes
+across Synnefo. All users are strongly encouraged to upgrade. Please see
+the Synnefo documentation for upgrade instructions to v0.16.
+
+The most notables changes are:
+
+Synnefo-wide
+-------------
+
+* Administrator Dashboard: Synnefo 0.16 features a new, integrated Web-based UI
+ for managing Synnefo users and resources. It allows the administrator to
+ view, filter, modify, and produce stats for all Synnefo resources. This new
+ interface will eventually replace the old read-only helpdesk interface, which
+ has been obsoleted and will be removed in Synnefo 0.17.
+* Pool projects: Projects have been upgraded to a new unified `pool project`
+ type in Synnefo 0.16. Resources get assigned to projects, and project members
+ may reserve resources from individual projects for their own use. Users may
+ choose the project where a new resource be charged (e.g., a new Cyclades VM or
+ a new IP address), and re-assign them freely. Pre-0.16 projects are a subset
+ of pool projects, with per-user constraints on resource consumption.
+* Management command logging: All Synnefo management commands and their outputs
+ are individually logged, which may prove very useful for debugging and
+ auditing purposes.
+* Archipelago has been adopted as the single backend for Pithos. It is a
+ unified access layer supporting NFS or Ceph/RADOS-based storage. Current
+ Pithos installations over NFS can be migrated seamlessly to Pithos over
+ Archipelago over NFS, please see the Synnefo upgrade notes (FIXME)
+ and the Archipelago Administrator's Guide for more details.
+ Cyclades VMs continue to run over all Ganeti-supported disk templates,
+ including DRBD, LVM, Ceph/RBD, and Archipelago.
+* Various bug fixes and performance improvements. Please see the Synnefo
+ `Changelog` for a complete list.
+
+
+Cyclades
+--------
+
+* Storage: Major improvements in VM storage handling.
+ This release introduces a complete implementation of the OpenStack Block Storage
+ (Cinder) API v2.0, with distinct /volumes, /snapshots/, /types API endpoints.
+* Storage: You can now add and remove virtual disks to and from VMs, even when
+ they are powered on, with hot-(un)plugging.
+* Storage: You can now snapshot running VMs to files on Pithos, provided
+ their storage is backed by Archipelago. New snapshots appear in Pithos and
+ are manage-able as Pithos files.
+* Storage: You can now spawn new VMs from pre-existing snapshots, regardless
+ of their disk template, e.g., you can use an existing snapshot to spawn a
+ DRBD VM. If the new VM is to be based on Archipelago, its creation is a thin
+ clone and completes in seconds.
+ PLEASE NOTE: Snapshot support is a work-in-progress, and Archipelago does not
+ yet perform garbage collection on snapshot deletion. This will be fixed in
+ Synnefo 0.17.
+* Console: Support noVNC-based console for an HTML5-based UI to server consoles
+ over VNC, with HTTPS encryption. This removes the need for a working Java Runtime
+ Environment in the client's browser.
+* Console: Support multiple console types, including VNC over raw TCP sockets
+ and VNC over Websockets, with optional encryption.
+* Projects: Supports arbitrary (re-)assignment of Cyclades resources to
+ individual projects.
+* Admin: Support end-to-end checking of the Ganeti-to-Synnefo update path,
+ including all intermediate daemons and the Message Queue.
+* Numerous bug fixes, performance improvements and improved usability in
+ management commands.
+
+
+Pithos
+------
+
+* Pithos has been refactored to use Archipelago as its single unified storage
+ backend. Individual storage types (e.g., NFS, or RADOS) are handled by
+ Archipelago itself.
+* Snapshots: Snapshots created from Cyclades VMs on Archipelago are now
+ presented as files on Pithos and may be downloaded via the Pithos UI.
+
+
.. _NEWS-0.15.2:
v0.15.2
@@ -145,7 +227,7 @@ Synnefo-wide
.. _NEWS-0.14.10:
v0.14.10
-=======
+========
Released: Tue Nov 26 11:03:37 EET 2013
@@ -247,7 +329,7 @@ Synnefo-wide
* Branding customization support across synnefo frontend components:
- * ability to adapt the Astakos, Pithos and Cyclades Web UI to a company’s
+ * ability to adapt the Astakos, Pithos and Cyclades Web UI to a company's
visual identity. This is possible using the snf-branding component, which
is automatically installed on the nodes running the API servers for
Astakos, Pithos and Cyclades.
diff --git a/README.md b/README.md
index 200fc99bd12d3d98e00f1ff90cf3ca7d293945b4..8687d628a5f174b15f0340ceae4cbd698643b8ad 100644
--- a/README.md
+++ b/README.md
@@ -47,36 +47,18 @@ for more information on the Synnefo users and developers lists.
Copyright and license
=====================
-Copyright (C) 2010-2014 GRNET S.A. All rights reserved.
-
-Redistribution and use in source and binary forms, with or
-without modification, are permitted provided that the following
-conditions are met:
-
- 1. Redistributions of source code must retain the above
- copyright notice, this list of conditions and the following
- disclaimer.
-
- 2. Redistributions in binary form must reproduce the above
- copyright notice, this list of conditions and the following
- disclaimer in the documentation and/or other materials
- provided with the distribution.
-
-THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A. OR
-CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-POSSIBILITY OF SUCH DAMAGE.
-
-The views and conclusions contained in the software and
-documentation are those of the authors and should not be
-interpreted as representing official policies, either expressed
-or implied, of GRNET S.A.
+Copyright (C) 2010-2014 GRNET S.A.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see .
diff --git a/astakosclient/MANIFEST.in b/astakosclient/MANIFEST.in
index 6106c5d89680afa76e234fbcf1feb50c52b9a5f6..ce7f614828c2d727bb93e8692c231749d45fdd5a 100644
--- a/astakosclient/MANIFEST.in
+++ b/astakosclient/MANIFEST.in
@@ -1 +1 @@
-include distribute_setup.py
+include distribute_setup.py README.md
diff --git a/astakosclient/README.md b/astakosclient/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..e08280b238b80f09ecfb032dd7cb541b45137c18
--- /dev/null
+++ b/astakosclient/README.md
@@ -0,0 +1,27 @@
+astakosclient
+=============
+
+Overview
+--------
+
+This is Synnefo's astakosclient component. Please see the [official Synnefo
+site](http://www.synnefo.org) for more information.
+
+
+Copyright and license
+=====================
+
+Copyright (C) 2010-2014 GRNET S.A.
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see .
diff --git a/astakosclient/astakosclient/__init__.py b/astakosclient/astakosclient/__init__.py
index 759e91afdeb218263b978d8cfd79ec41153aa062..161be1103d56561fa7b0e29baba11430345e3d57 100644
--- a/astakosclient/astakosclient/__init__.py
+++ b/astakosclient/astakosclient/__init__.py
@@ -1,35 +1,17 @@
-# Copyright (C) 2012, 2013 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
"""
Simple and minimal client for the Astakos authentication service
@@ -42,12 +24,18 @@ import hashlib
from base64 import b64encode
from copy import copy
-import simplejson
+try:
+ import simplejson as json
+except ImportError:
+ import json
+
from astakosclient.utils import \
- retry_dec, scheme_to_class, parse_request, check_input, join_urls
+ retry_dec, scheme_to_class, parse_request, check_input, join_urls, \
+ render_overlimit_exception
from astakosclient.errors import \
AstakosClientException, Unauthorized, BadRequest, NotFound, Forbidden, \
- NoUserName, NoUUID, BadValue, QuotaLimit, InvalidResponse, NoEndpoints
+ NoUserName, NoUUID, BadValue, QuotaLimit, InvalidResponse, NoEndpoints, \
+ ConnectionError
# --------------------------------------------------------------------
@@ -213,6 +201,10 @@ class AstakosClient(object):
def api_service_quotas(self):
return join_urls(self.account_prefix, "service_quotas")
+ @property
+ def api_service_project_quotas(self):
+ return join_urls(self.account_prefix, "service_project_quotas")
+
@property
def api_commissions(self):
return join_urls(self.account_prefix, "commissions")
@@ -229,10 +221,6 @@ class AstakosClient(object):
def api_projects(self):
return join_urls(self.account_prefix, "projects")
- @property
- def api_applications(self):
- return join_urls(self.api_projects, "apps")
-
@property
def api_memberships(self):
return join_urls(self.api_projects, "memberships")
@@ -258,7 +246,7 @@ class AstakosClient(object):
hashed_token.update(self.token)
self.logger.debug(
"Make a %s request to %s, using token with hash %s, "
- "with headers %s and body %s",
+ "with headers %r and body %r",
method, request_path, hashed_token.hexdigest(), headers,
body if log_body else "(not logged)")
@@ -300,11 +288,11 @@ class AstakosClient(object):
self.log_response = dict(
status=status, message=message, data=data)
except Exception as err:
- self.logger.error("Failed to send request: %s" % repr(err))
- raise AstakosClientException(str(err))
+ self.logger.error("Failed to send request: %r", err)
+ raise ConnectionError(err)
# Return
- self.logger.debug("Request returned with status %s" % status)
+ self.logger.debug("Request returned with status %s", status)
if status == 400:
raise BadRequest(message, data)
elif status == 401:
@@ -314,17 +302,18 @@ class AstakosClient(object):
elif status == 404:
raise NotFound(message, data)
elif status < 200 or status >= 300:
- raise AstakosClientException(message, data, status)
+ raise AstakosClientException(
+ message=message, status=status, response=data)
try:
if data:
- return simplejson.loads(unicode(data))
+ return json.loads(unicode(data))
else:
return None
except Exception as err:
- msg = "Cannot parse response \"%s\" with simplejson: %s"
+ msg = "Cannot parse response \"%r\" with simplejson: %s"
self.logger.error(msg % (data, str(err)))
- raise InvalidResponse(str(err), data)
+ raise InvalidResponse(message=str(err), response=data)
# ----------------------------------
# do a POST to ``API_USERCATALOGS`` (or ``API_SERVICE_USERCATALOGS``)
@@ -338,10 +327,10 @@ class AstakosClient(object):
if "uuid_catalog" in data:
return data.get("uuid_catalog")
else:
- msg = "_uuid_catalog request returned %s. No uuid_catalog found" \
+ msg = "_uuid_catalog request returned %r. No uuid_catalog found" \
% data
self.logger.error(msg)
- raise AstakosClientException(msg)
+ raise AstakosClientException(message=msg, response=data)
def get_usernames(self, uuids):
"""Return a uuid_catalog dictionary for the given uuids
@@ -389,10 +378,10 @@ class AstakosClient(object):
if "displayname_catalog" in data:
return data.get("displayname_catalog")
else:
- msg = "_displayname_catalog request returned %s. " \
+ msg = "_displayname_catalog request returned %r. " \
"No displayname_catalog found" % data
self.logger.error(msg)
- raise AstakosClientException(msg)
+ raise AstakosClientException(message=msg, response=data)
def get_uuids(self, display_names):
"""Return a displayname_catalog for the given names
@@ -540,27 +529,62 @@ class AstakosClient(object):
"""
return self._call_astakos(self.api_quotas)
+ def _join_if_list(self, val):
+ return ','.join(map(str, val)) if isinstance(val, list) else val
+
# ----------------------------------
# do a GET to ``API_SERVICE_QUOTAS``
- def service_get_quotas(self, user=None):
+ def service_get_quotas(self, user=None, project_id=None, project=None):
"""Get all quotas for resources associated with the service
Keyword arguments:
- user -- optionally, the uuid of a specific user
+ user -- optionally, the uuid of a specific user, or a list thereof
+ project_id -- optionally, the uuid of a specific project, or a list
+ thereof
+ project -- backwards compatibility (replaced by "project_id")
In case of success return a dict of dicts of dicts with current quotas
for all users, or of a specified user, if user argument is set.
Otherwise raise an AstakosClientException
"""
+ project_id = project if project_id is None else project_id
query = self.api_service_quotas
+ filters = {}
if user is not None:
- query += "?user=" + user
+ filters['user'] = self._join_if_list(user)
+ if project_id is not None:
+ filters['project'] = self._join_if_list(project_id)
+ if filters:
+ query += "?" + urllib.urlencode(filters)
+ return self._call_astakos(query)
+
+ # ----------------------------------
+ # do a GET to ``API_SERVICE_PROJECT_QUOTAS``
+ def service_get_project_quotas(self, project_id=None, project=None):
+ """Get all project quotas for resources associated with the service
+
+ Keyword arguments:
+ project -- optionally, the uuid of a specific project, or a list
+ thereof
+
+ In case of success return a dict of dicts with current quotas
+ for all projects, or of a specified project, if project argument is
+ set. Otherwise raise an AstakosClientException
+
+ """
+ project_id = project if project_id is None else project_id
+ query = self.api_service_project_quotas
+ filters = {}
+ if project_id is not None:
+ filters['project'] = self._join_if_list(project_id)
+ if filters:
+ query += "?" + urllib.urlencode(filters)
return self._call_astakos(query)
# ----------------------------------
# do a POST to ``API_COMMISSIONS``
- def issue_commission(self, request):
+ def _issue_commission(self, request):
"""Issue a commission
Keyword arguments:
@@ -579,54 +603,135 @@ class AstakosClient(object):
method="POST")
except AstakosClientException as err:
if err.status == 413:
- raise QuotaLimit(err.message, err.details)
+ try:
+ msg, details = render_overlimit_exception(
+ err.response, self.logger)
+ except Exception as perr:
+ self.logger.error("issue_commission request returned '413'"
+ " but response '%r' could not be parsed:"
+ " %s", err.response, str(perr))
+ msg, details = err.message, ""
+ raise QuotaLimit(message=msg,
+ details=details,
+ response=err.response)
else:
raise
if "serial" in response:
return response['serial']
else:
- msg = "issue_commission_core request returned %s. " + \
+ msg = "issue_commission_core request returned %r. " + \
"No serial found" % response
self.logger.error(msg)
- raise AstakosClientException(msg)
+ raise AstakosClientException(message=msg, response=response)
+
+ def _mk_user_provision(self, holder, source, resource, quantity):
+ holder = "user:" + holder
+ source = "project:" + source
+ return {"holder": holder, "source": source,
+ "resource": resource, "quantity": quantity}
+
+ def _mk_project_provision(self, holder, resource, quantity):
+ holder = "project:" + holder
+ return {"holder": holder, "source": None,
+ "resource": resource, "quantity": quantity}
- def issue_one_commission(self, holder, source, provisions,
+ def mk_provisions(self, holder, source, resource, quantity):
+ return [self._mk_user_provision(holder, source, resource, quantity),
+ self._mk_project_provision(source, resource, quantity)]
+
+ def issue_commission_generic(self, user_provisions, project_provisions,
+ name="", force=False, auto_accept=False):
+ """Issue commission (for multiple holder/source pairs)
+
+ keyword arguments:
+ user_provisions -- dict mapping user holdings
+ (user, project, resource) to integer quantities
+ project_provisions -- dict mapping project holdings
+ (project, resource) to integer quantities
+ name -- description of the commission (string)
+ force -- force this commission (boolean)
+ auto_accept -- auto accept this commission (boolean)
+
+ In case of success return commission's id (int).
+ Otherwise raise an AstakosClientException.
+
+ """
+ request = {}
+ request["force"] = force
+ request["auto_accept"] = auto_accept
+ request["name"] = name
+ try:
+ request["provisions"] = []
+ for (holder, source, resource), quantity in \
+ user_provisions.iteritems():
+ p = self._mk_user_provision(holder, source, resource, quantity)
+ request["provisions"].append(p)
+ for (holder, resource), quantity in project_provisions.iteritems():
+ p = self._mk_project_provision(holder, resource, quantity)
+ request["provisions"].append(p)
+ except Exception as err:
+ self.logger.error(str(err))
+ raise BadValue(str(err))
+
+ return self._issue_commission(request)
+
+ def issue_one_commission(self, holder, provisions,
name="", force=False, auto_accept=False):
"""Issue one commission (with specific holder and source)
keyword arguments:
holder -- user's id (string)
- source -- commission's source (ex system) (string)
- provisions -- resources with their quantity (dict from string to int)
+ provisions -- (source, resource) mapping to quantity
name -- description of the commission (string)
force -- force this commission (boolean)
auto_accept -- auto accept this commission (boolean)
In case of success return commission's id (int).
Otherwise raise an AstakosClientException.
- (See also issue_commission)
"""
check_input("issue_one_commission", self.logger,
- holder=holder, source=source,
- provisions=provisions)
+ holder=holder, provisions=provisions)
+
+ request = {}
+ request["force"] = force
+ request["auto_accept"] = auto_accept
+ request["name"] = name
+ try:
+ request["provisions"] = []
+ for (source, resource), quantity in provisions.iteritems():
+ ps = self.mk_provisions(holder, source, resource, quantity)
+ request["provisions"].extend(ps)
+ except Exception as err:
+ self.logger.error(str(err))
+ raise BadValue(str(err))
+
+ return self._issue_commission(request)
+
+ def issue_resource_reassignment(self, holder, provisions, name="",
+ force=False, auto_accept=False):
+ """Change resource assignment to another project
+ """
request = {}
request["force"] = force
request["auto_accept"] = auto_accept
request["name"] = name
+
try:
request["provisions"] = []
- for resource, quantity in provisions.iteritems():
- prov = {"holder": holder, "source": source,
- "resource": resource, "quantity": quantity}
- request["provisions"].append(prov)
+ for key, quantity in provisions.iteritems():
+ (from_source, to_source, resource) = key
+ ps = self.mk_provisions(
+ holder, from_source, resource, -quantity)
+ ps += self.mk_provisions(holder, to_source, resource, quantity)
+ request["provisions"].extend(ps)
except Exception as err:
self.logger.error(str(err))
raise BadValue(str(err))
- return self.issue_commission(request)
+ return self._issue_commission(request)
# ----------------------------------
# do a GET to ``API_COMMISSIONS``
@@ -713,13 +818,15 @@ class AstakosClient(object):
# ----------------------------
# do a GET to ``API_PROJECTS``
- def get_projects(self, name=None, state=None, owner=None):
+ def get_projects(self, name=None, state=None, owner=None, mode=None):
"""Retrieve all accessible projects
Arguments:
name -- filter by name (optional)
state -- filter by state (optional)
owner -- filter by owner (optional)
+ mode -- if value is 'member', return only active projects in which
+ the request user is an active member
In case of success, return a list of project descriptions.
"""
@@ -730,11 +837,13 @@ class AstakosClient(object):
filters["state"] = state
if owner is not None:
filters["owner"] = owner
+ if mode is not None:
+ filters["mode"] = mode
+ path = self.api_projects
+ if filters:
+ path += "?" + urllib.urlencode(filters)
req_headers = {'content-type': 'application/json'}
- req_body = (parse_request({"filter": filters}, self.logger)
- if filters else None)
- return self._call_astakos(self.api_projects,
- headers=req_headers, body=req_body)
+ return self._call_astakos(path, headers=req_headers)
# -----------------------------------------
# do a GET to ``API_PROJECTS``/
@@ -766,7 +875,7 @@ class AstakosClient(object):
method="POST")
# ------------------------------------------
- # do a POST to ``API_PROJECTS``/
+ # do a PUT to ``API_PROJECTS``/
def modify_project(self, project_id, specs):
"""Submit application to modify an existing project
@@ -780,7 +889,7 @@ class AstakosClient(object):
req_headers = {'content-type': 'application/json'}
req_body = parse_request(specs, self.logger)
return self._call_astakos(path, headers=req_headers,
- body=req_body, method="POST")
+ body=req_body, method="PUT")
# -------------------------------------------------
# do a POST to ``API_PROJECTS``//action
@@ -798,74 +907,53 @@ class AstakosClient(object):
path = join_urls(self.api_projects, str(project_id))
path = join_urls(path, "action")
req_headers = {'content-type': 'application/json'}
- req_body = parse_request({action: reason}, self.logger)
+ req_body = parse_request({action: {"reason": reason}}, self.logger)
return self._call_astakos(path, headers=req_headers,
body=req_body, method="POST")
- # --------------------------------
- # do a GET to ``API_APPLICATIONS``
- def get_applications(self, project=None):
- """Retrieve all accessible applications
-
- Arguments:
- project -- filter by project (optional)
-
- In case of success, return a list of application descriptions.
- """
- req_headers = {'content-type': 'application/json'}
- body = {"project": project} if project is not None else None
- req_body = parse_request(body, self.logger) if body else None
- return self._call_astakos(self.api_applications,
- headers=req_headers, body=req_body)
-
- # -----------------------------------------
- # do a GET to ``API_APPLICATIONS``/
- def get_application(self, app_id):
- """Retrieve application description, if accessible
-
- Arguments:
- app_id -- application identifier
-
- In case of success, return application description.
- """
- path = join_urls(self.api_applications, str(app_id))
- return self._call_astakos(path)
-
# -------------------------------------------------
- # do a POST to ``API_APPLICATIONS``//action
- def application_action(self, app_id, action, reason=""):
- """Perform action on an application
+ # do a POST to ``API_PROJECTS``//action
+ def application_action(self, project_id, app_id, action, reason=""):
+ """Perform action on a project application
Arguments:
- app_id -- application identifier
- action -- action to perform, one of "approve", "deny",
- "dismiss", "cancel"
- reason -- reason of performing the action
+ project_id -- project identifier
+ app_id -- application identifier
+ action -- action to perform, one of "approve", "deny",
+ "dismiss", "cancel"
+ reason -- reason of performing the action
In case of success, return nothing.
"""
- path = join_urls(self.api_applications, str(app_id))
+ path = join_urls(self.api_projects, str(project_id))
path = join_urls(path, "action")
req_headers = {'content-type': 'application/json'}
- req_body = parse_request({action: reason}, self.logger)
+ req_body = parse_request({action: {
+ "reasons": reason,
+ "app_id": app_id}}, self.logger)
return self._call_astakos(path, headers=req_headers,
body=req_body, method="POST")
# -------------------------------
# do a GET to ``API_MEMBERSHIPS``
- def get_memberships(self, project=None):
+ def get_memberships(self, project_id=None, project=None):
"""Retrieve all accessible memberships
Arguments:
- project -- filter by project (optional)
+ project_id -- filter by project (optional)
+ project -- backwards compatibility
In case of success, return a list of membership descriptions.
"""
+ project_id = project if project_id is None else project_id
req_headers = {'content-type': 'application/json'}
- body = {"project": project} if project is not None else None
- req_body = parse_request(body, self.logger) if body else None
- return self._call_astakos(self.api_memberships,
- headers=req_headers, body=req_body)
+ filters = {}
+ if project_id is not None:
+ filters["project"] = project_id
+ path = self.api_memberships
+ if filters:
+ path += '?' + urllib.urlencode(filters)
+ return self._call_astakos(path, headers=req_headers)
# -----------------------------------------
# do a GET to ``API_MEMBERSHIPS``/
diff --git a/astakosclient/astakosclient/errors.py b/astakosclient/astakosclient/errors.py
index a85d38da02da784637ea4316565f10a1e45fe83f..fdfd9fc8b41d3355b5e3a4ccc7f355f0a61d07b3 100644
--- a/astakosclient/astakosclient/errors.py
+++ b/astakosclient/astakosclient/errors.py
@@ -1,35 +1,17 @@
-# Copyright (C) 2012, 2013 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
"""
Astakos Client Exceptions
@@ -38,15 +20,25 @@ Astakos Client Exceptions
class AstakosClientException(Exception):
"""Base AstakosClientException Class"""
- def __init__(self, message='', details='', status=500):
+ def __init__(self, message='', details='', status=500,
+ response=None, errobject=None):
self.message = message
self.details = details
+ self.errobject = errobject
+ self.response = response
if not hasattr(self, 'status'):
self.status = status
super(AstakosClientException,
self).__init__(self.message, self.details, self.status)
+class ConnectionError(AstakosClientException):
+ """Connection failed"""
+ def __init__(self, errobject):
+ super(ConnectionError, self).__init__(
+ message=str(errobject), errobject=errobject)
+
+
class BadValue(AstakosClientException):
"""Re-define ValueError Exception under AstakosClientException"""
def __init__(self, details):
@@ -56,8 +48,6 @@ class BadValue(AstakosClientException):
class InvalidResponse(AstakosClientException):
"""Return simplejson parse Exception as AstakosClient one"""
- def __init__(self, message, details):
- super(InvalidResponse, self).__init__(message, details)
class BadRequest(AstakosClientException):
diff --git a/astakosclient/astakosclient/tests.py b/astakosclient/astakosclient/tests.py
index c67b6e1df7480ca29243aadb166b0bae46231f17..d327cb098ada0fa5d40f35fb72884f7a2a0dc864 100644
--- a/astakosclient/astakosclient/tests.py
+++ b/astakosclient/astakosclient/tests.py
@@ -1,37 +1,19 @@
#!/usr/bin/env python
#
-# Copyright (C) 2012, 2013 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
"""Unit Tests for the astakos-client module
@@ -42,7 +24,11 @@ the astakos client library
import re
import sys
-import simplejson
+
+try:
+ import simplejson as json
+except ImportError:
+ import json
import astakosclient
from astakosclient import AstakosClient
@@ -289,10 +275,10 @@ def _req_tokens(conn, method, url, **kwargs):
if 'body' in kwargs:
# Return endpoints with authenticate info
- return ("", simplejson.dumps(endpoints_with_info), 200)
+ return ("", json.dumps(endpoints_with_info), 200)
else:
# Return endpoints without authenticate info
- return ("", simplejson.dumps(endpoints), 200)
+ return ("", json.dumps(endpoints), 200)
def _req_catalogs(conn, method, url, **kwargs):
@@ -309,7 +295,7 @@ def _req_catalogs(conn, method, url, **kwargs):
return _request_status_401(conn, method, url, **kwargs)
# Return
- body = simplejson.loads(kwargs['body'])
+ body = json.loads(kwargs['body'])
if 'uuids' in body:
# Return uuid_catalog
uuids = body['uuids']
@@ -326,7 +312,7 @@ def _req_catalogs(conn, method, url, **kwargs):
return_catalog = {"displayname_catalog": catalogs, "uuid_catalog": {}}
else:
return_catalog = {"displayname_catalog": {}, "uuid_catalog": {}}
- return ("", simplejson.dumps(return_catalog), 200)
+ return ("", json.dumps(return_catalog), 200)
def _req_resources(conn, method, url, **kwargs):
@@ -340,7 +326,7 @@ def _req_resources(conn, method, url, **kwargs):
return _request_status_400(conn, method, url, **kwargs)
# Return
- return ("", simplejson.dumps(resources), 200)
+ return ("", json.dumps(resources), 200)
def _req_quotas(conn, method, url, **kwargs):
@@ -357,7 +343,7 @@ def _req_quotas(conn, method, url, **kwargs):
return _request_status_401(conn, method, url, **kwargs)
# Return
- return ("", simplejson.dumps(quotas), 200)
+ return ("", json.dumps(quotas), 200)
def _req_commission(conn, method, url, **kwargs):
@@ -375,22 +361,22 @@ def _req_commission(conn, method, url, **kwargs):
if method == "POST":
if 'body' not in kwargs:
return _request_status_400(conn, method, url, **kwargs)
- body = simplejson.loads(unicode(kwargs['body']))
+ body = json.loads(unicode(kwargs['body']))
if re.match('/?'+api_commissions+'$', url) is not None:
# Issue Commission
# Check if we have enough resources to give
if body['provisions'][1]['quantity'] > 420000000:
- return ("", simplejson.dumps(commission_failure_response), 413)
+ return ("", json.dumps(commission_failure_response), 413)
else:
return \
- ("", simplejson.dumps(commission_successful_response), 200)
+ ("", json.dumps(commission_successful_response), 200)
else:
# Issue commission action
serial = url.split('/')[3]
if serial == "action":
# Resolve multiple actions
if body == resolve_commissions_req:
- return ("", simplejson.dumps(resolve_commissions_rep), 200)
+ return ("", json.dumps(resolve_commissions_rep), 200)
else:
return _request_status_400(conn, method, url, **kwargs)
else:
@@ -406,12 +392,12 @@ def _req_commission(conn, method, url, **kwargs):
elif method == "GET":
if re.match('/?'+api_commissions+'$', url) is not None:
# Return pending commission
- return ("", simplejson.dumps(pending_commissions), 200)
+ return ("", json.dumps(pending_commissions), 200)
else:
# Return commissions's description
serial = re.sub('/?' + api_commissions, '', url)[1:]
if serial == str(57):
- return ("", simplejson.dumps(commission_description), 200)
+ return ("", json.dumps(commission_description), 200)
else:
return _request_status_404(conn, method, url, **kwargs)
else:
@@ -781,7 +767,7 @@ class TestCommissions(unittest.TestCase):
global auth_url
try:
client = AstakosClient(token['id'], auth_url)
- response = client.issue_commission(commission_request)
+ response = client._issue_commission(commission_request)
except Exception as err:
self.fail("Shouldn't raise Exception %s" % err)
self.assertEqual(response, commission_successful_response['serial'])
@@ -795,7 +781,7 @@ class TestCommissions(unittest.TestCase):
new_request['provisions'][1]['quantity'] = 520000000
try:
client = AstakosClient(token['id'], auth_url)
- client.issue_commission(new_request)
+ client._issue_commission(new_request)
except QuotaLimit:
pass
except Exception as err:
@@ -811,7 +797,8 @@ class TestCommissions(unittest.TestCase):
client = AstakosClient(token['id'], auth_url)
response = client.issue_one_commission(
"c02f315b-7d84-45bc-a383-552a3f97d2ad",
- "system", {"cyclades.vm": 1, "cyclades.ram": 30000})
+ {("system", "cyclades.vm"): 1,
+ ("system", "cyclades.ram"): 30000})
except Exception as err:
self.fail("Shouldn't have raised Exception %s" % err)
self.assertEqual(response, commission_successful_response['serial'])
diff --git a/astakosclient/astakosclient/utils.py b/astakosclient/astakosclient/utils.py
index 70cd87e82a81e405b58f917a651ec6c92874fed3..f2eb4e58385c0fb5aea052386e18024a47199f91 100644
--- a/astakosclient/astakosclient/utils.py
+++ b/astakosclient/astakosclient/utils.py
@@ -1,35 +1,17 @@
-# Copyright (C) 2012, 2013 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
"""
Astakos Client utility module
@@ -38,10 +20,14 @@ Astakos Client utility module
from httplib import HTTPConnection, HTTPSConnection
from contextlib import closing
-import simplejson
from objpool.http import PooledHTTPConnection
from astakosclient.errors import AstakosClientException, BadValue
+try:
+ import simplejson as json
+except ImportError:
+ import json
+
def retry_dec(func):
"""Class Method Decorator"""
@@ -99,7 +85,7 @@ def scheme_to_class(scheme, use_pool, pool_size):
def parse_request(request, logger):
"""Parse request with simplejson to convert it to string"""
try:
- return simplejson.dumps(request)
+ return json.dumps(request)
except Exception as err:
msg = "Cannot parse request \"%s\" with simplejson: %s" \
% (request, str(err))
@@ -120,3 +106,35 @@ def check_input(function_name, logger, **kwargs):
def join_urls(url_a, url_b):
"""Join_urls from synnefo.lib"""
return url_a.rstrip("/") + "/" + url_b.lstrip("/")
+
+
+def render_overlimit_exception(response, logger):
+ """Render a human readable message for QuotaLimit Exception"""
+ resource_name = {
+ "cyclades.disk": "Disk",
+ "cyclades.vm": "Virtual Machine",
+ "cyclades.cpu": "CPU",
+ "cyclades.ram": "RAM",
+ "cyclades.floating_ip": "Floating IP address",
+ "cyclades.network.private": "Private Network",
+ "pithos.diskspace": "Storage space",
+ "astakos.pending_app": "Pending Applications"
+ }
+ response = json.loads(response)
+ data = response['overLimit']['data']
+ usage = data["usage"]
+ limit = data["limit"]
+ available = limit - usage
+ provision = data['provision']
+ requested = provision['quantity']
+ resource = provision['resource']
+ try:
+ resource = resource_name[resource]
+ except KeyError:
+ logger.error("Unknown resource name '%s'", resource)
+
+ msg = "Resource Limit Exceeded for your account."
+ details = "Limit for resource '%s' exceeded for your account."\
+ " Available: %s, Requested: %s"\
+ % (resource, available, requested)
+ return msg, details
diff --git a/astakosclient/docs/index.rst b/astakosclient/docs/index.rst
index cdf9aea732629ba4fe43bfcc5dd0f22d04d22d9f..1b7b5d81a2844f1aad78739d10a718af553d351e 100644
--- a/astakosclient/docs/index.rst
+++ b/astakosclient/docs/index.rst
@@ -133,15 +133,26 @@ retry=0, use_pool=False, pool_size=8, logger=None\ **)**
It returns user's current quotas (as dict of dicts). In case of error
it raises an AstakosClientException exception.
- **service_get_quotas(**\ user=None\ **)**
+ **service_get_quotas(**\ user=None, project_id=None\ **)**
It returns all users' current quotas for the resources associated with
the service (as dict of dicts of dicts). Optionally, one can query the
- quotas of a specific user with argument user=UUID. In case of error it
- raises an AstakosClientException exception.
+ quotas of a specific user with argument user=UUID (or a list of UUID).
+ Likewise one can specify a project (or a list of projects). In case of
+ error it raises an AstakosClientException exception.
+
+ **service_get_project_quotas(**\ project_id=None\ **)**
+ It returns all projects' current quotas for the resources
+ associated with the service (as dict of dicts).
+ Optionally, one can query the quotas of a specific project with
+ argument project_id=UUID (or a list of UUID). In case of error it raises an
+ AstakosClientException exception.
- **issue_commission(**\ request\ **)**
- Issue a commission. In case of success it returns commission's id
- (int). Otherwise it raises an AstakosClientException exception.
+ **issue_commission_generic(**\ user_provisions, project_provisions, name="", force=False, auto_accept=False\ **)**
+ Issue a commission. User provisions are specified as a dict from
+ (user, project, resource) to int; project provisions as a dict from
+ (project, resource) to int.
+ In case of success return commission's id (int).
+ Otherwise raise an AstakosClientException exception.
**issue_one_commission(**\ holder, source, provisions, name="", force=False, auto_accept=False\ **)**
Issue a commission. We have to specify the holder, the source and the
@@ -150,6 +161,9 @@ retry=0, use_pool=False, pool_size=8, logger=None\ **)**
commission's id (int). Otherwise it raises an AstakosClientException
exception.
+ **issue_resource_reassignment(**\ holder, provisions, name="", force=False, auto_accept=False\ **)**
+ Change resource assignment to another project
+
**get_pending_commissions()**
It returns the pending commissions (list of integers). In case of
error it raises an AstakosClientException exception.
@@ -176,7 +190,7 @@ retry=0, use_pool=False, pool_size=8, logger=None\ **)**
rejected and which failed to resolved. Otherwise raise an
AstakosClientException exception.
- **get_projects(**\ name=None, state=None, owner=None\ **)**
+ **get_projects(**\ name=None, state=None, owner=None, mode=None\ **)**
Retrieve all accessible projects
**get_project(**\ project_id\ **)**
@@ -191,16 +205,10 @@ retry=0, use_pool=False, pool_size=8, logger=None\ **)**
**project_action(**\ project_id, action, reason=""\ **)**
Perform action on a project
- **get_applications(**\ project=None\ **)**
- Retrieve all accessible applications
-
- **get_application(**\ app_id\ **)**
- Retrieve application description, if accessible
-
- **application_action(**\ app_id, action, reason=""\ **)**
- Perform action on an application
+ **application_action(**\ project_id, app_id, action, reason=""\ **)**
+ Perform action on a project application
- **get_memberships(**\ project=None\ **)**
+ **get_memberships(**\ project_id=None\ **)**
Retrieve all accessible memberships
**get_membership(**\ memb_id\ **)**
diff --git a/astakosclient/setup.py b/astakosclient/setup.py
index ab2425038adc8831ba8d21568cf268504fed4bb3..d1fd8752b8d43a1686b12048e5cfe867f6df450a 100644
--- a/astakosclient/setup.py
+++ b/astakosclient/setup.py
@@ -1,35 +1,17 @@
-# Copyright 2012, 2013 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
#
import distribute_setup
distribute_setup.use_setuptools()
@@ -60,9 +42,12 @@ CLASSIFIERS = []
# Package requirements
INSTALL_REQUIRES = [
"objpool>=0.3",
- "simplejson"
]
+EXTRAS_REQUIRES = {
+ 'SimpleJSON': ['simplejson'],
+}
+
# Provided as an attribute, so you can append to these instead
# of replicating them:
standard_exclude = ["*.py", "*.pyc", "*$py.class", "*~", ".*", "*.bak"]
@@ -159,7 +144,7 @@ def find_package_data(
setup(
name='astakosclient',
version=VERSION,
- license='BSD',
+ license='GNU GPLv3',
url='http://www.synnefo.org/',
description=SHORT_DESCRIPTION,
classifiers=CLASSIFIERS,
@@ -176,6 +161,7 @@ setup(
zip_safe=False,
install_requires=INSTALL_REQUIRES,
+ extras_require=EXTRAS_REQUIRES,
tests_require=['mock'],
entry_points={},
diff --git a/ci/ci_squeeze.conf b/ci/ci_squeeze.conf
deleted file mode 100644
index 72f0a4d32c5702c11b937eed9f6006cfdb7975c4..0000000000000000000000000000000000000000
--- a/ci/ci_squeeze.conf
+++ /dev/null
@@ -1,83 +0,0 @@
-[Global]
-# Timeouts in seconds
-build_timeout = 240
-# Apt repository to use
-apt_repo =
- deb http://apt.dev.grnet.gr squeeze/
- deb http://packages.x2go.org/debian squeeze main
-
-# Synnefo git repo.
-# If not set, snf-ci will copy and use the local repo.
-synnefo_repo = https://code.grnet.gr/git/synnefo
-# Git branch to test (specify sha1 or branch name). If not set, the
-# branch/sha will result from the current repository.
-synnefo_branch =
-
-# pithos-web-client git repo
-pithos_webclient_repo = https://code.grnet.gr/git/pithos-web-client
-# Git branch to use for pithos-web-client
-# If not set, snf-ci will decide which one to use
-pithos_webclient_branch =
-
-# Defines the schema that snf-deploy will use
-schema = one_node_squeeze
-# Local dir to save builded packages
-pkgs_dir = /tmp/synnefo_pkgs
-# If True patch the pydist.py module (see Debian bug #657665)
-patch_pydist = True
-
-# Configuration of git (on remote server)
-git_config_name = Buildbot
-git_config_mail = synnefo@builder.dev.grnet.gr
-
-# Network address from which we allow access to server.
-# If not set, access to server is not restricted.
-accept_ssh_from =
-# Config file to save temporary options (eg IPs, passwords etc)
-temporary_config = /tmp/ci_temp_conf
-# File to save the x2goplugin html file
-x2go_plugin_file = /tmp/x2go.html
-
-
-[Deployment]
-# Choose the 'cloud' to use from .kamakirc
-kamaki_cloud =
-# Server name to use for our machine
-server_name = Synnefo_CI
-# A list of flavors (comma seperated) to choose from
-# The user can specify a flavor name (reg expression)
-# with "name:" or a flavor id with "id:".
-flavors = name:C2R2...D20ext_.*, name:C2R2...D20drbd, id:1
-# A list of images (comma seperated) to choose from
-# The user can specify an image name (reg expression)
-# with "name:" or an image id with "id:".
-images = name:SynnefoCISqueeze.*, name:Debian Base \(OldStable\), id:72d9844f-1024-4a07-a3c3-60d650b8f5cd
-# File containing the ssh keys to upload/install to server
-# If not set, no ssh keys will be installed
-ssh_keys = ~/.ssh/id_rsa.pub
-
-
-[Burnin]
-# Maybe add some burnin options
-# (e.g. tests to run/ignore, timeouts etc)
-cmd_options = --images "name:.*" --flavors "name:C1R512D2file" --no-ipv6
-
-
-[Unit Tests]
-component = astakos cyclades pithos astakosclient
-
-
-[Repository]
-# Projects reside on this repo
-projects =
- snf-common
- astakosclient
- snf-django-lib
- snf-webproject
- snf-branding
- snf-astakos-app
- snf-pithos-backend
- snf-cyclades-gtools
- snf-cyclades-app
- snf-pithos-app
- snf-tools
diff --git a/ci/ci_wheezy.conf b/ci/ci_wheezy.conf
index c3b8fb0275c0d82543fe44dd215eb1ff5da7c54d..9d3a27b0fc320450e0e53f57cb99216f220b64b9 100644
--- a/ci/ci_wheezy.conf
+++ b/ci/ci_wheezy.conf
@@ -8,11 +8,12 @@ apt_repo =
# Synnefo git repo.
# If not set, snf-ci will copy and use the local repo.
-synnefo_repo = https://code.grnet.gr/git/synnefo
+synnefo_repo = https://github.com/grnet/synnefo
# Git branch to test (specify sha1 or branch name). If not set, the
# branch/sha will result from the current repository.
synnefo_branch =
+build_pithos_webclient = True
# pithos-web-client git repo
pithos_webclient_repo = https://code.grnet.gr/git/pithos-web-client
# Git branch to use for pithos-web-client
@@ -35,6 +36,8 @@ git_config_mail = synnefo@builder.dev.grnet.gr
accept_ssh_from =
# Config file to save temporary options (eg IPs, passwords etc)
temporary_config = /tmp/ci_temp_conf
+# Install x2go and firefox
+setup_x2go = True
# File to save the x2goplugin html file
x2go_plugin_file = /tmp/x2go.html
@@ -47,7 +50,7 @@ server_name = Synnefo_CI
# A list of flavors (comma seperated) to choose from
# The user can specify a flavor name (reg expression)
# with "name:" or a flavor id with "id:".
-flavors = name:C2R2...D20ext_.*, name:C2R2...D20drbd, id:1
+flavors = name:C2R4...D20ext_.*, name:C2R4...D20drbd, id:1
# A list of images (comma seperated) to choose from
# The user can specify an image name (reg expression)
# with "name:" or an image id with "id:".
@@ -59,16 +62,19 @@ ssh_keys = ~/.ssh/id_rsa.pub
allocate_floating_ip = True
# List of networks IDs (comma seperated) to connect server
private_networks =
+# Connect to a specific ssh port. If not set, the ssh port is calculated
+# automatically.
+ssh_port =
[Burnin]
# Maybe add some burnin options
# (e.g. tests to run/ignore, timeouts etc)
-cmd_options = --images "name:.*" --flavors "name:C1R512D2file" --no-ipv6
+cmd_options = --images "name:.*" --flavors "name:C1R512D2\D.*" --no-ipv6
[Unit Tests]
-component = astakos cyclades pithos astakosclient
+component = astakos cyclades pithos astakosclient admin
[Repository]
@@ -83,5 +89,6 @@ projects =
snf-pithos-backend
snf-cyclades-gtools
snf-cyclades-app
+ snf-admin-app
snf-pithos-app
snf-tools
diff --git a/ci/config b/ci/config
index 4987150f584c2ddcd8562895c63929e9f76fc823..240ccd36f646d64fba6f07fb5a0443b2d2e4e851 100644
--- a/ci/config
+++ b/ci/config
@@ -9,4 +9,5 @@ PROJECTS="\
snf-cyclades-gtools\
snf-cyclades-app\
snf-pithos-app\
+ snf-admin-app\
snf-tools"
diff --git a/ci/develop-common.sh b/ci/develop-common.sh
new file mode 100755
index 0000000000000000000000000000000000000000..31e4801e3f4f037630016569b279fb472d592fc3
--- /dev/null
+++ b/ci/develop-common.sh
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+if [ -n "$VIRTUAL_ENV" ]; then
+ OPTIONS=--script-dir=$VIRTUAL_ENV/bin/
+ echo $OPTIONS
+else
+ OPTIONS=
+fi
+
+. ./ci/config
diff --git a/ci/install.sh b/ci/install.sh
index e949a541e6485f089e51585375fb528cedf9e1e1..ccf0be7ec2479f72548fea3ff51cbfecbee58823 100755
--- a/ci/install.sh
+++ b/ci/install.sh
@@ -1,17 +1,12 @@
#!/bin/sh
-if [ -n "$VIRTUAL_ENV" ]; then
- OPTIONS=--script-dir=$VIRTUAL_ENV/bin/
- echo $OPTIONS
-else
- OPTIONS=
-fi
-
+# `cd` to the top dir of synnefo repository
set -e
cwd=`dirname $0`
cd "$cwd"/..
-. ./ci/config
+# Do common tasks for install/uninstall purposes
+. ./ci/develop-common.sh
# Update version
devflow-update-version
diff --git a/ci/schemas/one_node_squeeze/deploy.conf b/ci/schemas/one_node_squeeze/deploy.conf
deleted file mode 100644
index c0fb53ed1f00871db3abb172fdb81c4602d013eb..0000000000000000000000000000000000000000
--- a/ci/schemas/one_node_squeeze/deploy.conf
+++ /dev/null
@@ -1,41 +0,0 @@
-[packages]
-# whether to use apt-get or local generated package found in packages dir
-use_local_packages = True
-
-# url to obtain latest synnefo packages.
-# To use them change USE_LOCAL_PACKAGES setting to yes
-# To get them run: snf-deploy packages
-package_url = http://builder.dev.grnet.gr/synnefo/packages/Squeeze/40/
-
-[dirs]
-# dir to find all template files used to customize setup
-# in case you want to add another setting please modify the corresponding file
-templates = /var/lib/snf-deploy/files
-# dir to store local images (disk0, disk1 of the virtual cluster)
-images = /var/lib/snf-deploy/images
-# dir to store/find local packages
-# dir to locally save packages that will be downloaded from package_url
-# put here any locally created packages (useful for development)
-packages = /var/lib/snf-deploy/packages
-# dir to store pidfiles (dnsmasq, kvm)
-run = /var/run/snf-deploy
-# dir to store dnsmasq related files
-dns = /var/lib/snf-deploy/dnsmasq
-# dir to lookup fabfile and ifup script
-lib = /usr/lib/snf-deploy
-# dir to store executed commands (to enforce sequential execution)
-cmd = /var/run/snf-deploy/cmd
-# dir to be used by Django for file-based mail backend
-mail_dir = /var/tmp/synnefo-mails
-
-[keys]
-# whether to create new keys
-keygen = False
-# whether to inject ssh keys found in templates/root/.ssh in nodes
-key_inject = True
-
-[options]
-# Deploy Synnefo, specially tuned for testing. This option improves the speed
-# of some operations, but is not safe for all enviroments. (e.g. disable
-# fsync of postgresql)
-testing_vm = True
diff --git a/ci/schemas/one_node_squeeze/ganeti.conf b/ci/schemas/one_node_squeeze/ganeti.conf
deleted file mode 100644
index d3507ed21856d2c2e5f4e9b58904d90ed21aaf6a..0000000000000000000000000000000000000000
--- a/ci/schemas/one_node_squeeze/ganeti.conf
+++ /dev/null
@@ -1,18 +0,0 @@
-[ganeti1]
-cluster_nodes = node1
-master_node = node1
-
-cluster_netdev = eth0
-cluster_name = ganeti1
-cluster_ip = 192.168.0.13
-
-vg = autovg
-
-synnefo_public_network_subnet = 10.2.1.0/24
-synnefo_public_network_gateway = 10.2.1.1
-synnefo_public_network_type = CUSTOM
-
-image_dir = /srv/okeanos
-
-# To add another cluster repeat the above section
-# with different header and nodes
diff --git a/ci/schemas/one_node_squeeze/nodes.conf b/ci/schemas/one_node_squeeze/nodes.conf
deleted file mode 100644
index c4d1a66e61918b01c37cd0ba5271f2bffd465edf..0000000000000000000000000000000000000000
--- a/ci/schemas/one_node_squeeze/nodes.conf
+++ /dev/null
@@ -1,42 +0,0 @@
-# please note that currently is only supported deployment
-# with nodes (both ganeti and synnefo) residing in the same subnet/domain
-[network]
-domain = synnefo.live
-
-[os]
-node1 = squeeze
-# node2 = wheezy
-
-[hostnames]
-node1 = auto1
-# node2 = auto2
-
-[ips]
-node1 = 192.168.0.1
-# node2 = 192.168.0.2
-
-# This is used only in case of vcluster
-# needed to pass the correct dhcp responces to the virtual nodes
-[macs]
-node1 = 52:54:00:00:00:01
-# node2 = 52:54:00:00:00:02
-
-[info]
-# Here we define which nodes from the predefined ones to use
-nodes = node1
-
-# login credentials for the nodes
-# please note that in case of vcluster these are preconfigured
-# and not editable.
-# in case of physical nodes all nodes should have the same login account
-user = root
-password = 12345
-
-public_iface = eth0
-vm_public_iface = eth1
-vm_private_iface = eth2
-
-# extra disk name inside the nodes
-# if defined, snf-deploy will create a VG for ganeti in order to support lvm storage
-# if not then only file disk template will be supported
-extra_disk = /dev/vdb
diff --git a/ci/schemas/one_node_squeeze/squeeze.conf b/ci/schemas/one_node_squeeze/squeeze.conf
deleted file mode 100644
index b29389a1c0c9f3d90b1e428a1d80e099db65a5cf..0000000000000000000000000000000000000000
--- a/ci/schemas/one_node_squeeze/squeeze.conf
+++ /dev/null
@@ -1,56 +0,0 @@
-[debian]
-rabbitmq-server = squeeze-backports
-gunicorn = squeeze-backports
-qemu-kvm = squeeze-backports
-qemu = squeeze-backports
-python-gevent = squeeze-backports
-apache2 =
-postgresql =
-python-psycopg2 =
-python-argparse =
-nfs-kernel-server = squeeze-backports
-nfs-common = squeeze-backports
-bind9 =
-vlan =
-vlan =
-lvm2 =
-curl =
-memcached =
-python-memcache =
-bridge-utils =
-python-progress =
-ganeti-instance-debootstrap =
-python-django-south = squeeze-backports
-python-django = squeeze-backports
-drbd8-utils =
-
-
-[synnefo]
-snf-astakos-app = squeeze
-snf-common = squeeze
-snf-cyclades-app = squeeze
-snf-cyclades-gtools = squeeze
-snf-django-lib = squeeze
-python-astakosclient = squeeze
-snf-branding = squeeze
-snf-webproject = squeeze
-snf-pithos-app = squeeze
-snf-pithos-backend = squeeze
-snf-tools = squeeze
-
-
-[ganeti]
-snf-ganeti = 2.6.2+ippool11+hotplug5+extstorage3+rbdfix1+kvmfix2+nolvm+netxen-1~squeeze
-ganeti-htools = 2.6.2+ippool11+hotplug5+extstorage3+rbdfix1+kvmfix2+nolvm+netxen-1~squeeze
-
-[other]
-snf-cloudcms = squeeze
-snf-vncauthproxy = squeeze
-snf-pithos-webclient = squeeze
-snf-image = squeeze
-snf-network = squeeze
-python-objpool = squeeze
-nfdhcpd = squeeze
-kamaki = squeeze
-python-bitarray = squeeze-backports
-nfqueue-bindings-python = 0.3+physindev-1
diff --git a/ci/schemas/one_node_squeeze/synnefo.conf b/ci/schemas/one_node_squeeze/synnefo.conf
deleted file mode 100644
index e0ac23ded0d6203e6b07bbca42e3354892031807..0000000000000000000000000000000000000000
--- a/ci/schemas/one_node_squeeze/synnefo.conf
+++ /dev/null
@@ -1,37 +0,0 @@
-[cred]
-synnefo_user = synnefo
-synnefo_db_passwd = example_passw0rd
-synnefo_rapi_passwd = example_rapi_passw0rd
-synnefo_rabbitmq_passwd = example_rabbitmq_passw0rd
-user_email = user@synnefo.org
-user_name = John
-user_lastname = Doe
-user_passwd = 12345
-
-
-[roles]
-accounts = node1
-compute = node1
-object-store = node1
-cyclades = node1
-pithos = node1
-cms = node1
-db = node1
-mq = node1
-ns = node1
-client = node1
-router = node1
-
-
-[synnefo]
-pithos_dir = /srv/pithos
-flavor_cpu = 1,2,4,8
-flavor_ram = 128,256,512,1024,2048,4096,8192
-flavor_disk = 2,5,10,20,40,60,80,100
-flavor_storage = file
-
-vm_public_bridge = br0
-vm_private_bridge = prv0
-common_bridge = br0
-
-debian_base_url = http://cdn.synnefo.org/debian_base-7.0-x86_64.diskdump
diff --git a/ci/schemas/one_node_squeeze/vcluster.conf b/ci/schemas/one_node_squeeze/vcluster.conf
deleted file mode 100644
index 377b2a9c7ec3784813a962d9a7035db0bcf29cf1..0000000000000000000000000000000000000000
--- a/ci/schemas/one_node_squeeze/vcluster.conf
+++ /dev/null
@@ -1,40 +0,0 @@
-[image]
-# url to get the base image. This is a debian base image with preconfigured
-# root password and installed rsa/dsa keys. Plus a NetworkManager hook that
-# changes the VM's name based on info provided by dhcp response.
-# To create it run: snf-deploy image
-squeeze_image_url = https://pithos.okeanos.grnet.gr/public/832xv
-ubuntu_image_url =
-
-# in order ganeti nodes to support lvm storage (plain disk template) it will
-# be needed an extra disk to eventually be able to create a VG. Ganeti requires
-# this VG to be at least of 30GB. To this end in order the virtual nodes to have
-# this extra disk an image should be created locally. There are three options:
-# 1. not create an extra disk (only file storage template will be supported)
-# 2. create an image of 30G in image dir (default /var/lib/snf-deploy/images)
-# using dd if=/dev/zero of=squeeze.disk1
-# 3. create this image in a local VG using lvgreate -L30G squeeze.disk1 lvg
-# and create a symbolic link in /var/lib/snf-deploy/images
-
-# Whether to create an extra disk or not
-create_extra_disk = False
-# lvg is the name of the local VG if any
-lvg =
-
-# OS istalled in the virtual cluster
-os = squeeze
-
-
-[cluster]
-# the bridge to use for the virtual cluster
-# on this bridge we will launch a dnsnmasq and provide
-# fqdns needed to the cluster.
-# In ordrer cluster nodes to have internet access, host must do NAT.
-# iptables -t nat -A POSTROUTING -s 192.0.0.0/28 -j MASQUERADE
-# ip addr add 192.0.0.14/28 dev auto_nodes_br
-# To create run: snf-deploy cluster
-bridge = auto_nodes_br
-
-[network]
-subnet = 192.168.0.0/28
-gateway = 192.168.0.14
diff --git a/ci/schemas/one_node_squeeze/wheezy.conf b/ci/schemas/one_node_squeeze/wheezy.conf
deleted file mode 100644
index 940c981ab245738de6b73ddd8a51dcbdfcf2f085..0000000000000000000000000000000000000000
--- a/ci/schemas/one_node_squeeze/wheezy.conf
+++ /dev/null
@@ -1,55 +0,0 @@
-[debian]
-rabbitmq-server =
-gunicorn =
-qemu-kvm =
-qemu =
-python-gevent =
-apache2 =
-postgresql =
-python-psycopg2 =
-python-argparse =
-nfs-kernel-server =
-nfs-common =
-bind9 =
-vlan =
-vlan =
-lvm2 =
-curl =
-memcached =
-python-memcache =
-bridge-utils =
-python-progress =
-ganeti-instance-debootstrap =
-python-django-south =
-python-django =
-drbd8-utils =
-
-
-[synnefo]
-snf-astakos-app = wheezy
-snf-common = wheezy
-snf-cyclades-app = wheezy
-snf-cyclades-gtools = wheezy
-snf-django-lib = wheezy
-python-astakosclient = wheezy
-snf-branding = wheezy
-snf-webproject = wheezy
-snf-pithos-app = wheezy
-snf-pithos-backend = wheezy
-snf-tools = wheezy
-
-
-[ganeti]
-snf-ganeti = 2.8.0+hotplug+ippoolfix-1
-ganeti-htools = 2.8.0+hotplug+ippoolfix-1
-[other]
-snf-cloudcms = wheezy
-snf-vncauthproxy = wheezy
-snf-pithos-webclient = wheezy
-snf-image = wheezy
-snf-network = wheezy
-python-objpool = wheezy
-nfdhcpd = wheezy
-kamaki = wheezy
-python-bitarray = wheezy
-nfqueue-bindings-python = 0.3+physindev-1~wheezy
diff --git a/ci/schemas/one_node_wheezy/deploy.conf b/ci/schemas/one_node_wheezy/deploy.conf
index c0fb53ed1f00871db3abb172fdb81c4602d013eb..952ef4991601ba6ab4b5d764040fa930ada6e04a 100644
--- a/ci/schemas/one_node_wheezy/deploy.conf
+++ b/ci/schemas/one_node_wheezy/deploy.conf
@@ -1,40 +1,34 @@
-[packages]
+[DEFAULT]
# whether to use apt-get or local generated package found in packages dir
use_local_packages = True
# url to obtain latest synnefo packages.
-# To use them change USE_LOCAL_PACKAGES setting to yes
# To get them run: snf-deploy packages
package_url = http://builder.dev.grnet.gr/synnefo/packages/Squeeze/40/
-[dirs]
+# dir to store snf-deploy status
+state_dir = /var/lib/snf-deploy
# dir to find all template files used to customize setup
# in case you want to add another setting please modify the corresponding file
-templates = /var/lib/snf-deploy/files
-# dir to store local images (disk0, disk1 of the virtual cluster)
-images = /var/lib/snf-deploy/images
+template_dir = /var/lib/snf-deploy/files
+# dir to store disks for the virtual cluster)
+vcluster_dir = /var/lib/snf-deploy/vcluster
# dir to store/find local packages
# dir to locally save packages that will be downloaded from package_url
# put here any locally created packages (useful for development)
-packages = /var/lib/snf-deploy/packages
+package_dir = /var/lib/snf-deploy/packages
# dir to store pidfiles (dnsmasq, kvm)
-run = /var/run/snf-deploy
+run_dir = /var/run/snf-deploy
# dir to store dnsmasq related files
-dns = /var/lib/snf-deploy/dnsmasq
+dns_dir = /var/lib/snf-deploy/dnsmasq
# dir to lookup fabfile and ifup script
-lib = /usr/lib/snf-deploy
-# dir to store executed commands (to enforce sequential execution)
-cmd = /var/run/snf-deploy/cmd
+lib_dir = /usr/lib/snf-deploy
# dir to be used by Django for file-based mail backend
mail_dir = /var/tmp/synnefo-mails
-[keys]
-# whether to create new keys
-keygen = False
# whether to inject ssh keys found in templates/root/.ssh in nodes
key_inject = True
-[options]
# Deploy Synnefo, specially tuned for testing. This option improves the speed
# of some operations, but is not safe for all enviroments. (e.g. disable
# fsync of postgresql)
diff --git a/ci/schemas/one_node_wheezy/ganeti.conf b/ci/schemas/one_node_wheezy/ganeti.conf
index d3507ed21856d2c2e5f4e9b58904d90ed21aaf6a..f3ca8bddc250f676e6902dfd65414d95c04686de 100644
--- a/ci/schemas/one_node_wheezy/ganeti.conf
+++ b/ci/schemas/one_node_wheezy/ganeti.conf
@@ -1,18 +1,27 @@
-[ganeti1]
-cluster_nodes = node1
-master_node = node1
+[DEFAULT]
+vg = ganeti
+# Ganeti has hard requiremend for VG larger than 20480M
+vg_size = 30G
+# whether to add synnefo related packages
+synnefo = True
-cluster_netdev = eth0
-cluster_name = ganeti1
-cluster_ip = 192.168.0.13
+[ganeti]
+name = ganeti
+domain = synnefo.live
+ip = 10.1.2.101
+netdev = eth0
-vg = autovg
-synnefo_public_network_subnet = 10.2.1.0/24
-synnefo_public_network_gateway = 10.2.1.1
-synnefo_public_network_type = CUSTOM
+[ganeti-qa]
+name = ganeti
+domain = qa.synnefo.live
+ip = 10.1.2.101
+netdev = eth0
+synnefo =
-image_dir = /srv/okeanos
-# To add another cluster repeat the above section
-# with different header and nodes
+[ganeti-vc]
+name = ganeti
+domain = vcluster.synnefo.live
+ip = 10.1.2.101
+netdev = eth0
diff --git a/ci/schemas/one_node_wheezy/nodes.conf b/ci/schemas/one_node_wheezy/nodes.conf
index 27f60f0d28a643819cafae19624e941e856073ed..6fb814e10e5e5d34810ae6fd1d2a6948cf161c99 100644
--- a/ci/schemas/one_node_wheezy/nodes.conf
+++ b/ci/schemas/one_node_wheezy/nodes.conf
@@ -1,42 +1,104 @@
-# please note that currently is only supported deployment
-# with nodes (both ganeti and synnefo) residing in the same subnet/domain
-[network]
+# In this section we define configuration setting common to all nodes
+[DEFAULT]
+# Currently both ganeti and synnefo must reside in the same domain
+# Instances will reside in the .vm. subdomain
domain = synnefo.live
-[os]
-node1 = wheezy
-# node2 = wheezy
+# Each node should define:
-[hostnames]
-node1 = auto1
-# node2 = auto2
-
-[ips]
-node1 = 192.168.0.1
-# node2 = 192.168.0.2
+# The node's desired hostname. It will be set
+hostname =
+# The node's primary IP
+ip =
# This is used only in case of vcluster
# needed to pass the correct dhcp responces to the virtual nodes
-[macs]
-node1 = 52:54:00:00:00:01
-# node2 = 52:54:00:00:00:02
-
-[info]
-# Here we define which nodes from the predefined ones to use
-nodes = node1
-
-# login credentials for the nodes
-# please note that in case of vcluster these are preconfigured
-# and not editable.
-# in case of physical nodes all nodes should have the same login account
+mac =
+
+# The node's OS (debian, ubuntu, etc)
+# Currently tested only under debian (wheezy)
+os = debian
+
+# The node's administrator account (with root priviledges)
user = root
-password = 12345
+# The root's password
+password =
+# The interface with internet access
public_iface = eth0
+# The interface for the instances' public traffic
vm_public_iface = eth1
+# The interface for the instances' private traffic
vm_private_iface = eth2
-# extra disk name inside the nodes
-# if defined, snf-deploy will create a VG for ganeti in order to support lvm storage
-# if not then only file disk template will be supported
+# The extra disk for the Ganeti VG needed for plain and drbd disk templates
extra_disk = /dev/vdb
+
+###################
+# synnefo/ci node #
+###################
+
+[node]
+name = node
+ip = 192.0.2.1
+extra_disk =
+
+############
+# qa nodes #
+############
+
+[dev]
+name = qa2
+ip = 10.1.2.10
+public_iface = eth1
+domain = qa.synnefo.live
+
+[qa1]
+name = qa1
+ip = 10.1.2.11
+public_iface = eth1
+domain = qa.synnefo.live
+
+[qa2]
+name = qa2
+ip = 10.1.2.12
+public_iface = eth1
+domain = qa.synnefo.live
+
+############
+# vc nodes #
+############
+
+[vc1]
+mac = 52:54:00:00:00:01
+name = vc1
+ip = 10.1.2.1
+public_iface = eth0
+domain = vcluster.synnefo.live
+
+[vc2]
+mac = 52:54:00:00:00:02
+name = vc2
+ip = 10.1.2.2
+public_iface = eth0
+domain = vcluster.synnefo.live
+
+[vc3]
+mac = 52:54:00:00:00:03
+name = vc3
+ip = 10.1.2.3
+public_iface = eth0
+domain = vcluster.synnefo.live
+
+[vc4]
+mac = 52:54:00:00:00:04
+name = vc4
+ip = 10.1.2.4
+public_iface = eth0
+domain = vcluster.synnefo.live
+
+[dummy]
+name = dummy
+ip = 1.2.3.4
+public_iface = eth0
+domain = synnefo.live
diff --git a/ci/schemas/one_node_wheezy/packages.conf b/ci/schemas/one_node_wheezy/packages.conf
new file mode 100644
index 0000000000000000000000000000000000000000..41b2b22f1d0252f5d5a399adb25841907b492bdb
--- /dev/null
+++ b/ci/schemas/one_node_wheezy/packages.conf
@@ -0,0 +1,9 @@
+[debian]
+python-nfqueue = 0.4+physindev-1~wheezy
+python-scapy = 2.2.0+rfc6355-1
+snf-ganeti = unstable
+ganeti2 = unstable
+python-django-eztables = 0.3.3-1~snf~0.2
+qemu-kvm = wheezy-backports
+
+[ubuntu]
diff --git a/ci/schemas/one_node_wheezy/setups.conf b/ci/schemas/one_node_wheezy/setups.conf
new file mode 100644
index 0000000000000000000000000000000000000000..ecf78432c2b8de130586f1d5a1dbbb4fe98f2b2c
--- /dev/null
+++ b/ci/schemas/one_node_wheezy/setups.conf
@@ -0,0 +1,79 @@
+[DEFAULT]
+
+#################################
+# snf-deploy synnefo --autoconf #
+#################################
+
+[auto]
+ns = node
+ca = node
+client = node
+router = node
+nfs = node
+db = node
+mq = node
+astakos = node
+cyclades = node
+admin = node
+vnc = node
+pithos = node
+cms = node
+stats = node
+dev = node
+clusters =
+ ganeti
+
+
+[ganeti]
+master = node
+vmc =
+ node
+
+###################################
+# snf-deploy ganeti-qa --setup qa #
+###################################
+
+[qa]
+ns = dev
+client = dev
+router = qa1
+nfs = dev
+dev = dev
+clusters =
+ ganeti-qa
+
+
+[ganeti-qa]
+master = qa1
+vmc =
+ qa1
+ qa2
+
+##################################
+# snf-deploy vcluster --setup vc #
+##################################
+
+[vc]
+ns = vc1
+client = vc4
+router = vc1
+nfs = vc1
+db = vc2
+mq = vc3
+astakos = vc1
+cyclades = vc2
+pithos = vc3
+cms = vc4
+stats = vc1
+dev = vc1
+clusters =
+ ganeti-vc
+
+
+[ganeti-vc]
+master = vc1
+vmc =
+ vc1
+ vc2
+ vc3
+ vc4
diff --git a/ci/schemas/one_node_wheezy/squeeze.conf b/ci/schemas/one_node_wheezy/squeeze.conf
deleted file mode 100644
index 4b302abab1b375c0f3125efe9765f58d5db75695..0000000000000000000000000000000000000000
--- a/ci/schemas/one_node_wheezy/squeeze.conf
+++ /dev/null
@@ -1,55 +0,0 @@
-[debian]
-rabbitmq-server = squeeze-backports
-gunicorn = squeeze-backports
-qemu-kvm = squeeze-backports
-qemu = squeeze-backports
-python-gevent = squeeze-backports
-apache2 =
-postgresql =
-python-psycopg2 =
-python-argparse =
-nfs-kernel-server = squeeze-backports
-nfs-common = squeeze-backports
-bind9 =
-vlan =
-vlan =
-lvm2 =
-curl =
-memcached =
-python-memcache =
-bridge-utils =
-python-progress =
-ganeti-instance-debootstrap =
-python-django-south = squeeze-backports
-drbd8-utils =
-
-
-[synnefo]
-snf-astakos-app = squeeze
-snf-common = squeeze
-snf-cyclades-app = squeeze
-snf-cyclades-gtools = squeeze
-snf-django-lib = squeeze
-python-astakosclient = squeeze
-snf-branding = squeeze
-snf-webproject = squeeze
-snf-pithos-app = squeeze
-snf-pithos-backend = squeeze
-snf-tools = squeeze
-
-
-[ganeti]
-snf-ganeti = 2.6.2+ippool11+hotplug5+extstorage3+rbdfix1+kvmfix2+nolvm+netxen-1~squeeze
-ganeti-htools = 2.6.2+ippool11+hotplug5+extstorage3+rbdfix1+kvmfix2+nolvm+netxen-1~squeeze
-
-[other]
-snf-cloudcms = squeeze
-snf-vncauthproxy = squeeze
-snf-pithos-webclient = squeeze
-snf-image = squeeze
-snf-network = squeeze
-python-objpool = squeeze
-nfdhcpd = squeeze
-kamaki = squeeze
-python-bitarray = squeeze-backports
-nfqueue-bindings-python = 0.3+physindev-1
diff --git a/ci/schemas/one_node_wheezy/synnefo.conf b/ci/schemas/one_node_wheezy/synnefo.conf
index bbcb7bb80a5b6ec09e98ecdb39a75a3d696d89db..e5cad0755835b39768ef5f827a9982d670814cef 100644
--- a/ci/schemas/one_node_wheezy/synnefo.conf
+++ b/ci/schemas/one_node_wheezy/synnefo.conf
@@ -1,39 +1,39 @@
-[cred]
+[DEFAULT]
+# various credentials
synnefo_user = synnefo
synnefo_db_passwd = example_passw0rd
synnefo_rapi_passwd = example_rapi_passw0rd
synnefo_rabbitmq_passwd = example_rabbitmq_passw0rd
+synnefo_vnc_passwd = example_vnc_passw0rd
+cyclades_secret = example_cyclades_secret
+oa2_secret = example_oa2_secret
+webproject_secret = example_webproject_secret
+stats_secret = example_stats_secret
+collectd_secret = example_collectd_secret
+
user_email = user@synnefo.org
user_name = John
user_lastname = Doe
user_passwd = 12345
-oa2_secret = 12345
-
-[roles]
-accounts = node1
-compute = node1
-object-store = node1
-cyclades = node1
-pithos = node1
-cms = node1
-db = node1
-mq = node1
-ns = node1
-client = node1
-router = node1
-stats = node1
+# one common shared dir for nfs
+shared_dir = /srv
-
-[synnefo]
-pithos_dir = /srv/pithos
flavor_cpu = 1,2,4,8
flavor_ram = 128,256,512,1024,2048,4096,8192
flavor_disk = 2,5,10,20,40,60,80,100
-flavor_storage = file
+flavor_storage = file,ext_archipelago,ext_shared-filer
+
+# url to download debian wheezy image
+debian_base_url = http://cdn.synnefo.org/debian_base-7.0-x86_64.diskdump
+
+# archipelago segment
+segment_size = 512
+# options related to synnefo networks
vm_public_bridge = br0
vm_private_bridge = prv0
common_bridge = br0
-
-debian_base_url = http://cdn.synnefo.org/debian_base-7.0-x86_64.diskdump
+synnefo_public_network_subnet = 10.2.1.0/24
+synnefo_public_network_gateway = 10.2.1.1
+synnefo_public_network_type = CUSTOM
diff --git a/ci/schemas/one_node_wheezy/vcluster.conf b/ci/schemas/one_node_wheezy/vcluster.conf
index 377b2a9c7ec3784813a962d9a7035db0bcf29cf1..52995cd4cf899e4a10efdc213fd280837af1bd8a 100644
--- a/ci/schemas/one_node_wheezy/vcluster.conf
+++ b/ci/schemas/one_node_wheezy/vcluster.conf
@@ -1,31 +1,7 @@
-[image]
-# url to get the base image. This is a debian base image with preconfigured
-# root password and installed rsa/dsa keys. Plus a NetworkManager hook that
-# changes the VM's name based on info provided by dhcp response.
-# To create it run: snf-deploy image
-squeeze_image_url = https://pithos.okeanos.grnet.gr/public/832xv
-ubuntu_image_url =
+[DEFAULT]
+disk0_size = 10G
+disk1_size = 30G
-# in order ganeti nodes to support lvm storage (plain disk template) it will
-# be needed an extra disk to eventually be able to create a VG. Ganeti requires
-# this VG to be at least of 30GB. To this end in order the virtual nodes to have
-# this extra disk an image should be created locally. There are three options:
-# 1. not create an extra disk (only file storage template will be supported)
-# 2. create an image of 30G in image dir (default /var/lib/snf-deploy/images)
-# using dd if=/dev/zero of=squeeze.disk1
-# 3. create this image in a local VG using lvgreate -L30G squeeze.disk1 lvg
-# and create a symbolic link in /var/lib/snf-deploy/images
-
-# Whether to create an extra disk or not
-create_extra_disk = False
-# lvg is the name of the local VG if any
-lvg =
-
-# OS istalled in the virtual cluster
-os = squeeze
-
-
-[cluster]
# the bridge to use for the virtual cluster
# on this bridge we will launch a dnsnmasq and provide
# fqdns needed to the cluster.
@@ -33,8 +9,7 @@ os = squeeze
# iptables -t nat -A POSTROUTING -s 192.0.0.0/28 -j MASQUERADE
# ip addr add 192.0.0.14/28 dev auto_nodes_br
# To create run: snf-deploy cluster
-bridge = auto_nodes_br
+bridge = vcluster_bridge
-[network]
-subnet = 192.168.0.0/28
-gateway = 192.168.0.14
+subnet = 10.1.2.0/24
+gateway = 10.1.2.254
diff --git a/ci/schemas/one_node_wheezy/wheezy.conf b/ci/schemas/one_node_wheezy/wheezy.conf
deleted file mode 100644
index b80b9e34b2e2ef8b9a79bb3aaa66a97ae06cf308..0000000000000000000000000000000000000000
--- a/ci/schemas/one_node_wheezy/wheezy.conf
+++ /dev/null
@@ -1,60 +0,0 @@
-[debian]
-rabbitmq-server =
-gunicorn =
-qemu-kvm =
-qemu =
-python-gevent =
-apache2 =
-postgresql =
-python-psycopg2 =
-python-argparse =
-nfs-kernel-server =
-nfs-common =
-bind9 =
-vlan =
-vlan =
-lvm2 =
-curl =
-memcached =
-python-memcache =
-bridge-utils =
-python-progress =
-ganeti-instance-debootstrap =
-python-django-south =
-python-django =
-drbd8-utils =
-collectd =
-
-
-[synnefo]
-snf-astakos-app = wheezy
-snf-common = wheezy
-snf-cyclades-app = wheezy
-snf-cyclades-gtools = wheezy
-snf-django-lib = wheezy
-python-astakosclient = wheezy
-snf-branding = wheezy
-snf-webproject = wheezy
-snf-pithos-app = wheezy
-snf-pithos-backend = wheezy
-snf-tools = wheezy
-snf-stats-app = wheezy
-
-
-[ganeti]
-snf-ganeti = wheezy
-ganeti-htools = wheezy
-ganeti-haskell = wheezy
-
-
-[other]
-snf-cloudcms = wheezy
-snf-vncauthproxy = unstable
-snf-pithos-webclient = wheezy
-snf-image = wheezy
-snf-network = wheezy
-python-objpool = wheezy
-nfdhcpd = wheezy
-kamaki = wheezy
-python-bitarray = wheezy
-python-nfqueue = 0.4+physindev-1~wheezy
diff --git a/ci/snf-ci b/ci/snf-ci
index a46b0b1df0b7772db95ef40745b1a151b0083861..b94e294c5bc1b9ff6606f96b4326ab34dd873b75 100755
--- a/ci/snf-ci
+++ b/ci/snf-ci
@@ -1,4 +1,19 @@
#!/usr/bin/env python
+#
+# Copyright (C) 2010-2014 GRNET S.A.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
# Invalid name for type module. pylint: disable-msg=C0103
@@ -7,6 +22,7 @@ Continuous Integration script for Synnefo.
"""
import os
+import sys
import utils
from optparse import OptionParser
@@ -18,8 +34,11 @@ DEPLOY_SYNNEFO_CMD = "deploy"
TEST_SYNNEFO_CMD = "test"
RUN_BURNIN_CMD = "burnin"
CREATE_X2GO_FILE = "x2goplugin"
+SHELL_CONNECT = "shell"
ALL_CMDS = "all"
+DEFAULT_CONFIG_FILE = os.path.expanduser("~/.synnefo_ci")
+
COMMANDS_IN_ALL_MODE = [
CREATE_SERVER_CMD,
BUILD_SYNNEFO_CMD,
@@ -32,6 +51,7 @@ COMMANDS_IN_ALL_MODE = [
AVAILABLE_COMMANDS = [
CREATE_X2GO_FILE,
DELETE_SERVER_CMD,
+ SHELL_CONNECT,
] + COMMANDS_IN_ALL_MODE
USAGE = """usage: %%prog [options] command[,command...]
@@ -45,6 +65,7 @@ command:
* %s: Run snf-burnin in the deployed Synnefo
* %s: Create x2go plugin file
* %s: Delete the slave server
+ * %s: Connect to the server using ssh
* %s: Run all the available commands
""" % tuple([CREATE_SERVER_CMD,
@@ -55,16 +76,25 @@ command:
RUN_BURNIN_CMD,
CREATE_X2GO_FILE,
DELETE_SERVER_CMD,
+ SHELL_CONNECT,
ALL_CMDS])
-def main(): # Too many branches. pylint: disable-msg=R0912
+def main(): # pylint: disable=too-many-statements, too-many-branches
"""Parse command line options and run the specified actions"""
parser = OptionParser(usage=USAGE)
- parser.add_option("-c", "--conf", dest="config_file", default=None,
- help="Configuration file for SynnefoCI script")
+ parser.add_option("-c", "--conf", dest="config_file",
+ default=DEFAULT_CONFIG_FILE,
+ help="Configuration file for SynnefoCI"
+ " script (Default: %s)" % DEFAULT_CONFIG_FILE)
parser.add_option("--cloud", dest="kamaki_cloud", default=None,
help="Use specified cloud, as is in .kamakirc")
+ parser.add_option("--synnefo-repo", dest="synnefo_repo", default=None,
+ help="Specify the synnefo repo to use")
+ parser.add_option("--synnefo-branch", dest="synnefo_branch", default=None,
+ help="Specify the synnefo branch to use")
+ parser.add_option("--pull-request", dest="pull_request", default=None,
+ help="Test a Github pull request.")
parser.add_option("-f", "--flavor", dest="flavor", default=None,
help="Flavor to use for the server."
" Supports both search by name (reg expression)"
@@ -79,7 +109,7 @@ def main(): # Too many branches. pylint: disable-msg=R0912
help="Upload/Install the public ssh keys contained"
" in this file to the server")
parser.add_option("--name", dest="server_name", default=None,
- help=""),
+ help="")
parser.add_option("-n", "--build-id", dest="build_id", default=None,
type="int",
help="Specify a number to use to identify this build."
@@ -108,6 +138,9 @@ def main(): # Too many branches. pylint: disable-msg=R0912
parser.add_option("--no-colors", dest="use_colors",
default=True, action="store_false",
help="Don't use colorful output messages.")
+ parser.add_option("--ignore-ssl", "-k", dest="ignore_ssl",
+ default=None, action="store_true",
+ help="Don't verify SSL certificates.")
(options, args) = parser.parse_args()
@@ -133,14 +166,28 @@ def main(): # Too many branches. pylint: disable-msg=R0912
parser.print_help()
print
print msg
- return
+ sys.exit(1)
else:
setattr(options, command, True)
+ if options.pull_request is not None:
+ if (options.synnefo_repo is not None or
+ options.synnefo_branch is not None):
+ print "ERROR: Options 'synnefo_repo' and/or 'synnefo_branch'" \
+ " cannot be given with 'pull_request'"
+ sys.exit(1)
+
# ----------------------------------
# Initialize SynnefoCi
utils.USE_COLORS = options.use_colors
- synnefo_ci = utils.SynnefoCI(config_file=options.config_file,
+ utils.IGNORE_SSL = options.ignore_ssl
+ config_file = options.config_file
+ if config_file is not None:
+ config_file = os.path.expanduser(config_file)
+ if not os.path.exists(config_file):
+ print "Configuration file '%s' does not exist!" % config_file
+ config_file = None
+ synnefo_ci = utils.SynnefoCI(config_file=config_file,
build_id=options.build_id,
cloud=options.kamaki_cloud)
@@ -151,7 +198,11 @@ def main(): # Too many branches. pylint: disable-msg=R0912
image=options.image,
ssh_keys=options.ssh_keys,
server_name=options.server_name)
- synnefo_ci.clone_repo(local_repo=options.local_repo)
+ synnefo_ci.clone_repo(
+ synnefo_repo=options.synnefo_repo,
+ synnefo_branch=options.synnefo_branch,
+ local_repo=options.local_repo,
+ pull_request=options.pull_request)
if getattr(options, BUILD_SYNNEFO_CMD, False):
synnefo_ci.build_packages()
if options.fetch_packages:
@@ -172,6 +223,8 @@ def main(): # Too many branches. pylint: disable-msg=R0912
synnefo_ci.x2go_plugin(options.x2go_output)
if getattr(options, DELETE_SERVER_CMD, False):
synnefo_ci.destroy_server()
+ if getattr(options, SHELL_CONNECT, False):
+ synnefo_ci.shell_connect()
if __name__ == "__main__":
diff --git a/ci/tests.sh b/ci/tests.sh
index 908746ed71b352a00ce704da0509284870b5ad71..7849911130b4d7b1ef85903e744cc0b1b7094b5d 100755
--- a/ci/tests.sh
+++ b/ci/tests.sh
@@ -1,47 +1,200 @@
#!/bin/sh
set -e
-SNF_MANAGE=$(which snf-manage) ||
- { echo "Cannot find snf-manage in $PATH" 1>&2; exit 1; }
+runAstakosTests () {
+ if [ -z "$astakos_tests" ]; then return; fi
-runTest () {
+ export SYNNEFO_EXCLUDE_PACKAGES="snf-cyclades-app:snf-admin-app:snf-pithos-app"
+ CURRENT_COMPONENT=astakos
+ createSnfManageTest $astakos_tests
+ runTest
+}
+
+runCycladesTests () {
+ if [ -z "$cyclades_tests" ]; then return; fi
+
+ export SYNNEFO_EXCLUDE_PACKAGES="snf-pithos-app:snf-astakos-app:snf-admin-app"
+ CURRENT_COMPONENT=synnefo
+ createSnfManageTest $cyclades_tests
+ runTest
+}
+
+runAdminTests () {
+ if [ -z "$admin_tests" ]; then return; fi
+
+ export SYNNEFO_EXCLUDE_PACKAGES="snf-pithos-app"
+ CURRENT_COMPONENT=synnefo_admin
+ createSnfManageTest $admin_tests
+ runTest
+}
+
+runPithosTests () {
+ if [ -z "$pithos_tests" ]; then return; fi
+
+ export SYNNEFO_EXCLUDE_PACKAGES="snf-cyclades-app:snf-astakos-app:snf-admin-app"
+ CURRENT_COMPONENT=pithos
+ createSnfManageTest $pithos_tests
+ runTest
+}
+
+runAstakosclientTests () {
+ if [ -z "$astakosclient_tests" ]; then return; fi
+
+ CURRENT_COMPONENT=astakosclient
+ for test in $astakosclient_tests; do
+ createNoseTest $test
+ runTest
+ done
+}
+
+createSnfManageTest () {
TEST="$SNF_MANAGE test $* --traceback --noinput --settings=synnefo.settings.test"
+}
- runCoverage "$TEST"
+createNoseTest () {
+ TEST="$NOSE $*"
}
-runCoverage () {
- if coverage >/dev/null 2>&1; then
- coverage run $1
- coverage report --include=snf-*
+runTest () {
+ if [ $COVERAGE_EXISTS ]; then
+ runCoverage "$TEST"
else
- echo "WARNING: Cannot find coverage in path, skipping coverage tests" 1>&2
- $1
+ # Stop here, if we are on dry run
+ if [ $DRY_RUN ]; then
+ echo "$TEST"
+ return
+ fi
+
+ eval $TEST
fi
}
+runCoverage () {
+ # Stop here, if we are on dry run
+ if [ $DRY_RUN ]; then
+ echo "coverage run $1"
+ return
+ fi
+
+ coverage erase
+ coverage run $1
+ coverage report --include="*${CURRENT_COMPONENT}*"
+}
+
+usage(){
+ echo "$1: Wrong input."
+ echo " Usage: $0 [--dry-run] component[.test]"
+ exit
+}
+
+# Append a string to a given variable.
+#
+# Arguments: $1: the variable name
+# $2: the string
+# Note, the variable must be passed by name, so we need to resort to a bit
+# complicated parameter expansions
+append () {
+ eval $(echo "$1=\"\$${1}\"\" \"\"$2\"")
+}
+
+# Check if a string contains a substring
+#
+# Arguments: $1: The string
+# $2: The substring
+contains () {
+ case "$1" in
+ *$2*) return 0;; # True
+ *) return 1;; # False
+ esac
+}
+
+# Get a list of tests for a given component.
+#
+# Arguments: $1: a component to extract tests from or a single component test
+# Returns: $(astakos/cyclades/pithos/ac)_tests,
+# a list with apps to be tested for each component
+extract_tests () {
+ # Check all components:
+ # If the given component matches one of the components:
+ # If total match, return all the tests of the component.
+ # Else, if its form matches "component.test", extract only the
+ # test.
+ # Anything else is considered wrong input
+
+ for c in $ALL_COMPONENTS; do
+ if contains $1 $c; then
+ if [ "$1" = "$c" ]; then
+ append "${c}_tests" "$(eval "echo \$"${c}"_all_tests")"
+ return
+ elif contains $1 "$c."; then
+ append "${c}_tests" $(echo $1 | sed -e 's/^[a-z]*\.//g')
+ return
+ fi
+ fi
+ done
+
+ usage $1
+}
+
export SYNNEFO_SETTINGS_DIR=/tmp/snf-test-settings
-ASTAKOS_APPS="im quotaholder_app oa2"
-CYCLADES_APPS="api db logic plankton quotas vmapi helpdesk userdata"
-PITHOS_APPS="api"
+astakos_all_tests="im quotaholder_app oa2"
+cyclades_all_tests="api db logic plankton quotas vmapi helpdesk userdata volume"
+admin_all_tests="admin"
+pithos_all_tests="api"
+astakosclient_all_tests="astakosclient"
+ALL_COMPONENTS="astakos cyclades admin pithos astakosclient"
+
+astakos_tests=""
+cyclades_tests=""
+admin_tests=""
+pithos_tests=""
+astakosclient_tests=""
+
+if [ "$1" = "--dry-run" ]; then
+ DRY_RUN=0
+ shift
+fi
TEST_COMPONENTS="$@"
if [ -z "$TEST_COMPONENTS" ]; then
- TEST_COMPONENTS="astakos cyclades pithos astakosclient"
+ TEST_COMPONENTS=$ALL_COMPONENTS
+fi
+
+# Check if coverage and snf-manage exist
+if command -v coverage >/dev/null 2>&1; then
+ COVERAGE_EXISTS=0
fi
+SNF_MANAGE=$(which snf-manage) ||
+ { echo "Cannot find snf-manage in $PATH" 1>&2; exit 1; }
+NOSE=$(which nosetests) ||
+ { echo "Cannot find nosetests in $PATH" 1>&2; exit 1; }
+
+# Extract tests from a component
for component in $TEST_COMPONENTS; do
- if [ "$component" = "astakos" ]; then
- runTest $ASTAKOS_APPS
- elif [ "$component" = "cyclades" ]; then
- export SYNNEFO_EXCLUDE_PACKAGES="snf-pithos-app"
- runTest $CYCLADES_APPS
- elif [ "$component" = "pithos" ]; then
- export SYNNEFO_EXCLUDE_PACKAGES="snf-cyclades-app"
- runTest $PITHOS_APPS
- elif [ "$component" = "astakosclient" ]; then
- TEST="nosetests astakosclient"
- runCoverage "$TEST"
- fi
+ extract_tests $component
done
+
+echo "|===============|============================"
+echo "| Component | Tests"
+echo "|---------------|----------------------------"
+echo "| Astakos | $astakos_tests"
+echo "| Cyclades | $cyclades_tests"
+echo "| Admin | $admin_tests"
+echo "| Pithos | $pithos_tests"
+echo "| Astakosclient | $astakosclient_tests"
+echo "|===============|============================"
+echo ""
+
+if [ ! $COVERAGE_EXISTS ]; then
+ echo "WARNING: Cannot find coverage in path." >&2
+ echo ""
+fi
+
+# For each component, run the specified tests.
+runAstakosTests
+runCycladesTests
+runAdminTests
+runPithosTests
+runAstakosclientTests
diff --git a/ci/uninstall.sh b/ci/uninstall.sh
new file mode 100755
index 0000000000000000000000000000000000000000..ee17419fd8b258554f04c1fd27e44ca2ae071f1f
--- /dev/null
+++ b/ci/uninstall.sh
@@ -0,0 +1,15 @@
+#!/bin/sh
+
+# `cd` to the top dir of synnefo repository
+set -e
+cwd=`dirname $0`
+cd "$cwd"/..
+
+# Do common tasks for install/uninstall purposes
+. ./ci/develop-common.sh
+
+for project in $PROJECTS; do
+ cd $project
+ python setup.py develop --uninstall $OPTIONS
+ cd -
+done
diff --git a/ci/utils.py b/ci/utils.py
index 2ee0639ef25412b926ecfd1fdb86a3d591f73ee5..1a9c78983ff4e59480282dedbf5b3bf445b342c0 100644
--- a/ci/utils.py
+++ b/ci/utils.py
@@ -1,4 +1,19 @@
#!/usr/bin/env python
+#
+# Copyright (C) 2010-2014 GRNET S.A.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
"""
Synnefo ci utils module
@@ -8,8 +23,10 @@ import os
import re
import sys
import time
+import httplib
import logging
import fabric.api as fabric
+import simplejson as json
import subprocess
import tempfile
from ConfigParser import ConfigParser, DuplicateSectionError
@@ -19,12 +36,15 @@ from kamaki.clients.astakos import AstakosClient, parse_endpoints
from kamaki.clients.cyclades import CycladesClient, CycladesNetworkClient
from kamaki.clients.image import ImageClient
from kamaki.clients.compute import ComputeClient
+from kamaki.clients.utils import https
from kamaki.clients import ClientError
import filelocker
DEFAULT_CONFIG_FILE = "ci_wheezy.conf"
# Is our terminal a colorful one?
USE_COLORS = True
+# Ignore SSL verification
+IGNORE_SSL = False
# UUID of owner of system images
DEFAULT_SYSTEM_IMAGES_UUID = [
"25ecced9-bf53-4145-91ee-cf47377e9fb2", # production (okeanos.grnet.gr)
@@ -77,6 +97,26 @@ def _check_fabric(fun):
return wrapper
+def _kamaki_ssl(ignore_ssl=None):
+ """Patch kamaki to use the correct CA certificates
+
+ Read kamaki's config file and decide if we are going to use
+ CA certificates and patch kamaki clients accordingly.
+
+ """
+ config = kamaki_config.Config()
+ if ignore_ssl is None:
+ ignore_ssl = config.get("global", "ignore_ssl").lower() == "on"
+ ca_file = config.get("global", "ca_certs")
+
+ if ignore_ssl:
+ # Skip SSL verification
+ https.patch_ignore_ssl()
+ else:
+ # Use ca_certs path found in kamakirc
+ https.patch_with_certs(ca_file)
+
+
def _check_kamaki(fun):
"""Check if kamaki has been initialized"""
def wrapper(self, *args, **kwargs):
@@ -185,6 +225,9 @@ class SynnefoCI(object):
Setup cyclades_client, image_client and compute_client
"""
+ # Patch kamaki for SSL verification
+ _kamaki_ssl(ignore_ssl=IGNORE_SSL)
+
config = kamaki_config.Config()
if self.kamaki_cloud is None:
try:
@@ -198,7 +241,7 @@ class SynnefoCI(object):
auth_url = config.get_cloud(self.kamaki_cloud, "url")
self.logger.debug("Authentication URL is %s" % _green(auth_url))
token = config.get_cloud(self.kamaki_cloud, "token")
- #self.logger.debug("Token is %s" % _green(token))
+ # self.logger.debug("Token is %s" % _green(token))
self.astakos_client = AstakosClient(auth_url, token)
endpoints = self.astakos_client.authenticate()
@@ -262,6 +305,12 @@ class SynnefoCI(object):
fip['floating_ip_address'])
self.network_client.delete_floatingip(fip['id'])
+ # pylint: disable= no-self-use
+ @_check_fabric
+ def shell_connect(self):
+ """Open shell to remote server"""
+ fabric.open_shell("export TERM=xterm")
+
def _create_floating_ip(self):
"""Create a new floating ip"""
networks = self.network_client.list_networks(detail=True)
@@ -272,12 +321,12 @@ class SynnefoCI(object):
try:
fip = self.network_client.create_floatingip(pub_net['id'])
except ClientError as err:
- self.logger.warning("%s: %s", err.message, err.details)
+ self.logger.warning("%s", str(err.message).strip())
continue
self.logger.debug("Floating IP %s with id %s created",
fip['floating_ip_address'], fip['id'])
return fip
- self.logger.error("No mor IP addresses available")
+ self.logger.error("No more IP addresses available")
sys.exit(1)
def _create_port(self, floating_ip):
@@ -322,7 +371,10 @@ class SynnefoCI(object):
server_id = server['id']
self.write_temp_config('server_id', server_id)
self.logger.debug("Server got id %s" % _green(server_id))
- server_user = server['metadata']['users']
+
+ # An image may have more than one user. Choose the first one.
+ server_user = server['metadata']['users'].split(" ")[0]
+
self.write_temp_config('server_user', server_user)
self.logger.debug("Server's admin user is %s" % _green(server_user))
server_passwd = server['adminPass']
@@ -349,16 +401,19 @@ class SynnefoCI(object):
_run(cmd, False)
# Setup apt, download packages
- self.logger.debug("Setup apt. Install x2goserver and firefox")
+ self.logger.debug("Setup apt")
cmd = """
echo 'APT::Install-Suggests "false";' >> /etc/apt/apt.conf
echo 'precedence ::ffff:0:0/96 100' >> /etc/gai.conf
apt-get update
- apt-get install curl --yes --force-yes
+ apt-get install -q=2 curl --yes --force-yes
echo -e "\n\n{0}" >> /etc/apt/sources.list
# Synnefo repo's key
curl https://dev.grnet.gr/files/apt-grnetdev.pub | apt-key add -
+ """.format(self.config.get('Global', 'apt_repo'))
+ _run(cmd, False)
+ cmd = """
# X2GO Key
apt-key adv --recv-keys --keyserver keys.gnupg.net E1F958385BFE2B6E
apt-get install x2go-keyring --yes --force-yes
@@ -377,9 +432,10 @@ class SynnefoCI(object):
echo 'Encoding=UTF-8' >> /usr/share/applications/xterm.desktop
echo 'Icon=xterm-color_48x48' >> /usr/share/applications/xterm.desktop
echo 'Categories=System;TerminalEmulator;' >> \
- /usr/share/applications/xterm.desktop
- """.format(self.config.get('Global', 'apt_repo'))
- _run(cmd, False)
+ /usr/share/applications/xterm.desktop"""
+ if self.config.get("Global", "setup_x2go") == "True":
+ self.logger.debug("Install x2goserver and firefox")
+ _run(cmd, False)
def _find_flavor(self, flavor=None):
"""Find a suitable flavor to use
@@ -502,14 +558,16 @@ class SynnefoCI(object):
# Use the first network as IPv4
server_ip = networks[0]['ipv4']
- if (".okeanos.io" in self.cyclades_client.base_url or
- ".demo.synnefo.org" in self.cyclades_client.base_url):
- tmp1 = int(server_ip.split(".")[2])
- tmp2 = int(server_ip.split(".")[3])
- server_ip = "gate.okeanos.io"
- server_port = 10000 + tmp1 * 256 + tmp2
- else:
- server_port = 22
+ # Check if config has ssh_port option and if so, use that port.
+ server_port = self.config.get("Deployment", "ssh_port")
+ if not server_port:
+ # No ssh port given. Get it from API (SNF:port_forwarding)
+ if '22' in server['SNF:port_forwarding']:
+ server_ip = server['SNF:port_forwarding']['22']['host']
+ server_port = int(server['SNF:port_forwarding']['22']['port'])
+ else:
+ server_port = 22
+
self.write_temp_config('server_ip', server_ip)
self.logger.debug("Server's IPv4 is %s" % _green(server_ip))
self.write_temp_config('server_port', server_port)
@@ -646,7 +704,8 @@ class SynnefoCI(object):
sys.exit(1)
@_check_fabric
- def clone_repo(self, local_repo=False):
+ def clone_repo(self, synnefo_repo=None, synnefo_branch=None,
+ local_repo=False, pull_request=None):
"""Clone Synnefo repo from slave server"""
self.logger.info("Configure repositories on remote server..")
self.logger.debug("Install/Setup git")
@@ -659,16 +718,65 @@ class SynnefoCI(object):
_run(cmd, False)
# Clone synnefo_repo
- synnefo_branch = self.clone_synnefo_repo(local_repo=local_repo)
+ synnefo_branch = self.clone_synnefo_repo(
+ synnefo_repo=synnefo_repo, synnefo_branch=synnefo_branch,
+ local_repo=local_repo, pull_request=pull_request)
# Clone pithos-web-client
- self.clone_pithos_webclient_repo(synnefo_branch)
+ if self.config.get("Global", "build_pithos_webclient") == "True":
+ # Clone pithos-web-client
+ self.clone_pithos_webclient_repo(synnefo_branch)
@_check_fabric
- def clone_synnefo_repo(self, local_repo=False):
+ def clone_synnefo_repo(self, synnefo_repo=None, synnefo_branch=None,
+ local_repo=False, pull_request=None):
"""Clone Synnefo repo to remote server"""
+
+ assert (pull_request is None or
+ (synnefo_branch is None and synnefo_repo is None))
+
+ pull_repo = None
+ if pull_request is not None:
+ # Get a Github pull request and run the testsuite in
+ # a sophisticated way.
+ # Sophisticated means that it will not just check the remote branch
+ # from which the pull request originated. Instead it will checkout
+ # the branch for which the pull request is indented (e.g.
+ # grnet:develop) and apply the pull request over it. This way it
+ # checks the pull request against the branch this pull request
+ # targets.
+ m = re.search("github.com/([^/]+)/([^/]+)/pull/(\d+)",
+ pull_request)
+ if m is None:
+ self.logger.error("Couldn't find a valid GitHub pull request"
+ " URL")
+ sys.exit(1)
+
+ group = m.group(1)
+ repo = m.group(2)
+ pull_number = m.group(3)
+
+ # Construct api url
+ api_url = "/repos/%s/%s/pulls/%s" % \
+ (group, repo, pull_number)
+ headers = {'User-Agent': "snf-ci"}
+ # Get pull request info
+ try:
+ conn = httplib.HTTPSConnection("api.github.com")
+ conn.request("GET", api_url, headers=headers)
+ response = conn.getresponse()
+ payload = json.load(response)
+ synnefo_repo = payload['base']['repo']['html_url']
+ synnefo_branch = payload['base']['ref']
+ pull_repo = (payload['head']['repo']['html_url'],
+ payload['head']['ref'])
+ finally:
+ conn.close()
+
# Find synnefo_repo and synnefo_branch to use
- synnefo_repo = self.config.get('Global', 'synnefo_repo')
- synnefo_branch = self.config.get("Global", "synnefo_branch")
+ if synnefo_repo is None:
+ synnefo_repo = self.config.get('Global', 'synnefo_repo')
+ if synnefo_branch is None:
+ synnefo_branch = self.config.get("Global", "synnefo_branch")
if synnefo_branch == "":
synnefo_branch = \
subprocess.Popen(
@@ -709,7 +817,7 @@ class SynnefoCI(object):
else:
# Clone Synnefo from remote repo
self.logger.debug("Clone synnefo from %s" % synnefo_repo)
- self._git_clone(synnefo_repo)
+ self._git_clone(synnefo_repo, directory="synnefo")
# Checkout the desired synnefo_branch
self.logger.debug("Checkout \"%s\" branch/commit" % synnefo_branch)
@@ -722,6 +830,16 @@ class SynnefoCI(object):
""" % (synnefo_branch)
_run(cmd, False)
+ # Apply a Github pull request
+ if pull_repo is not None:
+ self.logger.debug("Apply patches from pull request %s",
+ pull_number)
+ cmd = """
+ cd synnefo
+ git pull --no-edit --no-rebase {0} {1}
+ """.format(pull_repo[0], pull_repo[1])
+ _run(cmd, False)
+
return synnefo_branch
@_check_fabric
@@ -736,7 +854,7 @@ class SynnefoCI(object):
# Clone pithos-webclient from remote repo
self.logger.debug("Clone pithos-webclient from %s" %
pithos_webclient_repo)
- self._git_clone(pithos_webclient_repo)
+ self._git_clone(pithos_webclient_repo, directory="pithos-web-client")
# Track all pithos-webclient branches
cmd = """
@@ -781,7 +899,7 @@ class SynnefoCI(object):
""".format(pithos_webclient_branch)
_run(cmd, False)
- def _git_clone(self, repo):
+ def _git_clone(self, repo, directory=""):
"""Clone repo to remote server
Currently clonning from code.grnet.gr can fail unexpectedly.
@@ -791,7 +909,7 @@ class SynnefoCI(object):
cloned = False
for i in range(1, 11):
try:
- _run("git clone %s" % repo, False)
+ _run("git clone %s %s" % (repo, directory), False)
cloned = True
break
except BaseException:
@@ -824,7 +942,8 @@ class SynnefoCI(object):
# Build synnefo packages
self.build_synnefo()
# Build pithos-web-client packages
- self.build_pithos_webclient()
+ if self.config.get("Global", "build_pithos_webclient") == "True":
+ self.build_pithos_webclient()
@_check_fabric
def build_synnefo(self):
@@ -842,6 +961,7 @@ class SynnefoCI(object):
cmd = """
dpkg -i snf-deploy*.deb
apt-get -f install --yes --force-yes
+ snf-deploy keygen
"""
with fabric.cd("synnefo_build-area"):
with fabric.settings(warn_only=True):
@@ -901,12 +1021,17 @@ class SynnefoCI(object):
schema = self.config.get('Global', 'schema')
self.logger.debug("Will use \"%s\" schema" % _green(schema))
- schema_dir = os.path.join(self.ci_dir, "schemas/%s" % schema)
- if not (os.path.exists(schema_dir) and os.path.isdir(schema_dir)):
- raise ValueError("Unknown schema: %s" % schema)
-
- self.logger.debug("Upload schema files to server")
- _put(os.path.join(schema_dir, "*"), "/etc/snf-deploy/")
+ self.logger.debug("Update schema files to server")
+ cmd = """
+ schema_dir="synnefo/ci/schemas/{0}"
+ if [ -d "$schema_dir" ]; then
+ cp "$schema_dir"/* /etc/snf-deploy/
+ else
+ echo "$schema_dir" does not exist
+ exit 1
+ fi
+ """.format(schema)
+ _run(cmd, False)
self.logger.debug("Change password in nodes.conf file")
cmd = """
@@ -916,8 +1041,7 @@ class SynnefoCI(object):
self.logger.debug("Run snf-deploy")
cmd = """
- snf-deploy keygen --force
- snf-deploy --disable-colors --autoconf all
+ snf-deploy --disable-colors --autoconf synnefo
"""
_run(cmd, True)
@@ -929,9 +1053,7 @@ class SynnefoCI(object):
self.logger.debug("Install needed packages")
cmd = """
- pip install -U mock
- pip install -U factory_boy
- pip install -U nose
+ pip install -U mock factory_boy nose coverage
"""
_run(cmd, False)
diff --git a/contrib/migrate-data b/contrib/migrate-data
index 46171ad4d7e0481daa1a40f3a6226555cd7eef7e..f07c4bba257c1e0a7243d13ae355425f756f515b 100755
--- a/contrib/migrate-data
+++ b/contrib/migrate-data
@@ -1,37 +1,19 @@
#!/usr/bin/env python
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
-#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
-#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# Copyright (C) 2010-2014 GRNET S.A.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
from binascii import hexlify
diff --git a/contrib/migrate-db b/contrib/migrate-db
index f9a37aade6af1b82fcc0dbabc6491560ccc2e059..adf7433f4a06ad1599860c947b8505bb153c43d5 100755
--- a/contrib/migrate-db
+++ b/contrib/migrate-db
@@ -1,37 +1,19 @@
#!/usr/bin/env python
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
-#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
-#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# Copyright (C) 2010-2014 GRNET S.A.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
from sqlalchemy import Table
from sqlalchemy.sql import select, and_
diff --git a/contrib/migrate-users b/contrib/migrate-users
index 4a0b2370ebd76b2a0b0a5fe7acf10908556d4edb..5a01606d568423c5de0cc0ecebe19498b1fe5e64 100755
--- a/contrib/migrate-users
+++ b/contrib/migrate-users
@@ -1,37 +1,19 @@
#!/usr/bin/env python
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
-#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
-#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# Copyright (C) 2010-2014 GRNET S.A.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
from sqlalchemy import Table
from sqlalchemy.sql import select
diff --git a/contrib/migrate.py b/contrib/migrate.py
index ded082c5f27ef2775bf4cbe3252737c341ae57d0..452e8272e109a58b03ac5738e2c2408f0dde0de0 100644
--- a/contrib/migrate.py
+++ b/contrib/migrate.py
@@ -1,37 +1,19 @@
#!/usr/bin/env python
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
from sqlalchemy import create_engine
from sqlalchemy import Table, Column, String, MetaData
diff --git a/contrib/snf-pithos-tools/README b/contrib/snf-pithos-tools/README
index 560f527fc683eb45f0b110711075d89a935d2d69..b5e063b39908da9823f0a8fe6648052f87bc9f22 100644
--- a/contrib/snf-pithos-tools/README
+++ b/contrib/snf-pithos-tools/README
@@ -2,7 +2,7 @@ README
======
Pithos is a file storage service, built by GRNET using Django (https://www.djangoproject.com/).
-Learn more about Pithos at: http://code.grnet.gr/projects/pithos
+Learn more about Pithos at: https://www.synnefo.org/docs/synnefo/latest/pithos.html
Consult COPYRIGHT for licensing information.
diff --git a/contrib/snf-pithos-tools/pithos/__init__.py b/contrib/snf-pithos-tools/pithos/__init__.py
index d3e1674a2f8a9a764550afa47d9f6693947c73ee..5cda842903d0bfec117db1c3c5a43281d46eb551 100644
--- a/contrib/snf-pithos-tools/pithos/__init__.py
+++ b/contrib/snf-pithos-tools/pithos/__init__.py
@@ -1,35 +1,17 @@
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
# this is a namespace package
try:
diff --git a/contrib/snf-pithos-tools/pithos/tools/dispatcher.py b/contrib/snf-pithos-tools/pithos/tools/dispatcher.py
index acaa72333b3d24997e1492d8ef829d413eb44b4f..d818fafa28a82da091d7ad4cda46429062bb00ec 100755
--- a/contrib/snf-pithos-tools/pithos/tools/dispatcher.py
+++ b/contrib/snf-pithos-tools/pithos/tools/dispatcher.py
@@ -1,37 +1,19 @@
#!/usr/bin/env python
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
import sys
import logging
diff --git a/contrib/snf-pithos-tools/pithos/tools/fs.py b/contrib/snf-pithos-tools/pithos/tools/fs.py
index f8b860e505b4e91162fd6c8e2e309afa373dd1cc..18b01d3dc45afc8538a609a8c74fd28e1ddafaed 100755
--- a/contrib/snf-pithos-tools/pithos/tools/fs.py
+++ b/contrib/snf-pithos-tools/pithos/tools/fs.py
@@ -1,37 +1,19 @@
#!/usr/bin/env python
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
from cStringIO import StringIO
from errno import (EACCES, EBADF, EINVAL, EISDIR, EIO, ENOENT, ENOTDIR,
diff --git a/contrib/snf-pithos-tools/pithos/tools/lib/client.py b/contrib/snf-pithos-tools/pithos/tools/lib/client.py
index c0bb9e39fe87742b3d787da566f017af7c1a5eac..c85cb346249839db06a8867c023b7ee5b3d8365a 100644
--- a/contrib/snf-pithos-tools/pithos/tools/lib/client.py
+++ b/contrib/snf-pithos-tools/pithos/tools/lib/client.py
@@ -1,35 +1,17 @@
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
from httplib import HTTPConnection, HTTPSConnection, HTTP
from sys import stdin
diff --git a/contrib/snf-pithos-tools/pithos/tools/lib/hashmap.py b/contrib/snf-pithos-tools/pithos/tools/lib/hashmap.py
index 977d8fcdf76765c813624a9a03a7de3cc5903bfb..78ecd683d75e5982d6fccbd589fa7a44207d30b8 100644
--- a/contrib/snf-pithos-tools/pithos/tools/lib/hashmap.py
+++ b/contrib/snf-pithos-tools/pithos/tools/lib/hashmap.py
@@ -1,35 +1,17 @@
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
import hashlib
import os
diff --git a/contrib/snf-pithos-tools/pithos/tools/lib/transfer.py b/contrib/snf-pithos-tools/pithos/tools/lib/transfer.py
index 857af9f68f76342b5fe7cb8ef5820f0f74f8c45c..74a050d74b983842d7db3391902bee2f2724c4bd 100644
--- a/contrib/snf-pithos-tools/pithos/tools/lib/transfer.py
+++ b/contrib/snf-pithos-tools/pithos/tools/lib/transfer.py
@@ -1,35 +1,17 @@
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
import os
import types
diff --git a/contrib/snf-pithos-tools/pithos/tools/lib/util.py b/contrib/snf-pithos-tools/pithos/tools/lib/util.py
index 77685c5ad1f71a550e74257b38038d11ac6143a5..d22dbdc1d318f3f63156f4c178fb9538e7605d26 100644
--- a/contrib/snf-pithos-tools/pithos/tools/lib/util.py
+++ b/contrib/snf-pithos-tools/pithos/tools/lib/util.py
@@ -1,35 +1,17 @@
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
import os
diff --git a/contrib/snf-pithos-tools/pithos/tools/sh.py b/contrib/snf-pithos-tools/pithos/tools/sh.py
index eb279841445522e55b63ad16c4f8ee5855191197..329871ce970d0c10d95f79747b858cbc02c93470 100755
--- a/contrib/snf-pithos-tools/pithos/tools/sh.py
+++ b/contrib/snf-pithos-tools/pithos/tools/sh.py
@@ -1,37 +1,19 @@
#!/usr/bin/env python
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
from getpass import getuser
from optparse import OptionParser
diff --git a/contrib/snf-pithos-tools/pithos/tools/sync.py b/contrib/snf-pithos-tools/pithos/tools/sync.py
index 7ca4b2351386d8afe2c4c083193c80c83f6d79e6..f5eabc006838e8c2af0658c06b570d64b8973e08 100755
--- a/contrib/snf-pithos-tools/pithos/tools/sync.py
+++ b/contrib/snf-pithos-tools/pithos/tools/sync.py
@@ -1,37 +1,19 @@
#!/usr/bin/env python
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
import os
import sqlite3
diff --git a/contrib/snf-pithos-tools/pithos/tools/test.py b/contrib/snf-pithos-tools/pithos/tools/test.py
index 1a50d0588670bda4fde68b5d72101b8455225565..c2d9b216d987423184cdbab67c5a4d28049f3acb 100755
--- a/contrib/snf-pithos-tools/pithos/tools/test.py
+++ b/contrib/snf-pithos-tools/pithos/tools/test.py
@@ -1,38 +1,20 @@
#!/usr/bin/env python
#coding=utf8
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
from pithos.tools.lib.client import Pithos_Client, Fault
from pithos.tools.lib.util import get_user, get_auth, get_url
diff --git a/contrib/snf-pithos-tools/setup.py b/contrib/snf-pithos-tools/setup.py
index 87a844f77430e421e3c4f3a4895f2804f43df69e..371ecf40f3bbb3dc4934e177654508ddebaa0c90 100644
--- a/contrib/snf-pithos-tools/setup.py
+++ b/contrib/snf-pithos-tools/setup.py
@@ -1,37 +1,19 @@
#!/usr/bin/env python
-# Copyright 2011-2012 GRNET S.A. All rights reserved.
+# Copyright (C) 2010-2014 GRNET S.A.
#
-# Redistribution and use in source and binary forms, with or
-# without modification, are permitted provided that the following
-# conditions are met:
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
#
-# 1. Redistributions of source code must retain the above
-# copyright notice, this list of conditions and the following
-# disclaimer.
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
#
-# 2. Redistributions in binary form must reproduce the above
-# copyright notice, this list of conditions and the following
-# disclaimer in the documentation and/or other materials
-# provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY GRNET S.A. ``AS IS'' AND ANY EXPRESS
-# OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL GRNET S.A OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
-# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
-# AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
-# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
-# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-# POSSIBILITY OF SUCH DAMAGE.
-#
-# The views and conclusions contained in the software and
-# documentation are those of the authors and should not be
-# interpreted as representing official policies, either expressed
-# or implied, of GRNET S.A.
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see .
#
import distribute_setup
@@ -169,7 +151,7 @@ def find_package_data(
setup(
name='snf-pithos-tools',
version=VERSION,
- license='BSD',
+ license='GNU GPLv3',
url='http://www.synnefo.org/',
description=SHORT_DESCRIPTION,
long_description=README + '\n\n' + CHANGES,
diff --git a/devflow.conf b/devflow.conf
index f16d9164393c42c6df15b205902a03c1571fd644..959a2cbe112579af87097be5c476d92a7c85786e 100644
--- a/devflow.conf
+++ b/devflow.conf
@@ -7,6 +7,8 @@
version_file = "snf-astakos-app/astakos/version.py"
[[snf-cyclades-app]]
version_file = "snf-cyclades-app/synnefo/versions/app.py"
+ [[snf-admin-app]]
+ version_file = "snf-admin-app/synnefo_admin/version.py"
[[snf-cyclades-gtools]]
version_file = "snf-cyclades-gtools/synnefo/versions/ganeti.py"
[[snf-pithos-app]]
diff --git a/docs/admin-guide.rst b/docs/admin-guide.rst
index 53c252b73e3e2e3fd43ad80868b8d6f4e8232975..f9996dadfba3f0820ea53dd8fb7ff7e313331de9 100644
--- a/docs/admin-guide.rst
+++ b/docs/admin-guide.rst
@@ -21,6 +21,62 @@ all the interactions between them.
:target: _images/synnefo-arch2.png
+Required system users and groups (synnefo, archipelago)
+=======================================================
+
+Since v0.16, Synnefo requires an Archipelago installation for the Pithos
+backend. Archipelago on the other hand, supports both NFS and RADOS as
+storage backends. This leads us to various components that have specific
+access rights.
+
+Synnefo ships its own configuration files under ``/etc/synnefo``. In
+order those files not to be compromised, they are owned by
+``root:synnefo`` with group read access (mode 640). Since Gunicorn,
+which serves Synnefo by default, needs read access to the configuration
+files and we don't want it to run as root, it must run with group
+``synnefo``.
+
+Cyclades and Pithos talk to Archipelago over some named pipes under
+``/dev/shm/posixfd``. This directory is created by Archipelago, owned by
+the user/group that Archipelago runs as, and at the same time it must be
+accessible by Gunicorn. Therefore we let Gunicorn run as ``synnefo``
+user and Archipelago as ``archipelago:synnefo`` (by default it rus as
+``archipelago:archipelago``). Beware that the ``synnefo`` user and
+group is created by snf-common package.
+
+Archipelago must have a storage backend to physically store blocks, maps
+and locks. This can be either an NFS or a RADOS cluster.
+
+NFS backing store
+-----------------
+In case of NFS, Archipelago must have permissions to write on the
+exported dirs. We choose to have ``/srv/archip`` exported with
+``blocks``, ``maps``, and ``locks`` subdirectories. They are owned by
+``archipelago:synnefo`` and have ``g+ws`` access permissions. So
+Archipelago will be able to read/write in these directories. We could
+have the whole NFS isolated from Synnefo (owned by
+``archipelago:archipelago`` with ``640`` access permissions) but we
+choose not to (e.g. some future extension could require access to the
+backing store directly from Synnefo).
+
+Due to NFS restrictions, all Archipelago nodes must have common uid for
+the ``archipelago`` user and common gid for the ``synnefo`` group. So
+before any Synnefo installation, we create them here in advance. We
+assume that ids 200 and 300 are available across all nodes.
+
+.. code-block:: console
+
+ # addgroup --system --gid 200 synnefo
+ # adduser --system --uid 200 --gid 200 --no-create-home \
+ --gecos Synnefo synnefo
+
+ # addgroup --system --gid 300 archipelago
+ # adduser --system --uid 300 --gid 300 --no-create-home \
+ --gecos Archipelago archipelago
+
+Normally the ``snf-common`` and ``archipelago`` packages are responsible
+for creating the required system users and groups.
+
Identity Service (Astakos)
==========================
@@ -218,8 +274,8 @@ user in the following cases:
:ref:`authentication methods policies `).
If all of the above fail to trigger automatic activation, an email is sent to
-the persons listed in ``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings,
-notifing that there is a new user pending for moderation and that it's up to
+the persons listed in ``ACCOUNT_NOTIFICATIONS_RECIPIENTS`` setting,
+notifying that there is a new user pending for moderation and that it's up to
the administrator to decide if the user should be activated. The UI also shows
a corresponding 'pending moderation' message to the user. The administrator can
activate a user using the ``snf-manage user-modify`` command:
@@ -234,7 +290,7 @@ activate a user using the ``snf-manage user-modify`` command:
Once the activation process finishes, a greeting message is sent to the user
email address and a notification for the activation to the persons listed in
-``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings. Once activated the user is
+``ACCOUNT_NOTIFICATIONS_RECIPIENTS`` setting. Once activated the user is
able to login and access the Synnefo services.
Additional authentication methods
@@ -312,61 +368,60 @@ Upon success, the system renews the token (if it has expired), logins the user
and sets the cookie, before redirecting the user to the ``next`` parameter
value.
-Setting quota limits
-~~~~~~~~~~~~~~~~~~~~
-
-Set default quota
-`````````````````
-To inspect current default base quota limits, run::
-
- # snf-manage resource-list
-
-You can modify the default base quota limit for all future users with::
-
- # snf-manage resource-modify --default-quota
-
-Set base quota for individual users
-```````````````````````````````````
+Projects and quota
+~~~~~~~~~~~~~~~~~~
-For individual users that need different quota than the default
-you can set it for each resource like this::
+Synnefo supports granting resources and controling their quota through the
+mechanism of *projects*. A project is considered as a pool of finite
+resources. Every actual resources allocated by a user (e.g. a Cyclades VM or
+a Pithos container) is also assigned to a project where the user is a
+member to. For each resource a project specifies the maximum amount that can
+be assigned to it and the maximum amount that a single member can assign to it.
- # use this to display quota / uuid
- # snf-manage user-show 'uuid or email' --quota
+Default quota
+`````````````
- # snf-manage user-modify --base-quota 'cyclades.vm' 10
+Upon user creation, a special purpose user-specific project is automatically
+created in order to hold the quota provided by the system. These *system*
+projects are identified with the same UUID as the user.
-You can set base quota for all existing users, with possible exceptions, using::
+To inspect the quota that future users will receive by default through their
+base projects, check column ``system_default`` in::
- # snf-manage user-modify --all --base-quota cyclades.vm 10 --exclude uuid1,uuid2
+ # snf-manage resource-list
-All quota for which values different from the default have been set,
-can be listed with::
+You can modify the default system quota limit for all future users with::
- # snf-manage quota-list --with-custom=True
+ # snf-manage resource-modify --system-default
+You can also control the default quota a new project offers to its members
+if a limit is not specified in the project application (`project default`).
+In particular, if a resource is not meant to be visible to the end user,
+then it's best to set its project default to infinite.
-Enable the Projects feature
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. code-block:: console
-If you want to enable the projects feature so that users may apply
-on their own for resources by creating and joining projects,
-in ``20-snf-astakos-app-settings.conf`` set::
+ # snf-manage resource-modify cyclades.total_ram --project-default inf
- # this will make the 'projects' page visible in the dashboard
- ASTAKOS_PROJECTS_VISIBLE = True
-You can change the maximum allowed number of pending project applications
-per user with::
+Grant extra quota through projects
+``````````````````````````````````
- # snf-manage resource-modify astakos.pending_app --default-quota
+A user can apply for a new project through the web interface or the API.
+Once it is approved by the administrators, the applicant can join the
+project and let other users in too.
-You can also set a user-specific limit with::
+A project member can make use of the quota granted by the project by
+specifying this particular project when creating a new quotable entity.
- # snf-manage user-modify --base-quota 'astakos.pending_app' 5
+Note that quota are not accumulative: in order to allocate a 100GB disk,
+one must be in a project that grants at least 100GB; it is not possible to
+add up quota from different projects. Note also that if allocating an entity
+requires multiple resources (e.g. cpu and ram for a Cyclades VM) these must
+be all assigned to a single project.
-When users apply for projects they are not automatically granted
-the resources. They must first be approved by the administrator.
+Control projects
+````````````````
To list pending project applications in astakos::
@@ -381,13 +436,39 @@ To deny an application::
# snf-manage project-control --deny
-Users designated as *project admins* can approve, deny, or modify
+Before taking an action, on can inspect project status, settings and quota
+limits with::
+
+ # snf-manage project-show
+
+For an initialized project, option ``--quota`` also reports the resource
+usage.
+
+Users designated as *project admins* can approve or deny
an application through the web interface. In
``20-snf-astakos-app-settings.conf`` set::
# UUIDs of users that can approve or deny project applications from the web.
ASTAKOS_PROJECT_ADMINS = [, ...]
+Set quota limits
+````````````````
+
+One can change the quota limits of an initialized project with::
+
+ # snf-manage project-modify --limit
+
+One can set system quota for all accepted users (that is, set limits for system
+projects), with possible exceptions, with::
+
+ # snf-manage project-modify --all-system-projects --exclude , --limit ...
+
+Quota for a given resource are reported for all projects that the user is
+member in with::
+
+ # snf-manage user-show --quota
+
+With option ``--projects``, owned projects and memberships are also reported.
Astakos advanced operations
---------------------------
@@ -635,27 +716,24 @@ Enabling this feature consists of the following steps:
.. _select_pithos_storage:
-Select Pithos storage backend
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Pithos storage backend
+~~~~~~~~~~~~~~~~~~~~~~
+
+Starting from Synnefo version 0.16, we introduce Archipelago as the new storage
+backend. Archipelago will act as a storage abstraction layer between Pithos and
+NFS, RADOS or any other storage backend driver that Archipelago supports. For
+more information about backend drivers please check Archipelago documentation.
-Starting from Synnefo 0.15.1 we introduce the ability to select or change the
-storage backend. If you have already enabled and configured RADOS as your
-secondary storage solution you can now explicitly select your storage
-backend being only RADOS.
+Since this version care must be taken when restarting Archipelago on a Pithos
+worker node. Pithos acts as an Archipelago peer and must be stopped first
+before trying to restart Archipelago for any reason.
-A new variable has been introduced called PITHOS_BACKEND_STORAGE with
-possible values 'nfs' and 'rados', default value is 'nfs'.
-For those users that need to migrate from NFS to RADOS and have not enabled the
-dual mode of operation from the beginning of their installation, you can
-use a synchronization script that is provided in order to synchronize the data
-from NFS to Rados. The script can be found at
-`/usr/lib/pithos/tools/pithos-sync-rados.sh`.
+If you need to restart Archipelago on a running Pithos worker follow the
+procedure below::
-Since this version the dual mode of operation is not supported any more,
-meaning you will not be able to keep double Pithos objects anymore in NFS and
-RADOS.
-After installing v0.15.1 you will have to choose between the storage backend
-you want to use.
+ pithos-host$ /etc/init.d/gunicorn stop
+ pithos-host$ /etc/init.d/archipelago restart
+ pithos-host$ /etc/init.d/gunicorn start
Compute/Network/Image Service (Cyclades)
@@ -688,8 +766,8 @@ to Cyclades.
Working with Cyclades
---------------------
-Flavors
-~~~~~~~
+Flavors and Volume Types
+~~~~~~~~~~~~~~~~~~~~~~~~
When creating a VM, the user must specify the `flavor` of the virtual server.
Flavors are the virtual hardware templates, and provide a description about
@@ -701,8 +779,9 @@ Flavors are created by the administrator and the user can select one of the
available flavors. After VM creation, the user can resize his VM, by
adding/removing CPU and RAM.
-Cyclades support different storage backends that are described by the disk
-template of the flavor, which is mapped to Ganeti's instance `disk template`.
+Cyclades support different storage backends that are described by the `volume
+type` of the flavor. Each volume type contains a `disk template` attribute
+which is mapped to Ganeti's instance `disk template`.
Currently the available disk templates are the following:
* `file`: regulars file
@@ -715,16 +794,24 @@ Currently the available disk templates are the following:
- `ext_archipelago`: External shared storage provided by
`Archipelago `_.
+Volume types are created by the administrator using the `snf-manage
+volume-type-create` command and providing the `disk template` and a
+human-friendly name:
+
+.. code-block:: console
+
+ $ snf-manage volume-type-create --disk-template=drbd --name=DRBD
+
Flavors are created by the administrator using `snf-manage flavor-create`
command. The command takes as argument number of CPUs, amount of RAM, the size
-of the disks and the disk templates and create the flavors that belong to the
+of the disks and the volume type IDs and creates the flavors that belong to the
cartesian product of the specified arguments. For example, the following
-command will create two flavors of `40G` disk size with `drbd` disk template,
+command will create two flavors of `40G` disk size of volume type with ID `1`,
`4G` RAM and `2` or `4` CPUs.
.. code-block:: console
- $ snf-manage flavor-create 2,4 4096 40 drbd
+ $ snf-manage flavor-create 2,4 4096 40 1
To see the available flavors, run `snf-manage flavor-list` command. The
administrator can delete a flavor by using `flavor-modify` command:
@@ -804,7 +891,7 @@ delete the Cyclades Images but will leave the Pithos file as is (unregister).
Apart from using `kamaki` to see and hangle the available images, the
administrator can use `snf-manage image-list` and `snf-manage image-show`
-commands to list and inspect the available public images. Also, the `--user-id`
+commands to list and inspect the available public images. Also, the `--user`
option can be used the see the images of a specific user.
Virtual Servers
@@ -819,7 +906,7 @@ others, by a prefix in their names. This prefix is defined in
Apart from handling Cyclades VM at the Ganeti level, the administrator can
also use the `snf-manage server-*` commands. These command cover the most
-common tasks that are relative with VM handling. Below we describe come
+common tasks that are relative with VM handling. Below we describe some
of them, but for more information you can use the `--help` option of all
`snf-manage server-* commands`. These command cover the most
@@ -831,11 +918,11 @@ bypassing automatic VM allocation.
.. code-block:: console
- $ snf-manage server-create --flavor-id=1 --image-id=fc0f6858-f962-42ce-bf9a-1345f89b3d5e \
- --user-id=7cf4d078-67bf-424d-8ff2-8669eb4841ea --backend-id=2 \
+ $ snf-manage server-create --flavor=1 --image=fc0f6858-f962-42ce-bf9a-1345f89b3d5e \
+ --user=7cf4d078-67bf-424d-8ff2-8669eb4841ea --backend-id=2 \
--password='example_passw0rd' --name='test_vm'
-The above commnd will create a new VM for user
+The above command will create a new VM for user
`7cf4d078-67bf-424d-8ff2-8669eb4841ea` in the Ganeti backend with ID 2. By
default this command will issue a Ganeti job to create the VM
(`OP_INSTANCE_CREATE`) and return. As in other commands, the `--wait=True`
@@ -1036,14 +1123,14 @@ better understanding of these commands, refer to their help messages.
Create a virtual private network for user
`7cf4d078-67bf-424d-8ff2-8669eb4841ea` using the `PHYSICAL_VLAN` flavor, which
-means that the network will be uniquely assigned a phsyical VLAN. The network
+means that the network will be uniquely assigned a physical VLAN. The network
is assigned an IPv4 subnet, described by it's CIDR and gateway. Also,
the `--dhcp=True` option is used, to make `nfdhcpd` response to DHCP queries
from VMs.
.. code-block:: console
- $ snf-manage network-create --owner=7cf4d078-67bf-424d-8ff2-8669eb4841ea --name=prv_net-1 \
+ $ snf-manage network-create --user=7cf4d078-67bf-424d-8ff2-8669eb4841ea --name=prv_net-1 \
--subnet=192.168.2.0/24 --gateway=192.168.2.1 --dhcp=True --flavor=PHYSICAL_VLAN
Inspect the state of the network in Cyclades DB and in all the Ganeti backends:
@@ -1059,7 +1146,7 @@ subnet's IPv4 address allocation pool:
$ snf-manage subnet-inspect
-Connect a VM to the created private network. The port will be automatically
+Connect a VM to the created private network. The port will automatically
be assigned an IPv4 address from one of the network's available IPs. This
command will result in sending an `OP_INSTANCE_MODIFY` Ganeti command and
attaching a NIC to the specified Ganeti instance.
@@ -1114,14 +1201,14 @@ from `snf-manage` would look like this:
---------------------------------------------------------------------------------------------
1 Internet None ACTIVE True 10.2.1.0/24 10.2.1.1 False True
- $ snf-manage floating-ip-create --owner=7cf4d078-67bf-424d-8ff2-8669eb4841ea --network=1
+ $ snf-manage floating-ip-create --user=7cf4d078-67bf-424d-8ff2-8669eb4841ea --network=1
$ snf-manage floating-ip-list --user=7cf4d078-67bf-424d-8ff2-8669eb4841ea
id address network user.uuid server
------------------------------------------------------------------------
38 10.2.1.2 1 7cf4d078-67bf-424d-8ff2-8669eb4841ea 42
- $ snf-manage port-create --owner=7cf4d078-67bf-424d-8ff2-8669eb4841ea --network=1 \
+ $ snf-manage port-create --user=7cf4d078-67bf-424d-8ff2-8669eb4841ea --network=1 \
--ipv4-address=10.2.1.2 --floating-ip=38
$ snf-manage port-list --user=7cf4d078-67bf-424d-8ff2-8669eb4841ea
@@ -1234,7 +1321,7 @@ As already mentioned Cyclades use a pool of Bridges that must correspond
to Physical VLAN at the Ganeti level. A bridge from the pool is assigned to
each network of flavor `PHYSICAL_VLAN`. Creation of this pool is done
using `snf-manage pool-create` command. For example the following command
-will create a pool containing the brdiges from `prv1` to `prv21`.
+will create a pool containing the bridges from `prv1` to `prv21`.
.. code-block:: console
@@ -1269,7 +1356,7 @@ externally reserved, to exclude from allocation.
Quotas
~~~~~~
-The andling of quotas for Cyclades resources is powered by Astakos quota
+The handling of quotas for Cyclades resources is powered by Astakos quota
mechanism. During registration of Cyclades service to Astakos, the Cyclades
resources are also imported to Astakos for accounting and presentation.
@@ -1358,7 +1445,7 @@ Cyclades database may differ from the real state of VMs and networks in the
Ganeti backends. The reconciliation process is designed to synchronize the
state of the Cyclades DB with Ganeti. There are two management commands for
reconciling VMs and Networks that will detect stale, orphans and out-of-sync
-VMs and networks. To fix detected inconsistencies, use the `--fix-all`.
+VMs and networks. To fix detected inconsistencies, use the `--fix-all` option.
.. code-block:: console
@@ -1408,6 +1495,71 @@ To fix detected inconsistencies, use the `--fix` option.
$ snf-manage reconcile-pools
$ snf-manage reconcile-pools --fix
+
+.. _admin-guide-vnc:
+
+snf-vncauthproxy configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since ``snf-vncauthproxy-1.6`` and ``snf-cyclades-app-0.16``, it is possible
+to run snf-vncauthproxy on a separate node and have multiple snf-vncauthproxy
+instances / nodes, to serve clients.
+
+The ``CYCLADES_VNCAUTHPROXY_OPTS`` setting has become a list of dictionaries,
+each of which defines one snf-vncauthproxy instance. Each vncauthproxy should
+be properly configured to accept control connections by the Cylades host (via
+the ``--listen-address`` CLI parameter of snf-vncauthproxy) and VNC connections
+from clients (via the ``--proxy-listen-address`` CLI parameter.
+
+For a two-node vncauthproxy setup, the ``CYCLADES_VNCAUTHPROXY_OPTS`` would
+look like:
+
+.. code-block:: console
+
+ CYCLADES_VNCAUTHPROXY_OPTS = [
+ {
+ 'auth_user': 'synnefo',
+ 'auth_password': 'secret_password',
+ 'server_address': 'node1.synnefo.live',
+ 'server_port': 24999,
+ 'enable_ssl': True,
+ 'ca_cert': '/path/to/cacert',
+ 'strict': True,
+ },
+ {
+ 'auth_user': 'synnefo',
+ 'auth_password': 'secret_password',
+ 'server_address': 'node2.synnefo.live',
+ 'server_port': 24999,
+ 'enable_ssl': False,
+ 'ca_cert': '/path/to/cacert',
+ 'strict': True,
+ },
+ ]
+
+The ``server_address`` is the host / IP which Cyclades will use for the control
+connection, in order to set up the forwarding.
+
+The vncauthproxy ``DAEMON_OPTS`` option in ``/etc/default/vncauthproxy`` would
+look like:
+
+.. code-block:: console
+
+ DAEMON_OPTS="--pid-file=$PIDFILE --listen-address=node1.synnefo.live --proxy-listen-address=node1.synnefo.live"
+
+The ``--proxy-listen-address`` is the host / IP which clients (Web browsers /
+VNC clients) will use to connect to snf-vncauthproxy.
+
+In case that snf-vncauthproxy doesn't run on the same node as the Cyclades
+node, it is highly recommended to enable SSL on the control socket, using
+strict verification of the server certificate. The only caveat, for the time
+being, is that the same certificate, provided to snf-vncauthproxy, is used for
+both the control and the client connections. If the control and client host
+(``--listen-address`` and ``--proxy-listen-address`` parameters, respectively)
+differ, you should make sure to generate a certificate covering both (using the
+one as common name / CN, and specifying the other as a subject alternative
+name).
+
.. _admin-guide-stats:
VM stats collecting
@@ -1489,7 +1641,7 @@ host and export the RRD directory to the snf-stats-app node via e.g. NFS.
``GRAPH_PREFIX`` is the directory where collectd stores the resulting
stats graphs. You should create it manually, in case it doesn't exist.
-.. code-block::
+.. code-block:: console
# mkdir /var/cache/snf-stats-app/
# chown www-data:wwwdata /var/cache/snf-stats-app/
@@ -1501,6 +1653,7 @@ directory.
snf-stats-app, based on the ``STATS_BASE_URL`` setting will export the
following URL 'endpoints`:
+
* CPU stats bar: ``STATS_BASE_URL``/v1.0/cpu-bar/
* Network stats bar: ``STATS_BASE_URL``/v1.0/net-bar/
* CPU stats daily graph: ``STATS_BASE_URL``/v1.0/cpu-ts/
@@ -1510,7 +1663,7 @@ following URL 'endpoints`:
You can verify that these endpoints are exported by issuing:
-.. code-block::
+.. code-block:: console
# snf-manage show_urls
@@ -1518,9 +1671,10 @@ snf-cyclades-gtools configuration
`````````````````````````````````
To enable VM stats collecting, you will need to:
- * Install collectd on the every Ganeti (VM-capable) node.
+
+ * Install collectd on every Ganeti (VM-capable) node.
* Enable the Ganeti stats plugin in your collectd configuration. This can be
- achived by either copying the example collectd conf file that comes with
+ achieved by either copying the example collectd conf file that comes with
snf-cyclades-gtools
(``/usr/share/doc/snf-cyclades-gtools/examples/ganeti-stats-collectd.conf``)
or by adding the following line to your existing (or default) collectd
@@ -1568,14 +1722,14 @@ fetch them when needed.
Helpdesk
--------
-Helpdesk application provides the ability to view the virtual servers and
+The Helpdesk application provides the ability to view the virtual servers and
networks of all users, along with the ability to perform some basic actions
like administratively suspending a server. You can perform look-ups by
user UUID or email, by server ID (vm-$id) or by an IPv4 address.
If you want to activate the helpdesk application you can set to `True` the
`HELPDESK_ENABLED` setting. Access to helpdesk views (under
-`$BASE_URL/helpdesk`) is only to allowed to users that belong to Astakos
+`$BASE_URL/helpdesk`) is only allowed to users that belong to Astakos
groups defined in the `HELPDESK_PERMITTED_GROUPS` setting, which by default
contains the `helpdesk` group. For example, to allow
to access helpdesk view, you should run the following command in the Astakos
@@ -1625,6 +1779,128 @@ these messages and properly updates the state of the Cyclades DB. Subsequent
requests to the Cyclades API, will retrieve the updated state from the DB.
+Admin Dashboard (Admin)
+=======================
+
+Introduction
+------------
+
+Admin is the Synnefo component that provides to trusted users the ability to
+manage and view various different Synnefo entities such as users, VMs, projects
+etc. Additionally, it automatically generates charts and statistics using data
+from the Astakos/Cyclades stats.
+
+Access and permissions
+----------------------
+
+The Admin dashboard can be accessed by default from the ``ADMIN_BASE_URL`` URL.
+Since there is no login form, the user must login on Astakos first and then
+visit the above URL. Access will be granted only to users that belong to a
+predefined list of Astakos groups. By default, there are three group categories
+that are mapped 1-to-1 to Astakos groups:
+
+* ADMIN_READONLY_GROUP: 'admin-readonly'
+* ADMIN_HELPDESK_GROUP: 'helpdesk'
+* ADMIN_GROUP: 'admin'
+
+The group categories can be changed using the ``ADMIN_PERMITTED_GROUPS``
+setting. In order to change the Astakos group that a category corresponds to,
+the administrator can specify the group that he/she wants in the
+``ADMIN_READONLY_GROUP``, ``ADMIN_HELPDESK_GROUP`` or ``ADMIN_GROUP`` settings.
+
+Note that while any user that belongs to the ``ADMIN_PERMITTED_GROUPS`` has the
+same access to the administrator dashboard, the actions that are allowed for a
+group may differ. That's because Admin implements a Role-Based Access Control
+(RBAC) policy, which can be changed from the ``ADMIN_RBAC`` setting. By
+default, users in the ``ADMIN_READONLY_GROUP`` cannot perform any actions. On
+the other hand, users in the ``ADMIN_GROUP`` can perform all actions. In the
+middle of the spectrum is the ``ADMIN_HELPDESK_GROUP``, which by default
+performs a small subset of reversible actions.
+
+Seting up Admin
+---------------
+
+Admin is bundled by default with a list of sane settings. The most important
+one, ``ADMIN_ENABLED``, is set to ``True`` and defines whether Admin will be used
+or not.
+
+The administrator simply has to create the necessary Astakos groups and
+add trusted users in them. The following example will create an admin group and
+will add a user in it:
+
+.. code-block:: console
+
+ snf-manage group-add admin
+ snf-manage user-modify --add-group=admin
+
+Finally, the administrator must edit the ``20-snf-admin-app-general.conf``
+settings file, uncomment the ``ADMIN_BASE_URL`` setting and assign the
+appropriate URL to it. In most cases, this URL will be the top-level URL of the
+Admin node, with the optional addition of an extra path (e.g. ``/admin``) in
+order to distinguish it from different components.
+
+That's all that is required for a single-node setup. For a multi-node setup,
+please consult the following section:
+
+Multi-node Setup
+~~~~~~~~~~~~~~~~
+
+Admin by design does not use the Astakos/Cyclades API for any action. Instead,
+it requires direct access to the Astakos/Cyclades database as well as the
+settings of their nodes. As a result, when installing Admin in a node, the
+Astakos and Cyclades packages will also be installed.
+
+In order to disable the Astakos/Cyclades API in the Admin node, the
+administrator can add the following line in ``99-locals.conf`` (you can create
+it if doesn't exist):
+
+.. code-block:: console
+
+ ROOT_URLCONF="synnefo_admin.urls"
+
+Note that the above change does not interfere with the ``ADMIN_BASE_URL``,
+which will be used normally.
+
+Furthermore, if Astakos and Cyclades have separate databases, then they must be
+defined in the ``DATABASES`` setting of ``10-snf-webproject-database.conf``. An
+example setup is the following:
+
+.. code-block:: console
+
+ DATABASES = {
+ 'default': {
+ 'ENGINE': 'django.db.backends.postgresql_psycopg2',
+ 'NAME': 'snf_apps_cyclades',
+ 'HOST': ,
+ <...snip..>
+ }, 'cyclades': {
+ 'ENGINE': 'django.db.backends.postgresql_psycopg2',
+ 'NAME': 'snf_apps_cyclades',
+ 'HOST': ,
+ <...snip..>
+ }, 'astakos': {
+ 'ENGINE': 'django.db.backends.postgresql_psycopg2',
+ 'NAME': 'snf_apps_astakos',
+ 'HOST': ,
+ <...snip..>
+ }
+ }
+
+ DATABASE_ROUTERS = ['snf_django.utils.routers.SynnefoRouter']
+
+You may notice that there are three databases instead of two. That's because
+Django requires that every ``DATABASES`` setting has a *default* database. In
+our case, we suggest that you use as default the Cyclades database. Finally,
+you must not forget to add the ``DATABASE_ROUTERS`` setting in the above
+example that must always be used in multi-db setups.
+
+Disabling Admin
+---------------
+
+The easiest way to disable the Admin Dashboard is to set the ``ADMIN_ENABLED``
+setting to ``False``.
+
+
List of all Synnefo components
==============================
@@ -1638,8 +1914,9 @@ They are also available from our apt repository: ``apt.dev.grnet.gr``
* `snf-pithos-webclient `_
* `snf-cyclades-app `_
* `snf-cyclades-gtools `_
+ * `snf-admin-app `_
* `astakosclient `_
- * `snf-vncauthproxy `_
+ * `snf-vncauthproxy `_
* `snf-image `_
* `snf-image-creator `_
* `snf-occi `_
@@ -1650,10 +1927,10 @@ They are also available from our apt repository: ``apt.dev.grnet.gr``
Synnefo management commands ("snf-manage")
==========================================
-Each Synnefo service, Astakos, Pithos and Cyclades are controlled by the
+Each Synnefo service, Astakos, Pithos and Cyclades is controlled by the
administrator using the "snf-manage" admin tool. This tool is an extension of
the Django command-line management utility. It is run on the host that runs
-each service and provides different types of commands depending the services
+each service and provides different types of commands depending on the services
running on the host. If you are running more than one service on the same host
"snf-manage" adds all the corresponding commands for each service dynamically,
providing a unified admin environment.
@@ -1773,6 +2050,10 @@ network-remove Delete a network
flavor-create Create a new flavor
flavor-list List flavors
flavor-modify Modify a flavor
+volume-type-create Create a new volume type
+volume-type-list List volume types
+volume-type-show Show volume type details
+volume-type-modify Modify a volume type
image-list List images
image-show Show image details
pool-create Create a bridge or mac-prefix pool
@@ -1814,8 +2095,8 @@ Running:
snf-component-register []
-automates the registration of the standard Synnefo components (astakos,
-cyclades, and pithos) in astakos database. It internally uses the script:
+automates the registration of the standard Synnefo components (Astakos,
+Cyclades, and Pithos) in Astakos database. It internally uses the script:
.. code-block:: console
@@ -1842,7 +2123,7 @@ Name Description
============================ ===========================
delete Remove an account from the Pithos DB
export-quota Export account quota in a file
-list List existing/dublicate accounts
+list List existing/duplicate accounts
merge Move an account contents in another account
set-container-quota Set container quota for all or a specific account
============================ ===========================
@@ -1853,7 +2134,7 @@ The "kamaki" API client
To upload, register or modify an image you will need the **kamaki** tool.
Before proceeding make sure that it is configured properly. Verify that
-*image.url*, *file.url*, *user.url* and *token* are set as needed:
+*url* and *token* are set as needed:
.. code-block:: console
@@ -2124,52 +2405,52 @@ description and a link to their content:
* ``snf-astakos-app/astakos/im/templates/im/email.txt``
Base email template. Contains a contact email and a “thank you” message.
- (`Link `_)
+ (`Link `_)
* ``snf-astakos-app/astakos/im/templates/im/activation_email.txt`` Email sent to
user that prompts him/her to click on a link provided to activate the account.
- Extends “email.txt” (`Link `_)
+ Extends “email.txt” (`Link `_)
* ``snf-astakos-app/astakos/im/templates/im/invitation.txt`` Email sent to an
invited user. He/she has to click on a link provided to activate the account.
- Extends “email.txt” (`Link `_)
+ Extends “email.txt” (`Link `_)
* ``snf-astakos-app/astakos/im/templates/im/switch_accounts_email.txt`` Email
sent to user upon his/her request to associate this email address with a
shibboleth account. He/she has to click on a link provided to activate the
- association. Extends “email.txt” (`Link `_)
+ association. Extends “email.txt” (`Link `_)
* ``snf-astakos-app/astakos/im/templates/im/welcome_email.txt`` Email sent to
inform the user that his/ her account has been activated. Extends “email.txt”
- (`Link `_)
+ (`Link `_)
* ``snf-astakos-app/astakos/im/templates/registration/email_change_email.txt``
Email sent to user when he/she has requested new email address assignment. The
user has to click on a link provided to validate this action. Extends
- “email.txt” (`Link `_)
+ “email.txt” (`Link `_)
* ``snf-astakos-app/astakos/im/templates/registration/password_email.txt`` Email
sent for resetting password purpose. The user has to click on a link provided
- to validate this action. Extends “email.txt” (`Link `_)
+ to validate this action. Extends “email.txt” (`Link `_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_approval_notification.txt``
Informs the project owner that his/her project has been approved. Extends
- “email.txt” (`Link `_)
+ “email.txt” (`Link `_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_denial_notification.txt``
Informs the project owner that his/her project application has been denied
- explaining the reasons. Extends “email.txt” (`Link `_)
+ explaining the reasons. Extends “email.txt” (`Link `_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_change_notification.txt``
An email is sent to a user containing information about his project membership
(whether he has been accepted, rejected or removed). Extends “email.txt” (`Link
- `_)
+ `_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_enroll_notification.txt``
Informs a user that he/she has been enrolled to a project. Extends
- “email.txt” (`Link `_)
+ “email.txt” (`Link `_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_leave_request_notification.txt``
An email is sent to the project owner to make him aware of a user having
- requested to leave his project. Extends “email.txt” (`Link `_)
+ requested to leave his project. Extends “email.txt” (`Link `_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_request_notification.txt``
An email is sent to the project owner to make him/her aware of a user having
- requested to join his project. Extends “email.txt” (`Link `_)
+ requested to join his project. Extends “email.txt” (`Link `_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_suspension_notification.txt``
An email is sent to the project owner to make him/her aware of his/her project
- having been suspended. Extends “email.txt” (`Link `_)
+ having been suspended. Extends “email.txt” (`Link `_)
* ``snf-astakos-app/astakos/im/templates/im/projects/project_termination_notification.txt``
An email is sent to the project owner to make him/her aware of his/her project
- having been terminated. Extends “email.txt” (`Link `_)
+ having been terminated. Extends “email.txt” (`Link `_)
.. warning:: Django templates language:
@@ -2260,7 +2541,7 @@ The RabbitMQ nodes that form the cluster, are declared to Synnefo through the
`AMQP_HOSTS` setting. Each time a Synnefo component needs to connect to
RabbitMQ, one of these nodes is chosen in a random way. The client that Synnefo
uses to connect to RabbitMQ, handles connection failures transparently and
-tries to reconnect to a different node. As long as one of these nodes are up
+tries to reconnect to a different node. As long as one of these nodes is up
and running, functionality of Synnefo should not be downgraded by the RabbitMQ
node failures.
@@ -2341,6 +2622,12 @@ The logging configuration dictionary is defined in
The administrator can have logging control by modifying the ``LOGGING_SETUP``
dictionary, and defining subloggers with different handlers and log levels.
+By default snf-manage will log any command that is being executed along with
+its output under the directory ``LOG_DIR``/commands. The ``LOG_DIR`` directory
+can be changed from the ``00-snf-common-admins.conf`` configuration file and
+the whole snf-manage logging mechanism can be disabled by changing the
+``LOGGER_EXCLUDE_COMMANDS`` setting to ".\*".
+
.. _scale-up:
@@ -2540,6 +2827,112 @@ Node10:
All sections: :ref:`Scale out Guide `
+Regions, Zones and Clusters
+===========================
+
+Region
+------
+
+A Region is a single Synnefo installation, with
+Compute/Network/Image/Volume/Object Store services. A Region is associated with
+one set of Synnefo DBs (Astakos DB, Pithos DB and Cyclades DB). Every Region has a
+distinct set of API endpoints, e.g.,
+`https://cloud.example.com/cyclades/compute/v2.0`. Two Regions are most times
+located geographically far from each other, e.g. "Europe", "US-East". A Region
+comprises multiple Zones.
+
+Zone
+----
+
+A Zone is a set of Ganeti clusters, in a potentially geographically distinct
+location, e.g. "Athens", "Rome". All clusters have access to the same physical
+networks, and are considered a single failure domain, e.g., they access the
+network over the same router. A Zone comprises muliple Ganeti clusters.
+
+Ganeti cluster
+--------------
+
+A Ganeti cluster is a set of Ganeti nodes (physical machines). One of the nodes
+has the role of "Ganeti master". If this node goes down, another node may
+undertake the master role. Ganeti nodes run Virtual Machines (VMs). VMs can live
+migrate inside a Ganeti cluster. A Ganeti cluster comprises multiple physical
+hardware nodes, most times geographically close to each other.
+
+VM mobility
+-----------
+
+VMs may move across Regions, Zones, Ganeti clusters and physical nodes. Before we
+describe how that's possible, we will describe the different kinds of moving,
+providing the corresponding terminology:
+
+Live migration
+~~~~~~~~~~~~~~
+
+The act of moving a running VM from physical node to physical node without any
+impact on its operation. The VM continues to run on its new physical location,
+completely unaffected, and without any service downtime or dropped connections.
+Live migration typically requires shared storage and networking between the source
+and destination nodes.
+
+Live migration is issued by the administrator in the background and is transparent
+to the VM user.
+
+Failover
+~~~~~~~~
+
+The act of moving a VM from physical node to physical node by stopping it first on
+the source node, then re-starting it on the destination node. There is short
+service downtime, during the time the VM boots up, and client connections are
+dropped.
+
+Failover is issued by the administrator in the background and the VM user will
+experience a reboot.
+
+Snapshot Failover
+~~~~~~~~~~~~~~~~~
+
+The act of moving a VM from physical node to physical node via a point-in-time
+snapshot. That is, stopping a VM on the source node, taking a snapshot, then
+creating a new VM from that snapshot.
+
+Snapshot failover is issued by the VM user and not the administrator.
+
+Disaster Recovery
+-----------------
+
+In Synnefo terminology, Disaster Recovery is the process of sustaining a disaster
+in one datacenter, and ensuring business continuity by performing live migration
+or failover of running/existing VMs, or respawning VMs from previously made
+snapshots. Based on the method used, this can work inside a single Ganeti cluster,
+across Ganeti clusters in the same Zone, or across Zones.
+
+Specifically:
+
+Live migration is only supported inside a single Ganeti cluster. Ganeti supports
+live migration between nodes in the same cluster with or without shared storage.
+Live migration is done at the Ganeti level and is transparent to Synnefo.
+
+Failover is supported inside a Ganeti cluster, across Ganeti clusters and across
+Zones. Ganeti supports failover inside a Ganeti cluster with or without shared
+storage, which poses minimum downtime for the VM. Failover inside the same Ganeti
+cluster is done at the Ganeti level and is transparent to Synnefo.
+
+Ganeti also provides tools for failing over VMs across different Ganeti clusters,
+meaning that one can use them to failover VMs across Ganeti clusters of the same
+Zone or across Ganeti clusters of different Zones, thus moving across Zones.
+Failing over across different Ganeti clusters requires copying of data, resulting
+in longer downtimes, depending on the geographical distance and network between
+them. Failover across Ganeti clusters, either in the same or different Zones, is
+not transparent to Synnefo and requires manual import of intances at Synnefo level
+too, by the administrator.
+
+Snapshot failover supports moving VMs across all domains. It is issued by the VM
+user and is done at the Synnefo level without the need of running anything at the
+Ganeti level or by the administrator.
+
+In the future Synnefo will also support moving VMs across different Regions.
+
+
Upgrade Notes
=============
@@ -2554,12 +2947,14 @@ Upgrade Notes
v0.14.9 -> v0.14.10
v0.14 -> v0.15
v0.15 -> v0.15.1
+ v0.15 -> v0.16
Changelog, NEWS
===============
+* v0.16 :ref:`Changelog `, :ref:`NEWS `
* v0.15.2 :ref:`Changelog `, :ref:`NEWS `
* v0.15.1 :ref:`Changelog `, :ref:`NEWS `
* v0.15 :ref:`Changelog `, :ref:`NEWS `
diff --git a/docs/api-guide.rst b/docs/api-guide.rst
index b08f064fe9c1d988e668f97b573f137d83fc8f6c..9b69124228a07e1d88c2e66e76ff30fc0bd543e9 100644
--- a/docs/api-guide.rst
+++ b/docs/api-guide.rst
@@ -17,13 +17,14 @@ Most Synnefo services have a corresponding OpenStack API:
| Cyclades/Compute Service -> OpenStack Compute API
| Cyclades/Network Service -> OpenStack Networking ("Neutron") API
| Cyclades/Image Service -> OpenStack Image ("Glance") API
+| Cyclades/Block Storage Service -> OpenStack Block Storage ("Cinder") API
| Pithos/Storage Service -> OpenStack Object Storage API
| Astakos/Identity Service -> OpenStack Identity ("Keystone") API
| Astakos/Quota Service -> Proprietary API
| Astakos/Resource Service -> Proprietary API
| Astakos/Project Service -> Proprietary API
-Below, we will describe all Synnefo APIs with conjuction to the OpenStack APIs.
+Below, we will describe all Synnefo APIs with conjunction to the OpenStack APIs.
Identity Service API (Astakos)
@@ -51,6 +52,18 @@ following Synnefo specific (proprietary) API:
Resource and Quota API
+Weblogin Service API (Astakos)
+==============================
+
+The Weblogin Service is implemented inside Astakos and have the
+following Synnefo API:
+
+.. toctree::
+ :maxdepth: 2
+
+ Weblogin API
+
+
Project Service API
===================
@@ -105,6 +118,20 @@ This is the Cyclades/Image API:
Image API (Glance)
+Block Storage Service API (Cyclades)
+====================================
+
+The Block Storage Service is implemented inside Cyclades. It exposes the
+OpenStack Block Storage ("Cinder") API with minor changes wherever needed.
+
+This is the Cyclades/Block Storage API:
+
+.. toctree::
+ :maxdepth: 2
+
+ Block Storage API (Cinder)
+
+
Storage Service API (Pithos)
============================
diff --git a/docs/archipelago.rst b/docs/archipelago.rst
deleted file mode 100644
index a461fe0bc531902b3a89a0080a1294e9d28516e2..0000000000000000000000000000000000000000
--- a/docs/archipelago.rst
+++ /dev/null
@@ -1,22 +0,0 @@
-.. _archipelago:
-
-Volume Service (archipelago)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Introduction
-============
-
-Every Volume inside a VM can be thought of as a linearly addressable set of
-fixed-size blocks. The storage of the actual blocks is orthogonal to the task of
-exposing a single block device for use by each VM. Bridging the gap between the
-VMs performing random access to Volumes and the storage of actual blocks is
-Archipelago: a custom storage handling layer which handled volumes as set of
-distinct blocks in the backend, a process we call volume composition. For the
-actual storage of blocks we are currently experimenting with RADOS, the
-distributed object store underlying the Ceph parallel filesystem, to solve the
-problem of reliable, fault-tolerant object storage through replication on
-multiple storage nodes. Archipelago itself is agnostic to the actual block
-storage backend.
-
-Archipelago is under active development and will be available soon.
-
diff --git a/docs/astakos.rst b/docs/astakos.rst
index e58c617d99a8edcc56797db7d6ecc96d602e3bae..75c7839c8dc4c8b663a77a66e9ca51b182e2c73f 100644
--- a/docs/astakos.rst
+++ b/docs/astakos.rst
@@ -29,5 +29,5 @@ authentication, along with the Synnefo Account API for quota, user group and
project management.
Please also see the :ref:`Admin Guide ` for more information and the
-Installation Guide (:ref: `Debian `/:ref: `CentOS
+Installation Guide (:ref:`Debian ` / :ref:`CentOS
`) for installation instructions.
diff --git a/docs/blockstorage-api-guide.rst b/docs/blockstorage-api-guide.rst
new file mode 100644
index 0000000000000000000000000000000000000000..a3b62950146f6ad98526bdd52db0d32765836143
--- /dev/null
+++ b/docs/blockstorage-api-guide.rst
@@ -0,0 +1,1351 @@
+.. _blockstorage-api-guide:
+
+API Guide
+*********
+
+:ref:`Cyclades ` include the Block Storage Service of
+`Synnefo `_. The Cyclades/Block Storage API complies
+with
+`OpenStack Block Storage `_
+version 2, with custom extensions when needed.
+
+This document's goals are:
+
+* Define the Cyclades/Block Storage REST API
+* Clarify the differences between Cyclades and OpenStack/Block Storage
+
+Users and developers who wish to access Cyclades through its REST API are
+advised to use the
+`kamaki `_ command-line
+client and the associated python library, instead of making direct calls.
+
+General API Information
+=======================
+
+Authentication
+--------------
+
+All requests use the same authentication method: an ``X-Auth-Token`` header is
+passed to the service, which is used to authenticate the user and retrieve user
+related information. No other user details are passed through HTTP.
+
+
+API Operations
+==============
+
+.. rubric:: Volumes
+
+==================================== =============================== ====== ======== ==========
+Description URI Method Cyclades OS/Block Storage
+==================================== =============================== ====== ======== ==========
+`List <#list-volumes>`_ ``/volumes`` GET ✔ ✔
+\ ``/volumes/detail`` GET ✔ ✔
+`Create <#create-volume>`_ ``/volumes`` POST ✔ ✔
+`Get Details <#get-volume-details>`_ ``/volumes/`` GET ✔ ✔
+`Update <#update-volume>`_ ``/volumes/`` PUT ✔ ✔
+`Delete <#delete-volume>`_ ``/volumes/`` DELETE ✔ ✔
+`Reassign <#reassign-volume>`_ ``/volumes//action`` POST ✔ **✘**
+==================================== =============================== ====== ======== ==========
+
+.. rubric:: Snapshots
+
+====================================== ============================ ====== ======== ==========
+Description URI Method Cyclades OS/Block Storage
+====================================== ============================ ====== ======== ==========
+`List <#list-snapshots>`_ ``/snapshots`` GET ✔ ✔
+\ ``/snapshots/detail`` GET ✔ ✔
+`Create <#create-snapshot>`_ ``/snapshots`` POST ✔ ✔
+`Get Details <#get-snapshot-details>`_ ``/snapshots/`` GET ✔ ✔
+`Update <#update-snapshot>`_ ``/snapshots/`` PUT ✔ ✔
+`Delete <#delete-snapshot>`_ ``/snapshots/`` DELETE ✔ ✔
+====================================== ============================ ====== ======== ==========
+
+.. rubric:: Volume types
+
+========================================= ================================== ====== ======== ==========
+Description URI Method Cyclades OS/Block Storage
+========================================= ================================== ====== ======== ==========
+`List <#list-volume-types>`_ ``/volume-types`` GET ✔ ✔
+`Get Details <#get-volume-type-details>`_ ``/volume-types/`` GET ✔ ✔
+========================================= ================================== ====== ======== ==========
+
+List Volumes
+------------
+
+List all volumes owned by the user.
+
+.. rubric:: Request
+
+=================== ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+=================== ====== ======== ==========
+``/volumes`` GET ✔ ✔
+``/volumes/detail`` GET ✔ ✔
+=================== ====== ======== ==========
+
+* Both requests return a list of volumes. The first returns just ``id``,
+ ``display_name`` and ``links``, while the second returns the
+ `full collection <#volume-ref>`_ of volume attributes
+
+|
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+============== ========================= ======== ==========
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) Request succeeded
+401 (Unauthorized) Missing or expired user token
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) The service is not currently available
+=========================== =====================
+
+|
+
+Response body contents::
+
+ volumes: [
+ {
+ : ,
+ ...
+ }, ...
+ ]
+
+The volume attributes are listed `here <#volume-ref>`_
+
+*Example List Volumes: JSON (regular)*
+
+.. code-block:: javascript
+
+ GET https://example.org/cyclades/v2/volumes
+
+ {
+ "volumes": [
+ {
+ "links": [
+ {
+ "href": "https://example.org/cyclades/v2/volumes/42",
+ "rel": "self"
+ }, {
+ "href": "https://example.org/cyclades/v2/volumes/42",
+ "rel": "bookmark"
+ }
+ ],
+ "id": "42",
+ "display_name": "Volume One",
+ }, {
+ "links": [
+ {
+ "href": "https://example.org/cyclades/v2/volumes/43",
+ "rel": "self"
+ }, {
+ "href": "https://example.org/cyclades/v2/volumes/43",
+ "rel": "bookmark"
+ }
+ ],
+ "id": "43",
+ "display_name": "Volume Two",
+ }
+ ]
+ }
+
+*Example List Volumes: JSON (detail)*
+
+.. code-block:: javascript
+
+ GET https://example.org/cyclades/v2/volumes/detail
+
+ {
+ "volumes": [
+ {
+ "links": [
+ {
+ "href": "https://example.org/cyclades/v2/volumes/42",
+ "rel": "self"
+ }, {
+ "href": "https://example.org/cyclades/v2/volumes/42",
+ "rel": "bookmark"
+ }
+ ],
+ "id": "42",
+ "display_name": "Volume One",
+ "status": "AVAILABLE",
+ "size": 2,
+ "display_description": "The First Volume",
+ "created_at": "2014-02-21T19:52:04.949734",
+ "metadata": {},
+ "snapshot_id": null,
+ "source_volid": null,
+ "image_id": null,
+ "attachments": [],
+ "volume_type": 1,
+ "delete_on_termination": True,
+ "project": "1234"
+ }, {
+ "links": [
+ {
+ "href": "https://example.org/cyclades/v2/volumes/43",
+ "rel": "self"
+ }, {
+ "href": "https://example.org/cyclades/v2/volumes/43",
+ "rel": "bookmark"
+ }
+ ],
+ "id": "43",
+ "display_name": "Volume Two",
+ "status": "AVAILABLE",
+ "size": 3,
+ "display_description": "The Second Volume",
+ "created_at": "2014-03-21T19:52:04.949734",
+ "metadata": {"requested_by": "John"},
+ "snapshot_id": null,
+ "source_volid": null,
+ "image_id": null,
+ "attachments": [],
+ "volume_type": 2,
+ "delete_on_termination": False,
+ "project": "1234"
+ },
+ ]
+ }
+
+Get Volume Details
+------------------
+
+This operation returns detailed information for a volume
+
+.. rubric:: Request
+
+======================== ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+======================== ====== ======== ==========
+``/volumes/`` GET ✔ ✔
+======================== ====== ======== ==========
+
+|
+
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+============== ========================= ======== ==========
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) Request succeeded
+400 (Bad Request) Malformed volume id
+401 (Unauthorized) Missing or expired user token
+404 (Not Found) Volume not found
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) No available backends or service currently
+\ unavailable
+=========================== =====================
+
+|
+
+Response body contents::
+
+ volume: {
+ : ,
+ ...
+ }
+
+Volume attributes are explained `here <#volume-ref>`_
+
+*Example Get Volume Response*
+
+.. code-block:: javascript
+
+ GET https://example.org/cyclades/v2/volumes/44
+
+ {
+ "volume": {
+ "links": [
+ {
+ "href": "https://example.org/cyclades/v2/volumes/44",
+ "rel": "self"
+ }, {
+ "href": "https://example.org/cyclades/v2/volumes/44",
+ "rel": "bookmark"
+ }
+ ],
+ "id": "44",
+ "display_name": "Volume Three",
+ "status": "CREATING",
+ "size": 10,
+ "display_description": null,
+ "created_at": "2014-05-13T19:52:04.949734",
+ "metadata": {},
+ "snapshot_id": null,
+ "source_volid": null,
+ "image_id": null,
+ "attachments": [],
+ "volume_type": 2,
+ "delete_on_termination": False,
+ "project": "1234"
+ }
+ }
+
+Create Volume
+-------------
+
+Create a new volume
+
+.. rubric:: Request
+
+============ ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+============ ====== ======== ==========
+``/volumes`` POST ✔ ✔
+============ ====== ======== ==========
+
+|
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+Content-Type Type or request body required required
+Content-Length Length of request body required required
+============== ========================= ======== ==========
+
+Request body contents::
+
+ volume: {
+ : ,
+ ...
+ }
+
+=================== ================================ ======== ================
+Volume Attribute Value Cyclades OS/Block Storage
+=================== ================================ ======== ================
+size Volume size in GB required required
+server_id An existing VM to create from required **✘**
+availability_zone Respond in xml **✘** ✔
+source_volid Existing volume to create from **✘** ✔
+display_description A description ✔ ✔
+snapshot_id Existing snapshot to create from ✔ ✔
+display_name The name required ✔
+imageRef Image to create from ✔ ✔
+volume_type The associated volume type ✔ ✔
+bootable Whether the volume is bootable **✘** ✔
+metadata Key-Value metadata pairs ✔ ✔
+project Assigned project for quotas ✔ **✘**
+=================== ================================ ======== ================
+
+*Example Create Volume Request: JSON*
+
+.. code-block:: javascript
+
+ POST https://example.org/cyclades/v2/volumes
+
+ {
+ "volume": {
+ "size": 10,
+ "display_name": "Volume Three",
+ "server_id": "117",
+ "volume_type": 1,
+ }
+ }
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+202 (OK) Request succeeded
+400 (Bad Request) Malformed request data
+401 (Unauthorized) Missing or expired user token
+403 (Forbidden) User is not allowed to perform this operation
+404 (Not Found) Resource (server_id, imageRef, etc,) not found
+413 (Over Limit) Exceeded some resource limit
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) No available backends or service currently
+\ unavailable
+=========================== =====================
+
+|
+
+Response body contents::
+
+ volume: {
+ : ,
+ ...
+ }
+
+Volume attributes are `listed here <#server-ref>`_.
+
+*Example Create Volume Response: JSON*
+
+.. code-block:: javascript
+
+ {
+ "volume": {
+ "links": [
+ {
+ "href": "https://example.org/cyclades/v2/volumes/44",
+ "rel": "self"
+ }, {
+ "href": "https://example.org/cyclades/v2/volumes/44",
+ "rel": "bookmark"
+ }
+ ],
+ "id": "44",
+ "display_name": "Volume Three",
+ "status": "CREATING",
+ "size": 10,
+ "display_description": null,
+ "created_at": "2014-05-13T19:52:04.949734",
+ "metadata": {},
+ "snapshot_id": null,
+ "source_volid": null,
+ "image_id": null,
+ "attachments": [],
+ "volume_type": 1,
+ "delete_on_termination": True,
+ "project": "1234"
+ }
+ }
+
+Update Volume
+-------------
+
+.. rubric:: Response
+
+======================== ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+======================== ====== ======== ==========
+``/volumes/`` PUT ✔ ✔
+======================== ====== ======== ==========
+
+|
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+Content-Type Type or request body required required
+Content-Length Length of request body required required
+============== ========================= ======== ==========
+
+Request body contents::
+
+ volume: {
+ : ,
+ ...
+ }
+
+===================== ===================== ======== ==========
+Attribute Description Cyclades OS/Block Storage
+===================== ===================== ======== ==========
+display_name Server name ✔ ✔
+display_description Descrition ✔ ✔
+delete_on_termination Switch this attribute ✔ **✘**
+===================== ===================== ======== ==========
+
+*Example Rename Server Request: JSON*
+
+.. code-block:: javascript
+
+ POST https://example.org/cyclades/v2/volumes/42
+
+ {"volume": {"display_name": "New name"}}
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) Request succeeded
+400 (Bad Request) Malformed request
+401 (Unauthorized) Missing or expired user token
+403 (Forbidden) User is not allowed to perform this operation
+404 (Not Found) Volume not found
+409 (Build In Progress) Volume is not ready yet
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) No available backends or service currently
+\ unavailable
+=========================== =====================
+
+Response body contents::
+
+ volume: {
+ : ,
+ ...
+ }
+
+Volume attributes are explained `here <#volume-ref>`_
+
+*Example update volume Response*
+
+.. code-block:: javascript
+
+ {
+ "volume": {
+ "id": "42",
+ "display_name": "New Name",
+ ...
+ }
+ }
+
+Update Volume Metadata
+----------------------
+
+.. rubric:: Response
+
+================================= ======== ======== ==========
+URI Method Cyclades OS/Block Storage
+================================= ======== ======== ==========
+``/volumes//metadata`` POST/PUT ✔ ✔
+================================= ======== ======== ==========
+
+* POST will create new metadata for the specified Volume if the key doesn't
+ exist, while it will update metadata for which the key already exists.
+* PUT will delete any old existing metadata and it'll replace them with
+ the ones specified in the request.
+
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+Content-Type Type or request body required required
+Content-Length Length of request body required required
+============== ========================= ======== ==========
+
+Request body contents::
+
+ volume: {
+ : ,
+ ...
+ }
+
+*Example Append Metadata Request: JSON*
+
+.. code-block:: javascript
+
+ POST https://example.org/cyclades/v2/volumes/42/metadata
+
+ {"metadata": {"key_to_append": "value_to_append"}}
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) Request succeeded
+400 (Bad Request) Malformed request
+401 (Unauthorized) Missing or expired user token
+403 (Forbidden) User is not allowed to perform this operation
+404 (Not Found) Volume not found
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) No available backends or service currently
+\ unavailable
+=========================== =====================
+
+Response body contents::
+
+ metadata: {
+ : ,
+ ...
+ }
+
+*Example update volume Response*
+
+.. code-block:: javascript
+
+ {
+ "metadata": {
+ "key1": "value1",
+ "key2": "value2",
+ ...
+ }
+ }
+
+Delete Volume
+-------------
+
+.. rubric:: Request
+
+======================== ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+======================== ====== ======== ==========
+``/volumes/`` DELETE ✔ ✔
+======================== ====== ======== ==========
+
+|
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+============== ========================= ======== ==========
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+202 (OK) Request succeeded
+400 (Bad Request) Malformed server id
+401 (Unauthorized) Missing or expired user token
+404 (Not Found) Volume not found
+409 (Build In Progress) Volume is not ready yet
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) Action not supported or service currently
+\ unavailable
+=========================== =====================
+
+Reassign Volume
+---------------
+
+Reassign the volume to a (different) project (change quota limits)
+
+=============================== ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+=============================== ====== ======== ==========
+``/volumes//action`` POST ✔ ✔
+=============================== ====== ======== ==========
+
+|
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+============== ========================= ======== ==========
+
+.. rubric:: Request
+
+Request body contents::
+
+ reassign: {project: }
+
+*Example reassign volume Request*
+
+.. code-block:: javascript
+
+ POST https://example.org//cyclades/v2/volumes/42/action
+
+ {"reassign": {"project": "4321"}}
+
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) Request succeeded
+400 (Bad Request) Malformed request
+401 (Unauthorized) Missing or expired user token
+403 (Forbidden) User is not allowed to perform this operation
+404 (Not Found) Volume not found
+409 (Build In Progress) Volume is not ready yet
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) No available backends or service currently
+\ unavailable
+=========================== =====================
+
+
+List Snapshots
+--------------
+
+List all snapshots related to the user.
+
+.. rubric:: Request
+
+====================== ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+====================== ====== ======== ==========
+``/snapshots`` GET ✔ ✔
+``/snapshots/detail`` GET ✔ ✔
+====================== ====== ======== ==========
+
+* Both requests return a list of snapshots. The first returns just ``id``,
+ ``display_name`` and ``links``, while the second returns the
+ `full collection <#snapshot-ref>`_ of snapshot attributes
+
+|
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+============== ========================= ======== ==========
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) Request succeeded
+401 (Unauthorized) Missing or expired user token
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) The service is not currently available
+=========================== =====================
+
+|
+
+Response body contents::
+
+ snapshots: [
+ {
+ : ,
+ ...
+ }, ...
+ ]
+
+The snapshot attributes are listed `here <#snapshot-ref>`_
+
+*Example List Snapshots: JSON (regular)*
+
+.. code-block:: javascript
+
+ GET https://example.org/cyclades/v2/snapshots
+
+ {
+ "snapshots": [
+ {
+ "links": [
+ {
+ "href": "https://example.org/cyclades/v2/snapshots/42",
+ "rel": "self"
+ }, {
+ "href": "https://example.org/cyclades/v2/snapshots/42",
+ "rel": "bookmark"
+ }
+ ],
+ "id": "42",
+ "display_name": "Snapshot One",
+ "status": "AVAILABLE",
+ "size": 2,
+ "display_description": null,
+ "created_at": "2014-05-19T19:52:04.949734",
+ "metadata": {},
+ "volume_id": "123",
+ "os-extended-snapshot-attribute:progress": "100%"
+ }, {
+ "links": [
+ {
+ "href": "https://example.org/cyclades/v2/snapshots/43",
+ "rel": "self"
+ }, {
+ "href": "https://example.org/cyclades/v2/snapshots/43",
+ "rel": "bookmark"
+ }
+ ],
+ "id": "43",
+ "display_name": "Snapshot Two",
+ "status": "AVAILABLE",
+ "size": 3,
+ "display_description": null,
+ "created_at": "2014-05-20T19:52:04.949734",
+ "metadata": {},
+ "volume_id": "124",
+ "os-extended-snapshot-attribute:progress": "100%"
+ }
+ ]
+ }
+
+*Example List Snapshots: JSON (detail)*
+
+.. code-block:: javascript
+
+ GET https://example.org/cyclades/v2/snapshots/detail
+
+ {
+ "snapshots": [
+ {
+ "links": [
+ {
+ "href": "https://example.org/cyclades/v2/snapshots/42",
+ "rel": "self"
+ }, {
+ "href": "https://example.org/cyclades/v2/snapshots/42",
+ "rel": "bookmark"
+ }
+ ],
+ "id": "42",
+ "display_name": "Snapshot One",
+ "status": "AVAILABLE",
+ "size": 2,
+ "display_description": null,
+ "created_at": "2014-05-19T19:52:04.949734",
+ "metadata": {},
+ "volume_id": "123",
+ "os-extended-snapshot-attribute:progress": "100%"
+ }, {
+ "links": [
+ {
+ "href": "https://example.org/cyclades/v2/snapshots/43",
+ "rel": "self"
+ }, {
+ "href": "https://example.org/cyclades/v2/snapshots/43",
+ "rel": "bookmark"
+ }
+ ],
+ "id": "43",
+ "display_name": "Snapshot Two",
+ "status": "AVAILABLE",
+ "size": 3,
+ "display_description": null,
+ "created_at": "2014-05-20T19:52:04.949734",
+ "metadata": {},
+ "volume_id": "124",
+ "os-extended-snapshot-attribute:progress": "100%"
+ }
+ ]
+ }
+
+Get Snapshot Details
+--------------------
+
+This operation returns detailed information for a snapshot
+
+.. rubric:: Request
+
+============================ ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+============================ ====== ======== ==========
+``/snapshots/`` GET ✔ ✔
+============================ ====== ======== ==========
+
+|
+
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+============== ========================= ======== ==========
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) Request succeeded
+400 (Bad Request) Malformed volume id
+401 (Unauthorized) Missing or expired user token
+404 (Not Found) Snapshot not found
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) No available backends or service currently
+\ unavailable
+=========================== =====================
+
+|
+
+Response body contents::
+
+ snapshot: {
+ : ,
+ ...
+ }
+
+Snapshot attributes are explained `here <#snapshot-ref>`_
+
+*Example Get Snapshot Response*
+
+.. code-block:: javascript
+
+ GET https://example.org/cyclades/v2/snapshots/sn4p5h071
+
+ {
+ "snapshot": {
+ "links": [
+ {
+ "href": "https://example.org/cyclades/v2/snapshots/42",
+ "rel": "self"
+ }, {
+ "href": "https://example.org/cyclades/v2/snapshots/42",
+ "rel": "bookmark"
+ }
+ ],
+ "id": "42",
+ "display_name": "Snapshot One",
+ "status": "AVAILABLE",
+ "size": 2,
+ "display_description": null,
+ "created_at": "2014-05-19T19:52:04.949734",
+ "metadata": {},
+ "volume_id": "123",
+ "os-extended-snapshot-attribute:progress": "100%",
+ }
+ }
+
+Create Snapshot
+---------------
+
+Create a new snapshot
+
+.. rubric:: Request
+
+============== ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+============== ====== ======== ==========
+``/snapshots`` POST ✔ ✔
+============== ====== ======== ==========
+
+|
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+Content-Type Type or request body required required
+Content-Length Length of request body required required
+============== ========================= ======== ==========
+
+Request body contents::
+
+ snapshot: {
+ : ,
+ ...
+ }
+
+=================== ================================ ======== ================
+Volume Attribute Value Cyclades OS/Block Storage
+=================== ================================ ======== ================
+volume_id Volume to create snapshot from required required
+display_name The name ✔ ✔
+display_description A description ✔ ✔
+force Whether to snapshot **✘** ✔
+=================== ================================ ======== ================
+
+*Example Create Volume Request: JSON*
+
+.. code-block:: javascript
+
+ POST https://example.org/cyclades/v2/volumes
+
+ {
+ "volume": {
+ "volume_id": "44",
+ "display_name": "Snapshot Three"
+ }
+ }
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+202 (OK) Request succeeded
+400 (Bad Request) Malformed request data
+401 (Unauthorized) Missing or expired user token
+403 (Forbidden) User is not allowed to perform this operation
+404 (Not Found) Snapshot not found
+413 (Over Limit) Exceeded some resource limit
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) No available backends or service currently
+\ unavailable
+=========================== =====================
+
+|
+
+Response body contents::
+
+ snapshot: {
+ : ,
+ ...
+ }
+
+Snapshots attributes are `listed here <#snapshot-ref>`_.
+
+*Example Create Snapshot Response: JSON*
+
+.. code-block:: javascript
+
+ {
+ "snapshot": {
+ "links": [
+ {
+ "href": "https://example.org/cyclades/v2/snapshots/44",
+ "rel": "self"
+ }, {
+ "href": "https://example.org/cyclades/v2/snapshots/44",
+ "rel": "bookmark"
+ }
+ ],
+ "id": "44",
+ "display_name": "Snapshot Three",
+ "status": "CREATING",
+ "size": 10,
+ "display_description": null,
+ "created_at": "2014-05-19T19:52:04.949734",
+ "metadata": {},
+ "volume_id": "123",
+ "os-extended-snapshot-attribute:progress": "100%",
+ }
+ }
+
+Update Snapshot
+---------------
+
+.. rubric:: Response
+
+============================ ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+============================ ====== ======== ==========
+``/snapshots/`` PUT ✔ ✔
+============================ ====== ======== ==========
+
+|
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+Content-Type Type or request body required required
+Content-Length Length of request body required required
+============== ========================= ======== ==========
+
+Request body contents::
+
+ snapshot: {
+ : ,
+ ...
+ }
+
+=================== ===================== ======== ==========
+Attribute Description Cyclades OS/Block Storage
+=================== ===================== ======== ==========
+display_name Server name ✔ ✔
+display_description Descrition ✔ ✔
+=================== ===================== ======== ==========
+
+*Example Rename Server Request: JSON*
+
+.. code-block:: javascript
+
+ POST https://example.org/cyclades/v2/snapshots/44
+
+ {"snapshot": {"display_name": "New name"}}
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) Request succeeded
+400 (Bad Request) Malformed request
+401 (Unauthorized) Missing or expired user token
+403 (Forbidden) User is not allowed to perform this operation
+404 (Not Found) Snapshot not found
+409 (Build In Progress) Snapshot is not ready yet
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) No available backends or service currently
+\ unavailable
+=========================== =====================
+
+Response body contents::
+
+ snapshot: {
+ : ,
+ ...
+ }
+
+Snapshot attributes are explained `here <#snapshot-ref>`_
+
+*Example update snapshot Response*
+
+.. code-block:: javascript
+
+ {
+ "snapshot": {
+ "id": "44",
+ "display_name": "New Name",
+ ...
+ }
+ }
+
+Delete Snapshot
+---------------
+
+.. rubric:: Request
+
+============================ ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+============================ ====== ======== ==========
+``/snapshots/`` DELETE ✔ ✔
+============================ ====== ======== ==========
+
+|
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+============== ========================= ======== ==========
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+202 (OK) Request succeeded
+400 (Bad Request) Malformed server id
+401 (Unauthorized) Missing or expired user token
+404 (Not Found) Snapshot not found
+409 (Build In Progress) Snapshot is not ready yet
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) Action not supported or service currently
+\ unavailable
+=========================== =====================
+
+
+List Volume Types
+-----------------
+
+.. rubric:: Request
+
+========== ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+========== ====== ======== ==========
+``/types`` GET ✔ ✔
+========== ====== ======== ==========
+
+|
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+============== ========================= ======== ==========
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) Request succeeded
+401 (Unauthorized) Missing or expired user token
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) The service is not currently available
+=========================== =====================
+
+|
+
+Response body contents::
+
+ volume_types: [
+ {
+ : ,
+ ...
+ }, ...
+ ]
+
+The volume type attributes are listed `here <#volume-type-ref>`_
+
+*Example List Volumes: JSON (regular)*
+
+.. code-block:: javascript
+
+ GET https://example.org/cyclades/v2/types
+
+ {
+ "volumes": [
+ {
+ "id": 1,
+ "display_name": "Basic type",
+ "extra_specs": {...}
+ }, {
+ "id": 2,
+ "display_name": "Special type",
+ "extra_specs": {...}
+ }
+ ]
+ }
+
+Get Volume Type Details
+-----------------------
+
+This operation returns detailed information for a volume type
+
+.. rubric:: Request
+
+=========================== ====== ======== ==========
+URI Method Cyclades OS/Block Storage
+=========================== ====== ======== ==========
+``/types/`` GET ✔ ✔
+=========================== ====== ======== ==========
+
+|
+
+============== ========================= ======== ==========
+Request Header Value Cyclades OS/Block Storage
+============== ========================= ======== ==========
+X-Auth-Token User authentication token required required
+============== ========================= ======== ==========
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) Request succeeded
+400 (Bad Request) Malformed volume type id
+401 (Unauthorized) Missing or expired user token
+404 (Not Found) Volume type not found
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) No available backends or service currently
+\ unavailable
+=========================== =====================
+
+|
+
+Response body contents::
+
+ volume_type: {
+ : ,
+ ...
+ }
+
+Volume attributes are explained `here <#volume-type-ref>`_
+
+*Example Get Volume Response*
+
+.. code-block:: javascript
+
+ GET https://example.org/cyclades/v2/types/1
+
+ {
+ "volume_type": {
+ "id": 1,
+ "display_name": "Volume Three",
+ "extra_specs": {...}
+ }
+ }
+
+
+Index of Attributes
+-------------------
+
+.. _volume-ref:
+
+Volume Attributes
+.................
+
+===================== ======== ================
+Volume attribute Cyclades OS/Block Storage
+===================== ======== ================
+id ✔ ✔
+display_name ✔ ✔
+links ✔ ✔
+status ✔ ✔
+size ✔ ✔
+display_description ✔ ✔
+created_at ✔ ✔
+metadata ✔ ✔
+snapshot_id ✔ ✔
+source_volid ✔ ✔
+attachments ✔ ✔
+volume_type ✔ ✔
+delete_on_termination ✔ ✔
+image_id ✔ **✘**
+project ✔ **✘**
+availability_zone **✘** ✔
+bootable **✘** ✔
+===================== ======== ================
+
+* **id** The unique volume ID
+
+* **display_name** A name for the volume
+
+* **links** The reference links for the volume
+
+* **status** The volume status can be CREATING, AVAILABLE or DELETED
+
+* **size** The size of the volume in GB
+
+* **display_description** A description of the volume
+
+* **created_at** Date and time of volumes' creation
+
+* **metadata** A list of key-value metadata pairs
+
+* **snapshot_id** The ID of the snapshot this volume was created from
+
+* **source_volid** The ID of the source volume, this volume was created from
+
+* **attachments** One or more instance attachments
+
+* **volume_type** The type of the volume (See Volume types API)
+
+* **delete_on_termination** Whether this volume will be deleted on termination
+
+* **image_id** The ID of the image this volume was created from
+
+* **project** The ID of the project this volume is assigned to (quotas)
+
+.. _snapshot-ref:
+
+Snapshot Attributes
+...................
+
+========================================= ======== ================
+Snapshot attribute Cyclades OS/Block Storage
+========================================= ======== ================
+id ✔ ✔
+display_name ✔ ✔
+links ✔ ✔
+display_description ✔ ✔
+status ✔ ✔
+created_at ✔ ✔
+size ✔ ✔
+volume_id ✔ ✔
+metadata ✔ ✔
+os-extended-snapshot-attribute:progress ✔ ✔
+os-extended-snapshot-attribute:project_id **✘** ✔
+========================================= ======== ================
+
+* **id** The unique snapshot ID
+
+* **display_name** A name for the snapshot
+
+* **links** The reference links for the snapshot
+
+* **status** The snapshot status can be CREATING, AVAILABLE or DELETED
+
+* **size** The size of the snapshot in GB
+
+* **display_description** A description of the snapshot
+
+* **created_at** Date and time of snapshots' creation
+
+* **volume_id** The volume this is a snapshot of
+
+* **metadata** A list of key-value metadata pairs
+
+* **os-extended-snapshot-attribute:progress** creation progress
+
+.. _volume-type-ref:
+
+Volume Type Attributes
+......................
+
+================== ======== ================
+Snapshot attribute Cyclades OS/Block Storage
+================== ======== ================
+id ✔ ✔
+display_name ✔ ✔
+extra_specs ✔ ✔
+================== ======== ================
+
+* **id** The ID of the volume type
+
+* **display_name** A name for the volume type
+
+* **extra_specs** A dictionary of various specifications
+
diff --git a/docs/compute-api-guide.rst b/docs/compute-api-guide.rst
index 3362a8864b961ea90eb2bd4ce776d5ba5d84d1b6..3b9b5953ccb5975e097dd397c30806cea0e57326 100644
--- a/docs/compute-api-guide.rst
+++ b/docs/compute-api-guide.rst
@@ -33,7 +33,7 @@ Authentication
--------------
All requests use the same authentication method: an ``X-Auth-Token`` header is
-passed to the servive, which is used to authenticate the user and retrieve user
+passed to the service, which is used to authenticate the user and retrieve user
related information. No other user details are passed through HTTP.
Efficient Polling with the Changes-Since Parameter
@@ -51,7 +51,7 @@ Efficient Polling with the Changes-Since Parameter
Limitations
-----------
-* Version MIME type and vesionless requests are not currently supported.
+* Version MIME type and versionless requests are not currently supported.
* Cyclades only supports JSON Requests and JSON/XML Responses. XML Requests are
currently not supported.
@@ -96,7 +96,7 @@ Description URI
`Get Meta Item <#get-server-metadata-item>`_ ``/servers//metadata/`` GET ✔ ✔
`Update Meta Item <#update-server-metadata-item>`_ ``/servers//metadata/`` PUT ✔ ✔
`Delete Meta Item <#delete-server-metadata>`_ ``/servers//metadata/`` DELETE ✔ ✔
-`Actions <#server-actions>`_ ``servers//action`` POST ✔ ✔
+`Actions <#server-actions>`_ ``/servers//action`` POST ✔ ✔
================================================== ========================================= ====== ======== ==========
.. rubric:: Flavors
@@ -507,27 +507,49 @@ Request body contents::
...
}
-=========== ====================== ======== ==========
-Attributes Description Cyclades OS/Compute
-=========== ====================== ======== ==========
-name The server name ✔ ✔
-imageRef Image id ✔ ✔
-flavorRef Resources flavor ✔ ✔
-personality Personality contents ✔ ✔
-metadata Custom metadata ✔ ✔
-networks Connection information ✔ ✔
-.. project .. Project UUID .. ✔ .. **✘**
-=========== ====================== ======== ==========
+=========== ==================== ======== ==========
+Attributes Description Cyclades OS/Compute
+=========== ==================== ======== ==========
+name The server name ✔ ✔
+imageRef Image id ✔ ✔
+flavorRef Resources flavor ✔ ✔
+personality Personality contents ✔ ✔
+metadata Custom metadata ✔ ✔
+project Project assignment ✔ **✘**
+=========== ==================== ======== ==========
* **name** can be any string
-* **imageRed** and **flavorRed** should refer to existing images and hardware
+* **imageRef** and **flavorRef** should refer to existing images and hardware
flavors accessible by the user
* **metadata** are ``key``:``value`` pairs of custom server-specific metadata.
There are no semantic limitations, although the ``OS`` and ``USERS`` values
should rather be defined
+* **project** (optional) is the project where the VM is to be assigned. If not
+ given, user's system project is assumed (identified with the same uuid as the
+ user).
+
+* **personality** (optional) is a list of personality injections. A personality
+ injection is a way to add a file into a virtual server while creating it.
+ Each change modifies/creates a file on the virtual server. The injected data
+ (``contents``) should not exceed 10240 *bytes* in size and must be base64
+ encoded. The file mode should be a number, not a string. A personality
+ injection contains the following attributes:
+
+====================== =================== ======== ==========
+Personality Attributes Description Cyclades OS/Compute
+====================== =================== ======== ==========
+path File path on server ✔ ✔
+contents Data to inject ✔ ✔
+group User group ✔ **✘**
+mode File access mode ✔ **✘**
+owner File owner ✔ **✘**
+====================== =================== ======== ==========
+
+*Example Create Server Request: JSON*
+
* **personality** (optional) is a list of
`personality injections <#personality-ref>`_
@@ -762,6 +784,7 @@ netTimeSeries Network load / time graph URL
*Example Get Server Stats Response: JSON*
.. code-block:: javascript
+
GET https://example.org/compute/v2.0/servers/5678/stats
{
"stats": {
@@ -1584,6 +1607,7 @@ Operations Cyclades OS/Compute
`Reboot <#reboot-server>`_ ✔ ✔
`Get Console <#get-server-console>`_ ✔ **✘**
`Set Firewall <#set-server-firewall-profile>`_ ✔ **✘**
+`Reassign <#reassign-server>`_ ✔ **✘**
`Change Admin Password <#os-compute-specific>`_ **✘** ✔
`Rebuild <#os-compute-specific>`_ **✘** ✔
`Resize <#resize-server>`_ ✔ ✔
@@ -1767,23 +1791,27 @@ Request body contents::
.. note:: Response body should be empty
-.. Server reassign
-.. .............
+Reassign Server
+...............
-.. Each resource is assigned to a project. A Synnefo project is a set of resource
-.. limits e.g., maximum number of CPU cores per user, maximum ammount of RAM, etc.
+This operation assigns the VM to a different project.
+Each resource is assigned to a project. A Synnefo project is a set of resource
+limits e.g., maximum number of CPU cores per user, maximum ammount of RAM, etc.
-.. Although its resource is assigned exactly one project, a user may be a member
-.. of more, so that different resources are registered to different projects.
+Although its resource is assigned exactly one project, a user may be a member
+of more, so that different resources are registered to different projects.
-.. Project reassignment is the process of assigning a project to a different
-.. project
+Request body contents::
-.. Request body contents::
-.. reassign: { project: }
+ reassign: { project: }
-.. .. code-block:: javascript
-.. "reassign": { "project": "s0m3-pr0j3ct-1d"}
+*Example Action reassign: JSON**
+
+.. code-block:: javascript
+
+ {"reassign": {"project": "9969f2fd-86d8-45d6-9106-5e251f7dd92f"}}
+
+.. note:: Response body should be empty
OS/Compute Specific
...................
@@ -2564,6 +2592,8 @@ Return Code Description
503 (Service Unavailable) The server is not currently available
=========================== =====================
+.. note:: In case of a 204 code, the response body should be empty.
+
Index of Attributes
-------------------
diff --git a/docs/conf.py b/docs/conf.py
index a4a3f1e2a96c92b366f7615b27f2c1c8ea2d596c..0ec07f4a4bf7b8216bd20272825a2fdb102d0449 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -8,7 +8,7 @@ reload(synnefo.versions)
from synnefo.versions.app import __version__
project = u'synnefo'
-copyright = u'2012-2013, GRNET'
+copyright = u'2012-2014, GRNET'
version = __version__
release = __version__
html_title = 'synnefo ' + version
@@ -42,30 +42,23 @@ html_theme_options = {
'codetextcolor': '#333333'
}
-html_static_path = ['_static']
htmlhelp_basename = 'synnefodoc'
intersphinx_mapping = {
- 'pithon': ('http://docs.python.org/', None),
+ 'python': ('http://docs.python.org/', None),
'django': ('https://docs.djangoproject.com/en/dev/',
'https://docs.djangoproject.com/en/dev/_objects/')
}
-SYNNEFO_DOCS_BASE_URL = 'http://www.synnefo.org/docs'
-SYNNEFO_PROJECTS = {
- 'synnefo': 'dev',
- 'pithos': 'dev',
- 'snf-webproject': 'dev',
- 'snf-common': 'dev',
- 'snf-image': 'dev',
- 'snf-astakos-client': 'dev',
- 'snf-cyclades-app': 'dev'
-}
+SYNNEFO_PROJECTS = ['synnefo', 'archipelago', 'kamaki', 'snf-image',
+ 'snf-image-creator', 'nfdhcpd', 'snf-vncauthproxy',
+ 'snf-network']
+
+SYNNEFO_DOCS_BASEURL = 'https://www.synnefo.org/docs/%s/latest/objects.inv'
-for name, ver in SYNNEFO_PROJECTS.iteritems():
- intersphinx_mapping[name.replace("-","")] = (SYNNEFO_DOCS_BASE_URL +
- '%s/%s/' % (name, ver),
- None)
+for project in SYNNEFO_PROJECTS:
+ project_url = SYNNEFO_DOCS_BASEURL % project
+ intersphinx_mapping[project.replace('-', '')] = (os.path.dirname(project_url), project_url)
extensions = ['sphinx.ext.autodoc',
'sphinx.ext.intersphinx',
diff --git a/docs/cyclades.rst b/docs/cyclades.rst
index d13e40df0c491028d7a6472ffad764b9c125e5c9..8bceaa755d4d0044bfa8291796e0fc8a3f9ac791 100644
--- a/docs/cyclades.rst
+++ b/docs/cyclades.rst
@@ -1,18 +1,18 @@
.. _cyclades:
-Compute/Network/Image Service (Cyclades)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Compute/Network/Image/Volume Service (Cyclades)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Cyclades is the Synnefo component that implements the Compute, Network, Image
and Volume services. It exposes the associated OpenStack REST APIs: OpenStack
-Compute, Network, Glance and soon also Cinder. Cyclades is the part which
-manages multiple Ganeti clusters at the backend. Cyclades issues commands to a
-Ganeti cluster using Ganeti's Remote API (RAPI). The administrator can expand
-the infrastructure dynamically by adding new Ganeti clusters to reach
-datacenter scale. Cyclades knows nothing about low-level VM management
-operations, e.g., the handling of VM creations, migrations among physical
-nodes and node downtimes; the design and implementation of the
-end-user API is orthogonal to VM handling at the backend.
+Nova, Neutron, Glance and Cinder. Cyclades is the part which manages multiple
+Ganeti clusters at the backend. Cyclades issues commands to a Ganeti cluster
+using Ganeti's Remote API (RAPI). The administrator can expand the
+infrastructure dynamically by adding new Ganeti clusters to reach datacenter or
+multi-datacenter scale. Cyclades knows nothing about low-level VM management
+operations, e.g., the handling of VM creations, migrations among physical nodes
+and node downtimes; the design and implementation of the end-user API is
+orthogonal to VM handling at the backend.
There are two distinct, asynchronous paths in the interaction between Synnefo
and Ganeti. The `effect` path is activated in response to a user request;
@@ -26,8 +26,8 @@ Users have full control over their VMs: they can create new ones, start them,
shutdown, reboot, and destroy them. For the configuration of their VMs they can
select number of CPUs, size of RAM and system disk, and operating system from
pre-defined Images including popular Linux distros (Debian, Ubuntu, CentOS,
-Fedora, Gentoo, Archlinux, OpenSuse), MS-Windows Server 2008 R2 and 2012 as
-well as FreeBSD.
+Fedora, Gentoo, Archlinux, OpenSuse, Oracle Linux), MS-Windows Server 2008 R2,
+2012 and 2012 R2, as well as BSD (FreeBSD, NetBSD, OpenBSD).
The REST API for VM management, being OpenStack compatible, can interoperate
with 3rd party tools and client libraries.
diff --git a/docs/design/cyclades-networking.rst b/docs/design/cyclades-networking.rst
index 62a55b3c4c9171d991998156bcf65da98faa8288..01bde906a52f0f64b9ea1cd5d19f85901e3f9752 100644
--- a/docs/design/cyclades-networking.rst
+++ b/docs/design/cyclades-networking.rst
@@ -10,7 +10,7 @@ Flavor, either an isolated Layer-2 broadcast domain or a Layer-3 network.
Networks can either be private or public. Private networks are reserved for the
user who created it, while public networks are created by the administrator and
are visible to all users. Also, networks can be marked with the
-`--router:external` attribute to indicate that are external networks (public
+`--router:external` attribute to indicate they are external networks (public
internet)
Currently there are four available Networks Flavors:
@@ -23,7 +23,7 @@ Currently there are four available Networks Flavors:
unique MAC prefix that is assigned to each network.
The administrator can limit which networks can be created via API with the
-`API_ENABLED_NETOWRK_FLAVORS` setting.
+`API_ENABLED_NETWORK_FLAVORS` setting.
The attributes for network objects are the following:
@@ -299,7 +299,7 @@ Finally, the request can contain the following optional attributes:
Example request:
-.. code:: json
+.. code-block:: json
{"port": {
"name": "port1",
diff --git a/docs/design/logging-management-commands.rst b/docs/design/logging-management-commands.rst
new file mode 100644
index 0000000000000000000000000000000000000000..13900693cbf3b183f4631e0f2792cde721138ada
--- /dev/null
+++ b/docs/design/logging-management-commands.rst
@@ -0,0 +1,113 @@
+===================================================
+Logging mechanism for Synnefo's management commands
+===================================================
+
+
+Abstract
+========
+
+Log all stdout and stderr output of every invocation of snf-manage, on unique
+filenames under a given directory.
+
+
+Current state and shortcomings
+==============================
+
+All Synnefo's management commands are written as custom django-admin commands.
+This means that every management command is in fact a class that extends
+Django's BaseCommand class.
+
+Django's *BaseCommand* provides the attributes ``self.stdout`` and
+``self.stderr`` and Django's documentation encourages the users to use these
+attributes if they wish to write to the console. Django doesn't provide an
+option to write the output to files and the user has to implement this
+explicitly when implementing the ``handle`` method.
+
+We would like to extend the above mechanism to allow every ``snf-manage``
+command to log all stdout and stderr output on a unique filename under a given
+directory. The implementation should change nothing in the way that users write
+management commands (only acceptable change is that the new commands may have
+to inherit a new class and not the *BaseCommand* one). This means that
+existing management commands should play out of the box and also that the
+logging mechanism will globally apply to all of them.
+
+A new Synnefo setting named **LOGGER_EXCLUDE_COMMANDS** has been added that
+specifies which commands will not be logged. By default, commands that do not
+alter the state of the server (i.e. \*-list and \*-show commands) will be
+excluded from the logging mechanism. One can disable this logging mechanism all
+together by setting the above variable to ".\*".
+
+
+Proposed changes
+================
+
+In this section we will try to explain the way that the new logging mechanism
+will be implemented as well as the reasons behind these decisions.
+
+As we previously saw, we want the logging mechanism to be global and to work
+for all the ``snf-manage`` commands without extra effort. This means that the
+management commands will continue to use the ``self.stdout`` and
+``self.stderr`` attributes from *BaseCommand* class to provide console
+output. Therefor we have to provide our own ``self.stdout`` and ``self.stderr``
+objects that will preserve the previous functionality and log to files at the
+same time. There are two ways to achieve that:
+
+Patch the Django's *BaseCommand* class and replace ``self.stdout`` and
+``self.stderr`` attributes.
+
+ This solution requires the minimum amount of changes to the management
+ commands' code as they will use our patched version of *BaseCommand*.
+ The downside is that we have to patch a library provided class. We are not
+ encouraging these type of patches because it obfuscates the code (the
+ programmer is expecting to use Django's *BaseCommand* class, not ours)
+ and does not preserve compatibility with other Django versions (if the
+ implementation of Django's *BaseCommand* changes our patch will not
+ work).
+
+Create a new class that extends Django's *BaseCommand*.
+
+ The downside of this solution is that we have to change the existing code
+ so all management commands will inherit our new class and not Django's
+ *BaseCommand*. But we find this solution to be cleaner.
+
+For the above reasons we decided to go with the second option.
+
+Django's ``self.stdout`` and ``self.stderr`` are implemented as *OutputWrapper*
+objects. We will create our own class (**SynnefoOutputWrapper**) which will use
+python's logging library to handle the file part of the logging and the
+original *OutputWrapper* object to handle the console part (since we want to
+preserve the functionality of Django's *OutputWrapper* and the style functions
+it uses to pretty print the messages).
+
+Our new class has to be a **descriptor**. This is because *BaseCommand* doesn't
+initialize the ``stdout`` and ``stderr`` attributes at ``__init__`` but sets
+them only when it needs to (meaning inside the *execute* method).
+
+The above classes will be written in snf-django-lib package meaning that all
+the other packages will have a dependency in snf-django-lib.
+
+We will combine timestamp, command name and PID to form unique names, e.g.:
+20140120113432-server-modify-4564, where "4564" was the PID. The timestamp will
+be first so that files will be chronologically sorted.
+
+
+Implementation details
+======================
+
+The implementation will follow the following steps:
+
+- Change current management commands to use ``self.stdout`` and ``self.stderr``
+ to provide console output instead of ``sys.stdout``, ``print`` or anything
+ else. This change complies with Django's documentation.
+
+- Write a new class that will replace Django's *OutputWrapper*.
+
+- Change the **SynnefoCommand** class so that it will extend Django's
+ *BaseCommand* and will replace ``stdout`` and ``stderr`` attributes.
+
+- Change all management commands to inherit the **SynnefoBaseCommand** class.
+
+- Update package dependencies.
+
+- Add a new Synnefo setting to allow the user to change the directory where
+ the output will be saved.
diff --git a/docs/design/multi-db-transactions.rst b/docs/design/multi-db-transactions.rst
new file mode 100644
index 0000000000000000000000000000000000000000..23c82e1f247706a1414aaae75119ae4b9d311222
--- /dev/null
+++ b/docs/design/multi-db-transactions.rst
@@ -0,0 +1,67 @@
+.. _multi-db-transactions:
+
+===============================
+Multi-DB transactions in Django
+===============================
+
+In this document, we will talk about Django transactions. We will explain why
+transactions are a problem in a multi-db scenario and propose a way to solve
+this problem.
+
+Django transactions
+===================
+
+Django has a useful library to handle database transactions called
+`transcation`. From this library, Synnefo uses two decorators:
+`commit_on_success` and `commit_manually`.
+
+* `commit_on_success` opens a transaction with the database on enter and
+ commits any queued changes on successful exit. Should an exception occur
+ midway, we can rest assured that there is nothing half-committed.
+* `commit_manually` opens a transaction with the database on enter but does not
+ commit/rollback anything, unless explicitly told so using the
+ `commit()`/`rollback()` functions.
+
+Using these decorators as-is is fine when the Cyclades/Astakos nodes have to
+deal with one database. In the case of two or more databases however, the user
+needs to add the `using` argument in these decorators to define the database
+that will be used to open a transaction. Else, they use the 'default' database.
+
+Solution
+========
+
+The most straight-forward solution is to add in all decorators a `using`
+argument with **cyclades** or **astakos** as database names, and enforce that
+all settings from now on will have these two entries.
+
+The only issue with this approach is that the core problem, that is, how to
+chose between multiple databases in the application level, will be solved with
+two different ways. The one way is with the `using` argument in transactions,
+while the other one is with *database routers*.
+
+Therefore, we propose a solution for the transaction problem that converges
+with database routers.
+
+The solution is the following:
+
+* We will write wrappers for the `commit_on_success` and `commit_manuall`
+ decorators. There will be one wrapper for Astakos and one for Cycades. *
+ When called, these wrappers will decide which is the appropriate database to
+ start a transaction. The decision will use a function that is also used by
+ database routers (`select_db`), and determines which is the correct db for a
+ model. This is the converging point between transactions and database
+ routers.
+* Once the decision is made, the wrappers will return the original
+ `commit_on_success` / `commit_manually` decorators, but with the chosen db
+ as value of the `using` argument.
+
+To sum up, the only changes we need to do is to add a `transaction.py` in
+Astakos and Cyclades, write the wrappers and replace this import:
+
+.. code-block:: python
+
+ from django.db import transaction
+
+with the Astakos/Cyclades `transaction.py` file, in any code that uses the
+`commit_on_success`/`commit_manually` decorators.
+
diff --git a/docs/design/resource-pool-projects.rst b/docs/design/resource-pool-projects.rst
index 5b04bc682fc47062a8530d0e932074e9cfa12c0b..6a050edc1f3a34c23ed4c0629442beca693d2188 100644
--- a/docs/design/resource-pool-projects.rst
+++ b/docs/design/resource-pool-projects.rst
@@ -40,8 +40,10 @@ cannot exceed the former. A limit on the number of members allowed is still
enforced.
Projects will be the sole source of resources. Current base quota offered to
-users by the system will be expressed in terms of special-purpose *base*
-projects.
+users by the system will be expressed in terms of special-purpose *system*
+projects. Due to the central role that projects now acquire, we will alter
+the project schema to facilitate project creation and modification without
+the extra overhead of submitting and approving applications.
Implementation details
======================
@@ -88,29 +90,31 @@ Deallocation is always allowed as long as usage does not fall below zero.
Counters with zero usage and limit could by garbage collected by Astakos, if
needed.
-Base projects
--------------
+System projects
+---------------
For reasons of uniformity, we replace the base quota mechanism with projects.
-In a similar vein to OpenStack tenants, we define new user-specific *base*
+In a similar vein to OpenStack tenants, we define new user-specific *system*
projects to account for the base quota for each user. These projects should
be clearly associated with a single user, restrict join/leave actions and
-specify the quota granted by the system. When a new user is created,
-their base project will be automatically created and linked back to the user.
-User activation will trigger project activation, granting the default resource
-quota. Base projects will have no owner, marked thusly as `system' projects.
-The administrator can, following the usual project logic, alter quota by
-modifying the project. Users cannot apply for modification of their base
-projects.
-
-Projects will, from now on, be identified by a UUID. Base projects will
+specify the quota granted by the system. When a user is accepted, their system
+project will be automatically created, activated, and linked back to the user,
+granting the default resource quota. These projects will have no owner, marked
+thusly as `system' projects. The administrator can, following the usual
+project logic, alter quota by modifying the project. Users cannot apply for
+modification of their system projects.
+
+Projects will, from now on, be identified by a UUID. System projects will
receive the same UUID as the user itself. ProjectID, which appears above in
-the Quotaholder entries, refers to the project UUID.
+the Quotaholder entries, refers to the project UUID. When a system project is
+created for a user, there is a slight probability the user's UUID is already
+in use by another project; in this case one can only delete the user and
+create a new one in its place.
Base quota will be expressed both in terms of a project-level and a
member-level limit. This will result in two operationally equivalent
Quotaholder counters, as in the following example. In the future, we could
-admit third-party users to a user's base project; in that case, those
+admit third-party users to a user's system project; in that case, those
counters would differ.
::
@@ -120,13 +124,54 @@ counters would differ.
cyclades.vm project:uuid None 5 1
cyclades.vm user:uuid project:uuid 5 1
-System default quota
---------------------
+Private projects
+----------------
+
+Since the introduction of system projects will explode the number of total
+projects, we will need to control their visibility. We add a new flag
+*private* in project definitions. A private project can only be accessed by
+its owner and members and not be advertized in the UI. System projects are
+marked as private.
+
+Decouple projects from applications
+-----------------------------------
+
+System projects do not fit well in the current project/application scheme,
+because no user has applied for them. Moveover, we would like to easily
+modify project properties, particularly quota limits, without the need to
+apply for an application for each project and then approve it.
+
+We will decouple projects from applications by incorporating the project
+definition into the project object rather than relying on an application.
+The system will directly make a new (system) project upon user creation and a
+privileged user will be able to modify an existing project by directly
+modifying it. An unprivileged user will still need to make an application.
+
+The project model is adapted to reference the *last* application that is
+related to the project, if any---projects automatically created by the
+system reference no application. For an uninitialized project, this
+denotes the original application through which the project was made. If
+the application is denied or cancelled, the whole project is considered
+deleted.
+
+Applications as modifications
+`````````````````````````````
+
+Application for a new project is created in state ``pending`` and its
+properties are copied into a new project object, which is in state
+``uninitialized``. To preserve this equality, we disallow modifications of
+uninitialized projects, either in-place or through an application. An
+already activated project can be modified by submitting an application
+containing just the desired changes. An application object stores the
+specified changes and should remain read-only.
+
+System default quota and resource registration
+----------------------------------------------
Each resource registered in the system is assigned a default quota limit.
A newly-activated user is given these limits as their base quota. This is
till now done by copying the default limits as user's entries in
-AstakosUserQuota. Default limits will from now on be copied into the base
+AstakosUserQuota. Default limits will from now on be copied into the system
project's resource definitions.
Conventional projects are created through a project application, which
@@ -139,14 +184,10 @@ one specifying the default base quota, in order to fill in missing limits
for conventional projects. It will be controled by a new option
``--project-default`` of command ``resource-modify``.
-Private projects
-----------------
-
-Since the introduction of base projects will explode the number of total
-projects, we will need to control their visibility. We add a new flag
-*private* in project definitions. A private project can only be accessed by
-its owner and members and not be advertised in the UI. Base projects are
-marked as private.
+When a project is activated, either directly in the case of system projects
+or through the approval of a project application, limits for resources not
+specified are automatically completed by consulting the appropriate
+skeleton.
Allocation of a new resource
----------------------------
@@ -198,10 +239,7 @@ In Cyclades, each VM, floating IP, or other distinct resource should be
linked to a project. Pithos should link containers to projects.
Astakos will handle its own resource ``astakos.pending_app`` in a special
-way: it will always be charged at the user's base project. This resource
-is marked with ``allow_in_projects = False`` in its definition. Since quota
-is now project-based, this flag will now be interpreted as forbidding usage
-in non-base projects.
+way: it will always be charged at the user's system project.
Resource reassignment
---------------------
@@ -268,8 +306,8 @@ This will issue the following provisions to the Quotaholder::
}
]
-API extensions
---------------
+API changes
+-----------
API call ``GET /quotas`` is extended to incorporate project-level quota. The
response contains entries for all projects for which a user/project pair
@@ -312,11 +350,15 @@ regardless of user::
}
}
+``GET /service_project_quotas`` will be used in a similar way as ``GET
+/service_quotas`` to get the project-level quotas for resources associated
+with the Synnefo component that makes the request.
+
All service API calls that create resources can specify the project where
they will be attributed.
In cyclades, ``POST /servers`` (likewise for networks and floating IPs) will
-receive an extra argument ``project``. If it is missing, the user's base
+receive an extra argument ``project``. If it is missing, the user's system
project will be assumed. In calls detailing a resource (e.g., ``GET
/servers/``), the field ``tenant_id`` will contain the
project id.
@@ -336,12 +378,30 @@ container to a given project, the latter reassigns an existing container.
Field ``x-container-policy-project`` will be retrieved by a ``HEAD`` call at
the container level.
+Changes in the projects API
+```````````````````````````
+
+``PUT /projects/`` will be used to mod a new project replacing
+``POST``. It now expects a dictionary with just the desired
+changes, not a complete project definition. It is only allowed if the
+project is already activated.
+
+``GET /projects/`` changes to include a ``last_application`` field,
+if applicable.
+
+Application actions (approve, deny, dismiss, cancel) are integrated into
+project actions and expect an extra ``app_id`` argument to specify the
+application. Actions are allowed only on a project's last application;
+the application id is required in order to avoid races.
+
+The applications API is removed, incorporated into the projects API.
+
User interface
--------------
User quota will be presented per project, including the aggregate activity
of other project members: the Resource Usage page will include a drop-down
-menu with all relevant projects. By default, user's base project will
+menu with all relevant projects. By default, user's system project will
be assumed. When choosing a project, usage for all resources will be
presented for the given project in the following style::
@@ -397,16 +457,15 @@ Quota can be queried per user or project::
------------------------
cyclades.vm 100 50
-A new command ``snf-manage project-modify`` will automate the process of
-applying/approving applications in order to modify some project settings,
-such as the quota limits.
+A new command ``snf-manage project-modify`` will enable in-place
+modification of project properties, such as their quota limits.
Currently, the administrator can change the user base quota with:
``snf-manage user-modify --base-quota ``.
This will be removed in favor of the ``project-modify`` command, so that all
quota are handled in a uniform way. Similar to ``user-modify --all``,
-``project-modify`` will get options ``--all-base`` and ``--all-non-base`` to
-allow updating quota in bulk.
+``project-modify`` will get options ``--all-system-projects`` to
+allow updating base quota in bulk.
Migration steps
===============
@@ -418,24 +477,24 @@ Existing projects need to be converted to resource-pool ones. The following
steps must be taken in Astakos:
* compute project-level limits for each resource as
max_members * member-level limit
- * create base projects based on base quota for each user
+ * create system projects based on base quota for each user
* make Quotaholder entries for projects and user/project pairs
- * assign all current usage to the base projects (both project
+ * assign all current usage to the system projects (both project
and user/project entries)
* set usage for all other entries to zero
Cyclades and Pithos should initialize their project attribute on each resource
-with the user's base project, that is, the same UUID as the resource owner.
+with the user's system project, that is, the same UUID as the resource owner.
Initial resource reassignment
-----------------------------
-Once migration has finished, users will be off-quota on their base project,
+Once migration has finished, users will be off-quota on their system project,
if they had used additional quota from projects. To alleviate this
situation, each service can attempt to reassign resources to other projects,
following this strategy:
* consult Astakos for projects and quota for a given user
* select resources that can fit in another project
- * issue a commission to decrease usage of the base project and likewise
+ * issue a commission to decrease usage of the system project and likewise
increase usage of the available project
* record the new ProjectUUID for the reassigned resources
diff --git a/docs/design/volume-snapshots.rst b/docs/design/volume-snapshots.rst
new file mode 100644
index 0000000000000000000000000000000000000000..344b4bd364809aef29ee950e2726e7e0a8811d98
--- /dev/null
+++ b/docs/design/volume-snapshots.rst
@@ -0,0 +1,209 @@
+====================
+Volume snapshots
+====================
+
+The goal of this design document is to record and present the proposed method
+for Volume snapshotting in Synnefo. We will describe the whole work-flow from
+the user's API down to Archipelago and up again.
+
+Snapshot functionality
+=======================
+
+The snapshot functionality aims to provide the user with the following:
+
+a. A thin snapshot of an Archipelago volume: |br|
+ The snapshot should capture the volume's data state at the time the
+ snapshot was requested. |br|
+ **Note:** The VM user is accountable for the consistency of its volume.
+ Journaled filesystems or prior shutdown of the VM is advised.
+#. Presenting the snapshot instantly as a regular file
+ on Pithos: |br|
+ Users can view their snapshots in Pithos as any other file that they have
+ uploaded and can subsequently download them.
+#. A registered Synnefo snapshot, ready to be deployed: |br|
+ Essentially, if a snapshotted volume includes an OS installation, it can be
+ used as any other Synnefo OS image. This allows users to create
+ *restoration points* for their VMs.
+
+Side goals
+^^^^^^^^^^
+
+In order to make the snapshot process as slim as possible, the following goals
+must also be met:
+
+a. Low computational/storage overhead: |br|
+ If many users send snapshot requests, the service should be able to respond
+ quick. Also, a snapshotted volume should not incur any significant storage
+ overhead.
+#. Solid reconciliation process: |br|
+ If a snapshot request fails, the system should do the necessary cleanups or
+ at least make it easy to reconcile the affected databases.
+
+Snapshot creation
+========================
+
+An illustration of the proposed method follows below:
+
+|create_snapshot|
+
+Each step of the procedure is explained below:
+
+#. The Cyclades App receives a snapshot request. The request is expected to
+ originate from the user and be sent via an API client or the Cyclades UI. A
+ valid snapshot request should target an existing volume ID.
+#. The Cyclades App uses its Pithos Backend to create a `snapshot record`_ in
+ the Pithos database. The snapshot record is explained in the following
+ section. It is initially set with the values that are seen in the diagram.
+#. The Cyclades App creates a snapshot job and sends it to the Ganeti
+ Master.
+#. The Ganeti Master in turn, designates the snapshot job to the appropriate
+ Ganeti Noded.
+#. The Ganeti Noded runs the corresponding Archipelago ``ExtStorage`` script
+ and invokes the Vlmc tool to create the snapshot.
+#. The Vlmc tool instructs Archipelago to create a snapshot by sending a
+ snapshot request.
+
+Once Archipelago has (un)successfully created the snapshot, the response is
+propagated to the Ganeti Master which in turn creates an event about this
+snapshot job and its execution result.
+
+7. snf-ganeti-eventd is informed about this event, using an ``inotify()``
+ mechanism, and forwards it to the snf-dispatcher.
+#. The snf-dispatcher uses its Pithos Backend to update the ``snapshot status``
+ property of the snapshot record that was created in Step 2. According to the
+ result of the snapshot request, the snapshot status is set as ``Ready`` or
+ ``Error``. This means that, as far as Cinder is concerned, the snapshot is
+ ready.
+#. The ``Available`` attribute however is still ``0``. Swift (Pithos) will make
+ it available (``1``) and thus usable, the first time it will try to ping
+ back (a.k.a. the first time someone tries to access it).
+
+Snapshot record
+^^^^^^^^^^^^^^^^^
+
+The snapshot record has the following attributes:
+
++-------------------+--------------------------------------+---------------+
+| Key | Value | Service |
++===================+======================================+===============+
+| file_name | Generated (string - see below) | Swift |
++-------------------+--------------------------------------+---------------+
+| available | Generated (boolean) | Swift |
++-------------------+--------------------------------------+---------------+
+
+and the following properties:
+
++-------------------+--------------------------------------+---------------+
+| Key | Value | Service |
++===================+======================================+===============+
+| snapshot_name | User-provided (string) | Cinder |
++-------------------+--------------------------------------+---------------+
+| snapshot_status | Generated (see Cinder API) | Cinder |
++-------------------+--------------------------------------+---------------+
+| EXCLUDE_ALL_TASKS | Generated (string) | Glance |
++-------------------+--------------------------------------+---------------+
+
+The file_name is a string that has the following form::
+
+ snf-snap--
+
+where:
+
+* ```` is the Volume ID and
+* ```` is the number of times that the volume has been snapshotted and
+ increases monotonically.
+
+The snapshot name should thus be unique.
+
+"Available" attribute
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``available`` attribute is a Swift attribute that is introduced with
+snapshots and applies to all Pithos files. When a Pithos file is "available",
+it means that its map file has been created and points to the correct data.
+Normally, all Pithos files have their map file properly created before adding a
+record in the Pithos database. Snapshots however are an exception to this rule,
+since their map file is created asynchronously.
+
+Therefore, the creation of a Pithos file has the following rules:
+
+* If the file is a snapshot, the ``available`` attribute is set to "0".
+* For all other files, the ``available`` attribute is set to "1".
+
+**Note:** ``available`` can change from "0" to "1", but never the opposite.
+
+The update of the ``available`` attribute happens implicitly after the creation
+of the snapshot, when a request reads the file record from the
+Pithos database. The following diagram shows how can a request (download
+request for example) update the ``available`` attribute.
+
+|available_attribute|
+
+In short, what happens is:
+
+#. The user asks to download the file from the Pithos App.
+#. The Pithos App checks the file record and finds that the ``available``
+ attribute is "0".
+#. It then pings Archipelago to check the status of the map.
+#. If the map exists, it sets ``available`` to "1" and can use the map file to
+ serve the data.
+
+VM creation from snapshot
+===============================
+
+The following diagram illustrates the VM creation process from a snapshot
+image.
+
+|spawn_from_snapshot|
+
+The steps are explained in detail below:
+
+#. A user who has the registered Images list (which includes all Snapshots
+ too), requests to create a VM from the Cyclades App, using one of the
+ registered snapshots of the list.
+#. The Cyclades App sends a VM creation job to the Ganeti Master with the
+ appropriate data. The data differ according to the disk template:
+
+ * If the template is ext, then the "origin" field has the archipelago map
+ name.
+ * For any other template, the archipelago map name is passed in the "img_id"
+ property of OS parameters.
+
+#. The Ganeti Master designates the job to the appropriate Ganeti Noded.
+#. The Ganeti Noded will create the disks, according to the requested disk
+ template:
+
+ a. If the disk template is "ext", the following execution path is taken:
+
+ 1. The Ganeti Noded runs the corresponding Archipelago ``ExtStorage``
+ script and invokes the Vlmc tool.
+ #. The Vlmc tool instructs Archipelago to create a volume from a
+ snapshot.
+
+ b. If the disk template is other than "ext", e.g. "drbd", Ganeti Noded
+ creates a new, empty disk.
+
+#. After the volume has been created, Ganeti Noded instructs snf-image to
+ prepare it for deployment. The parameters that are passed to snf-image are
+ the OS parameters of Step 2, and the main ones for each disk template are
+ shown in the diagram, next to the "Ext?" decision box. According to the disk
+ template, snf-image has two possible execution paths:
+
+ a. If the disk template is "ext", there is no need to copy any data.
+ Also, the image has the "EXCLUDE_ALL_TASKS" property set to "yes", so
+ snf-image will run no customization scripts and will simply return.
+ b. If the disk template is other than "ext", e.g. "drbd", snf-image will
+ copy the snapshot's data from Pithos to the created DRBD disk. As above,
+ since the image has the "EXCLUDE_ALL_TASKS" property set to "yes",
+ snf-image will run no customization scripts.
+
+.. |br| raw:: html
+
+
+
+.. |create_snapshot| image:: /images/create-snapshot.png
+
+.. |available_attribute| image:: /images/available-attribute.png
+
+.. |spawn_from_snapshot| image:: /images/spawn-from-snapshot.png
+
diff --git a/docs/design/volumes.rst b/docs/design/volumes.rst
new file mode 100644
index 0000000000000000000000000000000000000000..e5e817843e86eefaa1ab3c8ef1446b53ae046a9d
--- /dev/null
+++ b/docs/design/volumes.rst
@@ -0,0 +1,146 @@
+Cyclades Volumes
+^^^^^^^^^^^^^^^^
+
+This document describes the extension of Cyclades to handle block storage
+devices, referred to as Volumes.
+
+
+Current state and shortcomings
+==============================
+
+Currently one block storage device is created and destroyed together with the
+virtual servers. One disk is created per server with the size and the disk
+template that are described by the server's flavor. The disk is destroyed when
+the server is destroyed. Also, There is no way to attach/detach disks to
+existing servers and there is no API to expose information about the server's
+disks.
+
+
+Proposed changes
+================
+
+Cyclades will be able to manage volumes so that users can create volumes and
+dynamically attach them to servers. Volumes can be created either empty or
+filled with data from a specified source (image, volume or snapshot). Also,
+users can dynamically remove volumes. Finally, a server can be created with
+multiple volumes at once.
+
+
+Implementation details
+======================
+
+Known Limitations
+-----------------
+
+While addition and removal of disks is supported by Ganeti, it is not supported
+to handle disks that are not attached to an instance. There is no way to create
+a disk without attaching it to an instance, neither a way to detach a disk from
+an instance without deleting it. Future versions of Ganeti will make disks
+separate entities, thus overcoming the above mentioned issues. Until then,
+this issues will also force a limitation to the way Cyclades will be handling
+Volumes. Specifically, Cyclades volumes will not be attached/detached from
+servers; they will be only be added and removed from them.
+
+Apart from Ganeti's inability to manage disks as separate entities, attaching
+and detaching a disk is not meaningful for storage solutions that are not
+shared between Ganeti nodes and clusters, because this would require copying
+the data of the disks between nodes. Thus, the ability to attach/detach a disk
+will only be available for disks of externally mirrored template (EXT_MIRRORED
+disk templates).
+
+Also, apart from the root volume of an instance, initializing a volume with
+data is currently only available for Archipelago volumes, since there is no
+mechanism in Ganeti for filling data in other type of volumes (e.g. file, lvm,
+drbd). Until then, creating a volume from a source, other than the root volume
+of the instance, will only be supported for Archipelago.
+
+Finally, an instance can not have disks of different disk template.
+
+Cyclades internal representation
+--------------------------------
+
+Each volume will be represented by the new `Volume` database model, containing
+information about the Volume, like the following:
+
+* name: User defined name
+* description: User defined description
+* userid: The UUID of the owner
+* size: Volume size in GB
+* disk_template: The disk template of the volume
+* machine: The server the volume is currently attached, if any
+* source: The UUID of the source of the volume, prefixed by it's type, e.g. "snapshot:41234-43214-432"
+* status: The status of the volume
+
+
+Each Cyclades Volume corresponds to the disk of a Ganeti instance, uniquely
+named `snf-vol-$VOLUME_ID`.
+
+API extensions
+--------------
+
+The Cyclades API will be extended with all the needed operations to support
+Volume management. The API will be compatible with OpenStack Cinder API.
+
+Volume API
+``````````
+
+The new `volume` service will be implemented and will provide the basic
+operations for volume management. This service provides the `/volumes` and
+`/types` endpoints, almost as described in the OS Cinder API, with the only
+difference that we will extend `POST /volumes/volumes` with a required
+field to represent the server that the volume will be attached, and which is
+named `server_id`.
+
+
+Compute API extensions
+``````````````````````
+
+The API that is exposed by the `volume` service is enough for the addition and
+removal of volumes to existing servers. However, it does not cover creating a
+server from a volume or creating a server with more than one volumes. For this
+reason, the `POST /servers` API call will be extended with the
+`block_device_mapping_v2` attribute, as defined in the OS Cinder API, which is
+a list of dictionaries with the following keys:
+
+* source_type: The source of the volume (`volume` | `snapshot` | `image` | `blank`)
+* uuid: The uuid of the source (if not blank)
+* size: The size of the volume
+* delete_on_termination: Whether to preserve the volume on instance deletion.
+
+If the type is `volume`, then the volume that is specified by the `uuid` field
+will be used. Otherwise, a new volume will be created which will contain the
+data of the specified source (`image` or `snapshot`). If the source type
+is `blank` then the new volume will be empty.
+
+Also, we will implement `os_volume-attachments` extension, containing the
+following operations:
+
+* List server attachments(volumes)
+* Show server attachment information
+* Attach volume to server (notImplemented until Ganeti supports detachable volumes)
+* Detach volume from server (notImplemented until Ganeti supports detachable volumes)
+
+Quotas
+------
+
+The total size of volumes will be quotable, and will be directly mapped to
+existing `cyclades.disk` resource. In the future, we may implement having
+different quotas for each volume type.
+
+Command-line interface
+----------------------
+
+The following management commands will be implemented:
+
+* `volume-create`: Create a new volume
+* `volume-remove`: Remove a volume
+* `volume-list`: List volumes
+* `volume-info`: Show volume details
+* `volume-inspect`: Inspect volume in DB and Ganeti
+
+
+The following commands will be extended:
+
+* `server-create`: extended to specify server's volumes
+* `server-inspect`: extended to display server's volumes
+* `reconcile-servers`: extended to reconcile server's volumes
diff --git a/docs/dev-guide.rst b/docs/dev-guide.rst
index b8e22efadb5315cecf3c4de76b3d320fc90bbce9..7bd2dda883b77ad1aefdd96f5ede2ec42d9b907e 100644
--- a/docs/dev-guide.rst
+++ b/docs/dev-guide.rst
@@ -3,99 +3,259 @@
Synnefo Developer's Guide
^^^^^^^^^^^^^^^^^^^^^^^^^
-This is the complete Synnefo Developer's Guide.
+The suggested method of setting up an environment for development purposes is
+first to install Synnefo on a Debian Wheezy system and then to use Python's
+`development mode
+`_ to run
+Synnefo from a cloned repo and see your changes instantly.
-Environment set up
-==================
+Synnefo installation
+~~~~~~~~~~~~~~~~~~~~
-First of all you have to set up a developing environment for Synnefo.
+There are two main ways of installing Synnefo, each of which will be described
+below:
-**1. Create a new VM**
+* `Existing Debian Wheezy system (snf-deploy)`_: This method requires that you
+ have setup a Debian Wheezy system. On this setup, you can install Synnefo
+ using ``snf-deploy``.
+* `Synception (snf-ci)`_: This method builds Synnefo within Synnefo (you read
+ right), which means that you need to have an account in an existing Synnefo
+ installation. Then, using ``snf-ci`` in conjunction with ``kamaki``, you can
+ create a new Debian Wheezy VM in that installation. The rest is handled
+ automatically by ``snf-ci``, which uses ``snf-deploy`` to install Synnefo in
+ that VM.
-It has been tested on Debian Wheezy. It is expected to work with other
-releases (e.g., Squeeze) too, as long as they are supported by
-``snf-deploy``.
+.. note::
-**2. Build your own Synnefo installation**
+ The first method will build Synnefo from our stable branch. If your
+ development takes place in ``develop`` or in a ``feature`` branch, then you
+ may need to update your installation manually, e.g. run database migrations
+ or install new packages.
-Follow the instructions `here `_
-to build Synnefo on a single node using ``snf-deploy``.
-**3. Install GitPython**
+Existing Debian Wheezy system (snf-deploy)
+------------------------------------------
-.. code-block:: console
+**1. Install Synnefo**
- # pip install gitpython
+In order to create a one-node Synnefo installation, you can follow the
+instructions `here
+`_.
-**4. Install devflow**
+**2. Install Devflow**
-Devflow is a tool to manage versions, helps implement the git flow development process,
-and builds Python and Debian packages. You will need it to create your code's version.
+Once you're done, you will need to install ``devflow``. Devflow is a tool that
+helps you to manage package versions, implement the git flow development
+process and build Python and Debian packages. You will need it to create your
+code's version.
.. code-block:: console
# pip install devflow
-**5. Get Synnefo code**
-First you need to install git
+**3. Clone Synnefo repo**
+
+First, you need to install ``git``:
.. code-block:: console
# apt-get install git
-And now get the Synnefo code from the official Synnefo repository
+and clone the Synnefo repo, preferably as non-privileged user:
.. code-block:: console
- # su some_regular_user
- $ git clone https://code.grnet.gr/git/synnefo
+ $ git clone https://github.com/grnet/synnefo.git
-Make sure you clone the repository as a regular user. Otherwise you will
-have problems with file permissions when deploying.
-**6. Code and deploy**
+**4. Configure the code's version**
-1. Configure the version
+Enter the directory where you have cloned the Synnefo repository and issue the
+following command:
.. code-block:: console
$ devflow-update-version
-2. Code
-3. In every component you change, run as root
+
+Synception (snf-ci)
+-------------------
+
+**1. Install Kamaki**
+
+Kamaki is a cli tool for managing clouds. It implements the Openstack APIs and
+can be used to create VMs, networks and generally allow the user to do what
+he/she would normally do from the cloud's UI.
+
+If you haven't installed ``kamaki`` yet, you can follow `these
+`_ instructions.
+
+Once you have ``kamaki`` installed, you need to connect it to your account on
+an existing Synnefo installation (called *cloud* in kamaki lingo). To do so,
+you can consult the `Setup
+`_ section.
+Alternative, you can visit the **API access** page of your account in that
+cloud and download your personalized ``.kamakirc`` file.
+
+**2. Clone Synnefo repo**
+
+First you need to install ``git`` and ``fabric`` (the latter is used internally
+by the ``snf-ci`` script):
.. code-block:: console
- # python setup.py develop -N
+ # apt-get install git fabric
+
+and clone the Synnefo repo, preferably as non-privileged user:
+
+.. code-block:: console
-This does not automatically install dependencies, in order to avoid
-confusion with Synnefo packages installed by ``snf-deploy``. External
-dependencies have already been installed by ``snf-deploy``; if you introduce
-a new dependency, you will have to explicitly install it.
+ $ git clone https://github.com/grnet/synnefo.git
-4. You will need to restart the server with
+
+**3. Install Synnefo remotely on a VM**
+
+Enter the directory where you have cloned the Synnefo repository and then enter
+the ``ci`` folder. In this folder, you will find the ``snf-ci`` script. A
+common usage of ``snf-ci`` is the following:
.. code-block:: console
- # service gunicorn restart
+ $ ./ci/snf-ci create,build,deploy --cloud
+
+The above command will use your ``kamaki`` *cloud* that you have setup in
+**Step 1**. In this cloud, ``snf-ci`` will create a Debian Wheezy VM, checkout
+the **develop** branch from the official Synnefo repo, build the Synnefo
+packages from source and install them using ``snf-deploy``. Of course, all the
+previous actions can be tweaked with command-line arguments or configuration
+files. To see a list of possible command-line arguments, you can use ``snf-ci
+-h``. Also, you can edit the ``ci_wheezy.conf`` configuration file for more
+permanent changes.
-5. If your changes affected ``snf-dispatcher`` (from package
- ``snf-cyclades-app``) or ``snf-ganeti-eventd`` (from
- ``snf-cyclades-gtools``) you will need to restart these daemons, too.
- Since step 3 installed the former under ``/usr/local/``, you need to
- make sure that the correct version is evoked. You can override the
- version installed by ``snf-deploy`` with
+.. tip::
+
+ You can use the **--local-repo** argument to instruct snf-ci to use the
+ current branch. This means that you can install Synnefo from any branch,
+ even your own.
+
+.. tip::
+
+ You can view details for the created VM, such as IP, username, password
+ etc., by doing ``cat /tmp/ci_temp_conf``.
+
+
+Development mode
+~~~~~~~~~~~~~~~~
+
+At this point you should have a working Synnefo installation. The rest of the
+instructions will take place in that Synnefo installation.
+
+**1. Use Python's development mode**
+
+From the top directory of your Synnefo repo, you can use the following script:
.. code-block:: console
- # ln -sf /usr/local/bin/snf-dispatcher /usr/bin/snf-dispatcher
+ $ ./ci/install.sh
-and then restart the daemons
+This means that every installed ``snf-*`` package will be overridden (``python
+setup.py develop -N``) with the code that exists in the currently checked-out
+branch of the cloned Synnefo repo. If you wish to leave the development mode,
+you can use another script:
.. code-block:: console
- # service snf-dispatcher restart
- # service snf-ganeti-eventd restart
+ $ ./ci/uninstall.sh
+
+
+**2. Change Gunicorn permissions**
+
+If you have cloned the repository as root, then Gunicorn will not be able to
+read your source files, since by default its user/group permissions are
+``www-data``. However, you can change the Gunicorn permissions by editing the
+``/etc/gunicorn.d/synnefo`` configuration file and replacing every ``www-data``
+instance with ``root``.
+
+.. warning::
+
+ Gunicorn's should never run as ``root`` in production environments.
+
+Accessing the Synnefo UI
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to access the Synnefo UI through your browser, you can take a look
+at the :ref:`access-synnefo` section.
+
+Caveats regarding code changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+At this point, you would expect that every change you've made in your branch
+will be instantly visible to Synnefo. Most of the times this will be the case,
+but in certain circumstances you may need to restart a service or move files
+around:
+
+* If you have changed a Synnefo setting or Django view/url, you may need to
+ restart Gunicorn with:
+
+ .. code-block:: console
+
+ # service gunicorn restart
+
+* If your changes affected ``snf-dispatcher`` (from package
+ ``snf-cyclades-app``) or ``snf-ganeti-eventd`` (from ``snf-cyclades-gtools``)
+ you will need to restart these daemons, too.
+
+ .. code-block:: console
+
+ # service snf-dispatcher restart
+ # service snf-ganeti-eventd restart
+
+* If you have edited a static file of a Synnefo component, you will need to
+ copy it to the respective folder under ``/usr/share/synnefo/static/``.
+
+
+Logs
+~~~~
+
+You will find useful information for debugging in the following files:
+
+* ``/var/log/gunicorn/synnefo.log``, for most Synnefo components.
+* ``/var/log/apache2/other_vhosts_access.log``, for issues with site
+ configuration.
+* ``/var/log/postgresql/postgresql-9.1-main.log``, for database issues.
+
+
+Testing
+~~~~~~~
+
+Synnefo has two main testing endpoints:
+
+* The ``snf-ci`` script can create a VM from a custom branch and run all
+ available tests on it.
+* The ``./ci/tests.sh`` script can be used in an existing Synnefo installation
+ to run all or specific tests. You can use it as following:
+
+ .. code-block:: console
+
+ $ ./ci/tests.sh [--dry-run] component1[.test] component2[.test] ...
+
+
+Developer Guidelines
+~~~~~~~~~~~~~~~~~~~~
+
+* If you want to use transactions in your code, you **must NOT** use the
+ default Django transactions (see the :ref:`Synnefo transactions design doc
+ `). Depending on the models you want to edit, you must
+ import the corresponding transaction implementation. For Astakos models do:
+
+ .. code-block:: python
+
+ from astakos.im import transaction
+
+
+ while for Cyclades models do:
+
+ .. code-block:: python
-6. Refresh the web page and see your changes
+ from synnefo.db import transaction
diff --git a/docs/identity-api-guide.rst b/docs/identity-api-guide.rst
index 94e459cafb9a4dfaf872c4b0fe4f5eb9d8f3b099..146771de1a09eb4706db33888ae2694c41fbb869 100644
--- a/docs/identity-api-guide.rst
+++ b/docs/identity-api-guide.rst
@@ -126,9 +126,6 @@ Return Code Description
500 (Internal Server Error) The request cannot be completed because of an internal error
=========================== =====================
-.. warning:: The service is also available under ``/feedback``.
- It will be removed in the next version.
-
Get User catalogs
^^^^^^^^^^^^^^^^^
@@ -180,9 +177,6 @@ Return Code Description
500 (Internal Server Error) The request cannot be completed because of an internal error
=========================== =====================
-.. warning:: The service is also available under ``/user_catalogs``.
- It will be removed in the next version.
-
Service API Operations
----------------------
@@ -243,9 +237,6 @@ Return Code Description
500 (Internal Server Error) The request cannot be completed because of an internal error
=========================== =====================
-.. warning:: The service is also available under ``/service/api/user_catalogs``.
- It will be removed in the next version.
-
Tokens API Operations
----------------------
diff --git a/docs/image-api-guide.rst b/docs/image-api-guide.rst
index 9756942fa054fc52a89dba1644dfe4f1fb3d4b99..a19d771be76032ed203d181d3e0244c30baccd22 100644
--- a/docs/image-api-guide.rst
+++ b/docs/image-api-guide.rst
@@ -48,6 +48,17 @@ identity credentials.
`_,
with the only difference being the suggested identity manager.
+Image Metadata Format
+---------------------
+
+In Cyclades Image API all image metadata are viewed as HTTP headers that are
+starting with the `x-image-meta-` prefix. All metadata must be encoded with the
+`UTF-8` encoding. Since the image metadata must be valid HTTP headers, user
+defined metadata like the image's name or properties must also be properly
+quoted. Finally, image properties that are viewed as HTTP headers and are
+starting with the `x-image-meta-property-` prefix, are not case-sensitive and
+all punctuation characters will be replaced with underscore.
+
List Available Images
---------------------
diff --git a/docs/images/available-attribute.png b/docs/images/available-attribute.png
new file mode 100644
index 0000000000000000000000000000000000000000..af5038210768bd244200b7f0936105225546710a
Binary files /dev/null and b/docs/images/available-attribute.png differ
diff --git a/docs/images/create-snapshot.png b/docs/images/create-snapshot.png
new file mode 100644
index 0000000000000000000000000000000000000000..f4303f87cdf94627e37933efc693ccf3a043eacb
Binary files /dev/null and b/docs/images/create-snapshot.png differ
diff --git a/docs/images/spawn-from-snapshot.png b/docs/images/spawn-from-snapshot.png
new file mode 100644
index 0000000000000000000000000000000000000000..5c48675d88944d283bc3005b5273823bcba42d78
Binary files /dev/null and b/docs/images/spawn-from-snapshot.png differ
diff --git a/docs/index.rst b/docs/index.rst
index 1755477d8f83f8190091f1b3fa24117284f35fe9..5e540419a3dce4fe5296bc05bec702e88cb71126 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -3,27 +3,30 @@ Welcome to the Synnefo documentation
.. image:: /images/synnefo-logo.png
-Synnefo is a complete open source cloud stack written in Python that provides
-Compute, Network, Image, Volume and Storage services, similar to the ones
-offered by AWS. Synnefo manages multiple `Ganeti
-`_ clusters at the backend for handling
+Synnefo is a complete open source IaaS cloud stack written in Python that
+provides Compute, Network, Image, Volume and Object Storage services, similar to
+the ones offered by AWS. Synnefo manages multiple `Ganeti
+`_ clusters at the backend for handling of
low-level VM operations and uses `Archipelago
`_ to unify cloud storage. To
boost 3rd-party compatibility, Synnefo exposes the OpenStack APIs to users.
-Synnefo powers GRNET's `~okeanos public cloud service
+Synnefo came out of GRNET's `~okeanos public cloud service
`_ and you can try it out live at `demo.synnefo.org
`_.
-Synnefo has three main components providing the corresponding services:
+Synnefo has three main components which provide the corresponding services:
.. toctree::
:maxdepth: 1
- Astakos: Identity/Account services
- Pithos: File/Object Storage service
+ Astakos: Identity/Account/Quota services
+ Pithos: Object Storage service
Cyclades: Compute/Network/Image/Volume services
+It :ref:`unifies storage resources ` (Objects/Volumes/Images/Snapshots)
+using Archipelago as the common storage substrate for all services.
+
This is an overview of the Synnefo services:
.. image:: images/synnefo-overview.png
@@ -127,6 +130,17 @@ There are also the following tools:
Design
======
+v0.16
+-----
+
+.. toctree::
+ :maxdepth: 1
+
+ Logging mechanism for Synnefo's management commands
+ Resource-pool projects design
+ Volumes design
+ Volume Snapshots design
+
v0.15
-----
@@ -143,7 +157,7 @@ Drafts
.. toctree::
:maxdepth: 1
- Resource-pool projects design
+ Multi-DB transactions in Django
Contact
diff --git a/docs/install-guide-centos.rst b/docs/install-guide-centos.rst
index 5e7d2611e3f5f035cab8f2a15b32aea2cd4cb990..93f7add699c26d13eedfc675698490a2752c6c87 100644
--- a/docs/install-guide-centos.rst
+++ b/docs/install-guide-centos.rst
@@ -3,6 +3,13 @@
Administrator's Installation Guide (on CentOS 6.5)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. warning::
+
+ Currently, we don't provide packages for release candidate versions of
+ Synnefo for CentOS 6.5. Please consult the docs for `Synnefo 0.15.2
+ `_
+ instead.
+
This is the Administrator's installation guide on CentOS 6.5.
It describes how to install the whole Synnefo stack on two (2) physical nodes,
@@ -25,13 +32,6 @@ guide and just stop after the "Testing of Pithos" section.
Installation of Synnefo / Introduction
======================================
-We will install the services with the above list's order. The last three
-services will be installed in a single step (at the end), because at the moment
-they are contained in the same software component (Cyclades). Furthermore, we
-will install all services in the first physical node, except Pithos which will
-be installed in the second, due to a conflict between the snf-pithos-app and
-snf-cyclades-app component (scheduled to be fixed in the next version).
-
For the rest of the documentation we will refer to the first physical node as
"node1" and the second as "node2". We will also assume that their domain names
are "node1.example.com" and "node2.example.com" and their public IPs are "203.0.113.1" and
@@ -79,9 +79,9 @@ You also need a shared directory visible by both nodes. Pithos will save all
data inside this directory. By 'all data', we mean files, images, and Pithos
specific mapping data. If you plan to upload more than one basic image, this
directory should have at least 50GB of free space. During this guide, we will
-assume that node1 acts as an NFS server and serves the directory ``/srv/pithos``
+assume that node1 acts as an NFS server and serves the directory ``/srv/arhip``
to node2 (be sure to set no_root_squash flag). Node2 has this directory
-mounted under ``/srv/pithos``, too.
+mounted under ``/srv/arhip``, too.
Before starting the Synnefo installation, you will need basic third party
software to be installed and configured on the physical nodes. We will describe
@@ -107,6 +107,7 @@ General Synnefo dependencies
* ntp (NTP daemon)
* gevent
* dnsmasq (DNS server)
+ * Archipelago
You can install apache, ntp and rabbitmq by running:
@@ -300,33 +301,76 @@ exchanges:
We do not need to initialize the exchanges. This will be done automatically,
during the Cyclades setup.
-Pithos data directory setup
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-As mentioned in the General Prerequisites section, there should be a directory
-called ``/srv/pithos`` visible by both nodes. We create and setup the ``data``
-directory inside it:
+System user/group setup
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Before we continue with the installation we have to mention the user and
+group that our components will run as. In short Archipelago (and
+specifically the ``archipelago`` package) creates the ``archipelago``
+system user and group while synnefo (and specifically the ``snf-common``
+package) creates the ``synnefo`` system user and group.
+
+This guide uses NFS for Archipelago's physical storage backend.
+Archipelago must have permissions to write on the shared dir. As
+explained below the shared dir will be owned by ``archipelago:synnefo``.
+Due to NFS restrictions, all nodes nodes must have common uid for the
+``archipelago`` user and common gid for the ``synnefo`` group. So before
+any Synnefo installation, we create them here in advance. We assume that
+ids 200 and 300 are available across all nodes.
.. code-block:: console
- # mkdir /srv/pithos
- # cd /srv/pithos
- # mkdir data
- # chown apache:apache data
- # chmod g+ws data
+ # addgroup --system --gid 200 synnefo
+ # adduser --system --uid 200 --gid 200 --no-create-home \
+ --gecos Synnefo synnefo
+
+ # addgroup --system --gid 300 archipelago
+ # adduser --system --uid 300 --gid 300 --no-create-home \
+ --gecos Archipelago archipelago
-This directory must be shared via `NFS `_.
-In order to do this, run:
+
+
+NFS data directory setup
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Archipelago directory must be shared via
+`NFS `_.
+As mentioned in the General Prerequisites section, there should be a
+directory called ``/srv/archip/`` with ``blocks``, ``maps``, and
+``locks`` subdirectories visible by both nodes. To create it run:
+
+.. code-block:: console
+
+ # mkdir /srv/archip/
+ # cd /srv/archip/
+ # mkdir -p {maps,blocks,locks}
+
+Currently Archipelago is the only one that needs to have access to the
+backing store. We could have the whole NFS isolated from Synnefo (owned
+by ``archipelago:archipelago`` with ``640`` access permissions) but we
+choose not to (e.g. some future extension could require access to the
+backing store directly from Synnefo). Thus we set the ownership to
+``archipelago:synnefo`` and access permissions to ``g+ws``.
.. code-block:: console
- # yum install rpcbind nfs-utils
+ # cd /srv/archip
+ # chown archipelago:synnefo {maps,blocks,locks}
+ # chmod 770 {maps,blocks,locks}
+ # chmod g+s {maps,blocks,locks}
+
+In order to install the NFS server, run:
+
+.. code-block:: console
+
+ # yum install rpcbind nfs-kernel-server
Now edit ``/etc/exports`` and add the following line:
.. code-block:: console
- /srv/pithos/ 203.0.113.2(rw,no_root_squash,sync,subtree_check)
+ /srv/archip/ 203.0.113.2(rw,no_root_squash,sync,subtree_check)
Once done, run:
@@ -335,6 +379,45 @@ Once done, run:
# service rpcbind restart
# service nfs restart
+Archipelago setup
+~~~~~~~~~~~~~~~~~
+
+To install Archipelago, run:
+
+.. code-block:: console
+
+ # yum install archipelago
+
+Now edit ``/etc/archipelago/archipelago.conf`` and tweak the following settings:
+
+* ``USER``: Let Archipelago run as ``archipelago`` user (default)
+
+* ``GROUP``: Let Archipelago run as ``synnefo`` group (archipelago by default)
+
+* ``SEGMENT_SIZE``: Adjust shared memory segment size according to your machine's
+ RAM. The default value is 2GB which in some situations might exceed your
+ machine's physical RAM. Consult also with `Archipelago administrator's guide
+ `_ for an
+ appropriate value.
+
+Adjust the following settings of ``blockerb`` and ``blockerm`` to point to
+their corresponding directories.
+
+In section ``blockerb`` set:
+
+* ``archip_dir``: ``/srv/archip/blocks``
+
+In section ``blockerm`` set:
+
+* ``archip_dir``: ``/srv/archip/maps``
+* ``lock_dir``: ``/srv/archip/locks``
+
+Finally, start Archipelago:
+
+.. code-block:: console
+
+ # archipelago restart
+
DNS server setup
~~~~~~~~~~~~~~~~
@@ -346,7 +429,7 @@ In order to set up a dns server using dnsmasq do the following:
# yum install dnsmasq
-Then edit your ``/etc/hosts/`` file as follows:
+Then edit your ``/etc/hosts`` file as follows:
.. code-block:: console
@@ -405,6 +488,7 @@ General Synnefo dependencies
* gevent
* certificates
* dnsmasq (DNS server)
+ * Archipelago
You can install the above by running:
@@ -553,10 +637,8 @@ Copy the file ``/etc/gunicorn.d/synnefo.example`` to
.. warning:: Do NOT start the server yet, because it won't find the
- ``synnefo.settings`` module. Also, in case you are using ``/etc/hosts``
- instead of a DNS to get the hostnames, change ``--worker-class=gevent`` to
- ``--worker-class=sync``. We will start the server after successful
- installation of Astakos. If the server is running::
+ ``synnefo.settings`` module. Also set the Gunicorn config file to
+ ``--config=/etc/synnefo/gunicorn-hooks/gunicorn-archipelago.py``.
# service gunicorn stop
@@ -665,18 +747,19 @@ notify administrators with a notice that a new account has just been verified.
More specifically Astakos sends emails in the following cases
- An email containing a verification link after each signup process.
-- An email to the people listed in ``ADMINS`` setting after each email
- verification if ``ASTAKOS_MODERATION`` setting is ``True``. The email
- notifies administrators that an additional action is required in order to
- activate the user.
-- A welcome email to the user email and an admin notification to ``ADMINS``
- right after each account activation.
-- Feedback messages submited from Astakos contact view and Astakos feedback
- API endpoint are sent to contacts listed in ``HELPDESK`` setting.
-- Project application request notifications to people included in ``HELPDESK``
- and ``MANAGERS`` settings.
+- An email to the people listed in ``ACCOUNT_NOTIFICATIONS_RECIPIENTS``
+ setting after each email verification if ``ASTAKOS_MODERATION`` setting is
+ ``True``. The email notifies administrators that an additional action is
+ required in order to activate the user.
+- A welcome email to the user email and a notification to
+ ``ACCOUNT_NOTIFICATIONS_RECIPIENTS`` right after each account activation.
+- Feedback messages submitted from Astakos contact view and Astakos feedback
+ API endpoint are sent to contacts listed in
+ ``FEEDBACK_NOTIFICATIONS_RECIPIENTS`` setting.
+- Project application request notifications to people included in
+ ``PROJECT_NOTIFICATIONS_RECIPIENTS`` setting.
- Notifications after each project members action (join request, membership
- accepted/declinde etc.) to project members or project owners.
+ accepted/declined etc.) to project members or project owners.
Astakos uses the Django internal email delivering mechanism to send email
notifications. A simple configuration, using an external smtp server to
@@ -713,11 +796,13 @@ Refer to
`Django documentation `_
for additional information on available email settings.
-As refered in the previous section, based on the operation that triggers
-an email notification, the recipients list differs. Specifically, for
-emails whose recipients include contacts from your service team
-(administrators, managers, helpdesk etc) Synnefo provides the following
-settings located in ``00-snf-common-admins.conf``:
+As referred in the previous section, based on the operation that triggers an
+email notification, the recipients list differs. For convenience (and backward
+compatibility), Astakos defines three service teams (administrators, managers
+and helpdesk) and send the above notifications to these teams in a
+preconfigured way (ie. project notifications are sent to the members of
+managers and helpdesk teams). These settings are located in
+``00-snf-common-admins.conf``:
.. code-block:: python
@@ -877,15 +962,17 @@ Notice that in this installation astakos and cyclades are in node1 and pithos is
Setting Default Base Quota for Resources
----------------------------------------
-We now have to specify the limit on resources that each user can employ
-(exempting resources offered by projects). When specifying storage or
-memory size limits you can append a unit to the value, i.e. 10240 MB,
-10 GB etc. Use the special value ``inf``, if you don't want to restrict a
-resource.
+All resources are registered with unlimited quota. We now have to restrict
+the limit on the resources we wish to control. We can set the default quota
+a new user is offered by the system (`system default`) with
.. code-block:: console
- # snf-manage resource-modify --default-quota-interactive
+ # snf-manage resource-modify --system-default
+
+When specifying storage or memory size limits you can append a unit to the
+value, i.e. 10240 MB, 10 GB etc. Use the special value ``inf``, if you don't
+want to restrict a resource.
Setting Resource Visibility
---------------------------
@@ -1008,6 +1095,42 @@ This package provides the standalone Pithos web client. The web client is the
web UI for Pithos and will be accessible by clicking "Pithos" on the Astakos
interface's cloudbar, at the top of the Astakos homepage.
+Installation of Archipelago on node 2
+=====================================
+
+To install Archipelago, run:
+
+.. code-block:: console
+
+ # yum install archipelago
+
+
+Now edit ``/etc/archipelago/archipelago.conf`` and tweak the following settings:
+
+* ``USER``: Let Archipelago run as ``archipelago`` user (default)
+
+* ``GROUP``: Let Archipelago run as ``synnefo`` group (defaults to archipelago)
+
+* ``SEGMENT_SIZE``: Adjust shared memory segment size according to your machine's
+ RAM. The default value is 2GB which in some situations might exceed your
+ machine's physical RAM. Consult also with `Archipelago administrator's guide
+ `_ for an
+ appropriate value.
+
+In section ``blockerb`` set:
+
+* ``archip_dir``: ``/srv/arhip/blocks``
+
+In section ``blockerm`` set:
+
+* ``archip_dir``: ``/srv/arhip/maps``
+* ``lock_dir``: ``/srv/arhip/locks``
+
+Finally, restart Archipelago:
+
+.. code-block:: console
+
+ # service archipelago restart
.. _conf-pithos:
@@ -1027,10 +1150,8 @@ Copy the file ``/etc/gunicorn.d/synnefo.example`` to
.. warning:: Do NOT start the server yet, because it won't find the
- ``synnefo.settings`` module. Also, in case you are using ``/etc/hosts``
- instead of a DNS to get the hostnames, change ``--worker-class=gevent`` to
- ``--worker-class=sync``. We will start the server after successful
- installation of Astakos. If the server is running::
+ ``synnefo.settings`` module. Also set the Gunicorn config file to
+ ``--config=/etc/synnefo/gunicorn-hooks/gunicorn-archipelago.py``.
# service gunicorn stop
@@ -1052,7 +1173,6 @@ this options:
PITHOS_BASE_URL = 'https://node2.example.com/pithos'
PITHOS_BACKEND_DB_CONNECTION = 'postgresql://synnefo:example_passw0rd@node1.example.com:5432/snf_pithos'
- PITHOS_BACKEND_BLOCK_PATH = '/srv/pithos/data'
PITHOS_SERVICE_TOKEN = 'pithos_service_token22w'
@@ -1063,11 +1183,6 @@ find the Pithos backend database. Above we tell Pithos that its database is
``example_passw0rd``. All those settings where setup during node1's "Database
setup" section.
-The ``PITHOS_BACKEND_BLOCK_PATH`` option tells to the Pithos app where to find
-the Pithos backend data. Above we tell Pithos to store its data under
-``/srv/pithos/data``, which is visible by both nodes. We have already setup this
-directory at node1's "Pithos data directory setup" section.
-
The ``ASTAKOS_AUTH_URL`` option informs the Pithos app where Astakos is.
The Astakos service is used for user management (authentication, quotas, etc.)
@@ -1152,12 +1267,12 @@ First install the package nfs-common by running:
root@node2:~ # yum install nfs-utils
-now create the directory /srv/pithos/ and mount the remote directory to it:
+Now create the directory /srv/arhip/ and mount the remote directory to it:
.. code-block:: console
- root@node2:~ # mkdir /srv/pithos/
- root@node2:~ # mount -t nfs 203.0.113.1:/srv/pithos/ /srv/pithos/
+ root@node2:~ # mkdir /srv/arhip/
+ root@node2:~ # mount -t nfs 203.0.113.1:/srv/arhip/ /srv/arhip/
Servers Initialization
----------------------
@@ -1360,28 +1475,36 @@ Ganeti nodes:
It's time to install Ganeti. To be able to use hotplug (which will be part of
the official Ganeti 2.10), we recommend using our Ganeti package version:
-``2.8.4+snap1+b64v1+kvm2+ext1+lockfix1+ipfix1+ifdown1+backports5-1``
+``2.10.7+snap1+b64v1+ext1+lockfix1+ifdown1+qmp1+bpo1-1``
Let's briefly explain each patch set:
- * snap adds snapshot support for ext disk template
+ * snap extends snapshot support for the ext disk template (separate LU)
* b64 saves networks' bitarrays in a more compact representation
- * kvm adds migration_caps hypervisor param
* ext
- * exports logical id in hooks
* allows arbitrary params to reach kvm command (i.e. cache overrides
disk_cache hvparam, heads and secs define the disk's geometry)
* lockfix is a workaround for Issue #621
- * ipfix does not require IP if mode is routed (needed for IPv6 only NICs)
* ifdown cleans up node's configuration upon instance migration/shutdown
- * backports is a set of patches backported from stable-2.10
+ * qmp replace HMP with QMP commands during hotplug
+ * bpo is a set of patches backported from later branches
+
+ * Make name and uuid Disk attributes reach bdev (2.11)
+ * IDiskParams fixes (2.11)
+ * Proper support for the --cdrom option (2.12)
+ * Add migration capabilities as an hvparam (2.13)
+ * Convert hv_kvm to a package (2.12)
+ * Extend QMP support (2.12)
+ * Add access to IDiskParams (2.13)
+ * Support userspace access for ExtStorage (2.13)
+ * Allow NICs with routed mode and no IP (2.13)
+ * Add support for KVM multiqueue virtio-net (2.12)
+ * Support Snapshot() for the ExtStorage interface (2.13)
+ * Support disk hotplug even with chroot or SM (2.13)
+ * Some refactor wrt NICs at the HV level (2.12)
- * Hotplug support
- * Better networking support (NIC configuration scripts)
- * Change IP pool to support NAT instances
- * Change RAPI to accept depends body argument and shutdown_timeout
.. note::
@@ -1398,7 +1521,7 @@ Ganeti will make use of drbd. To install drbd, you're gonna need to use packages
from the `ELRepo `_. To install ELRepo,
run:
-.. code-block:: consolse
+.. code-block:: console
# rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
@@ -1479,18 +1602,11 @@ Configuration
~~~~~~~~~~~~~
snf-image supports native access to Images stored on Pithos. This means that
it can talk directly to the Pithos backend, without the need of providing a
-public URL. More details, are described in the next section. For now, the only
-thing we need to do, is configure snf-image to access our Pithos backend.
-
-To do this, we need to set the corresponding variable in
-``/etc/default/snf-image``, to reflect our Pithos setup:
-
-.. code-block:: console
-
- PITHOS_DATA="/srv/pithos/data"
+public URL. More details, are described in the next section.
If you have installed your Ganeti cluster on different nodes than node1 and
-node2 make sure that ``/srv/pithos/data`` is visible by all of them.
+node2 make sure that ``/srv/arhip/`` is visible by all of them and
+Archipelago is installed and configured properly.
If you would like to use Images that are also/only stored locally, you need to
save them under ``IMAGE_DIR``, however this guide targets Images stored only on
@@ -2005,7 +2121,7 @@ Edit ``/etc/synnefo/20-snf-cyclades-app-plankton.conf``:
.. code-block:: console
BACKEND_DB_CONNECTION = 'postgresql://synnefo:example_passw0rd@node1.example.com:5432/snf_pithos'
- BACKEND_BLOCK_PATH = '/srv/pithos/data/'
+ BACKEND_BLOCK_PATH = '/srv/arhip/'
In this file we configure the Image Service. ``BACKEND_DB_CONNECTION``
denotes the Pithos database (where the Image files are stored). So we set that
@@ -2040,7 +2156,7 @@ Configure the vncauthproxy settings in
.. code-block:: console
- CYCLADES_VNCAUTHPROXY_OPTS = {
+ CYCLADES_VNCAUTHPROXY_OPTS = [{
'auth_user': 'synnefo',
'auth_password': 'secret_password',
'server_address': '127.0.0.1',
@@ -2048,13 +2164,32 @@ Configure the vncauthproxy settings in
'enable_ssl': False,
'ca_cert': None,
'strict': False,
- }
+ }]
Depending on your snf-vncauthproxy setup, you might want to tweak the above
settings. Check the `documentation
`_ of
snf-vncauthproxy for more information.
+You should also provide snf-vncauthproxy with SSL certificates signed by a
+trusted CA. You can either copy them to `/var/lib/vncauthproxy/{cert,key}.pem`
+or inform vncauthproxy about the location of the certificates (via the
+`DAEMON_OPTS` setting in `/etc/default/vncauthproxy`).
+
+::
+
+ DAEMON_OPTS="--pid-file=$PIDFILE --cert-file= --key-file="
+
+Both files should be readable by the `vncauthproxy` user or group.
+
+.. note::
+
+ When installing snf-vncauthproxy on the same node as Cyclades and using the
+ default settings for snf-vncauthproxy, the certificates should be issued to
+ the FQDN of the Cyclades worker. Refer to the :ref:`admin guide
+ `, for more information on how to setup vncauthproxy on a
+ different host / interface.
+
We have now finished with the basic Cyclades configuration.
Database Initialization
@@ -2095,7 +2230,7 @@ You can see everything has been setup correctly by running:
Enable the new backend by running:
-.. code-block::
+.. code-block:: console
$ snf-manage backend-modify --drained False 1
@@ -2455,5 +2590,51 @@ to state 'Running' and you will be able to use it. Click 'Console' to connect
through VNC out of band, or click on the machine's icon to connect directly via
SSH or RDP (for windows machines).
+
+Installation of Admin on node1
+==============================
+
+This section describes the installation of Admin. Admin is a Synnefo component
+that provides to trusted users the ability to manage and view various different
+Synnefo entities such as users, VMs, projects etc.
+
+We will install Admin on node1. To do so, we install the corresponding
+package by running on node1 the following command:
+
+.. code-block:: console
+
+ # yum install snf-admin-app
+
+Once the package is installed, we must configure the ``ADMIN_BASE_URL``
+setting. This setting is located in the ``20-snf-admin-app-general.conf``
+settings file. Uncomment it and assign the following URL to it:
+
+ ``https://node1.example.com/admin``
+
+Now, we can proceed with testing Admin.
+
+Testing of Admin
+================
+
+In order to test the Admin Dashboard, we need a user that belongs to the
+`admin` group. We will use the user that was created in `Testing of Astakos`_
+section:
+
+.. code-block:: console
+
+ root@node1:~ # snf-manage group-add admin
+ root@node1:~ # snf-manage user-modify 1 --add-group=admin
+
+Then, you need to login to the Astakos node by visiting the following URL:
+
+ ``https://node1.example.com/astakos``
+
+Once you login successfully, you can access the Admin Dashboard from this URL:
+
+ ``https://node1.example.com/admin``
+
+This should redirect you to the **Users** table, where there should be an entry
+with this user.
+
Congratulations. You have successfully installed the whole Synnefo stack and
connected all components.
diff --git a/docs/install-guide-debian.rst b/docs/install-guide-debian.rst
index 971751059d88720e58d2357d6293f8da5b10e825..5a54014ce2d8c101cab272ed0018e89e56d714a1 100644
--- a/docs/install-guide-debian.rst
+++ b/docs/install-guide-debian.rst
@@ -25,13 +25,6 @@ guide and just stop after the "Testing of Pithos" section.
Installation of Synnefo / Introduction
======================================
-We will install the services with the above list's order. The last three
-services will be installed in a single step (at the end), because at the moment
-they are contained in the same software component (Cyclades). Furthermore, we
-will install all services in the first physical node, except Pithos which will
-be installed in the second, due to a conflict between the snf-pithos-app and
-snf-cyclades-app component (scheduled to be fixed in the next version).
-
For the rest of the documentation we will refer to the first physical node as
"node1" and the second as "node2". We will also assume that their domain names
are "node1.example.com" and "node2.example.com" and their public IPs are "203.0.113.1" and
@@ -62,13 +55,13 @@ Update your list of packages and continue with the installation:
# apt-get update
-You also need a shared directory visible by both nodes. Pithos will save all
-data inside this directory. By 'all data', we mean files, images, and Pithos
-specific mapping data. If you plan to upload more than one basic image, this
-directory should have at least 50GB of free space. During this guide, we will
-assume that node1 acts as an NFS server and serves the directory ``/srv/pithos``
-to node2 (be sure to set no_root_squash flag). Node2 has this directory
-mounted under ``/srv/pithos``, too.
+From version 0.16 Synnefo is backed by Archipelago and uses it to store all of
+its data. In this guide, we will use NFS as a storage backend for Archipelago.
+If you plan to upload more than one basic image, this directory should have at
+least 50GB of free space. During this guide, we will assume that node1 acts as
+an NFS server and serves the directory ``/srv/archip/`` to node2 (be sure to set
+no_root_squash flag). Node2 has this directory mounted under ``/srv/archip/``,
+too.
Before starting the Synnefo installation, you will need basic third party
software to be installed and configured on the physical nodes. We will describe
@@ -94,6 +87,7 @@ General Synnefo dependencies
* ntp (NTP daemon)
* gevent
* dnsmasq (DNS server)
+ * Archipelago
You can install apache2, postgresql, ntp and rabbitmq by running:
@@ -313,23 +307,65 @@ exchanges:
We do not need to initialize the exchanges. This will be done automatically,
during the Cyclades setup.
-Pithos data directory setup
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-As mentioned in the General Prerequisites section, there should be a directory
-called ``/srv/pithos`` visible by both nodes. We create and setup the ``data``
-directory inside it:
+System user/group setup
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Before we continue with the installation we have to mention the user and
+group that our components will run as. In short Archipelago (and
+specifically the ``archipelago`` package) creates the ``archipelago``
+system user and group while synnefo (and specifically the ``snf-common``
+package) creates the ``synnefo`` system user and group.
+
+This guide uses NFS for Archipelago's physical storage backend.
+Archipelago must have permissions to write on the shared dir. As
+explained below the shared dir will be owned by ``archipelago:synnefo``.
+Due to NFS restrictions, all nodes nodes must have common uid for the
+``archipelago`` user and common gid for the ``synnefo`` group. So before
+any Synnefo installation, we create them here in advance. We assume that
+ids 200 and 300 are available across all nodes.
.. code-block:: console
- # mkdir /srv/pithos
- # cd /srv/pithos
- # mkdir data
- # chown www-data:www-data data
- # chmod g+ws data
+ # addgroup --system --gid 200 synnefo
+ # adduser --system --uid 200 --gid 200 --no-create-home \
+ --gecos Synnefo synnefo
-This directory must be shared via `NFS `_.
-In order to do this, run:
+ # addgroup --system --gid 300 archipelago
+ # adduser --system --uid 300 --gid 300 --no-create-home \
+ --gecos Archipelago archipelago
+
+
+NFS data directory setup
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Archipelago directory must be shared via
+`NFS `_.
+As mentioned in the General Prerequisites section, there should be a
+directory called ``/srv/archip/`` with ``blocks``, ``maps``, and
+``locks`` subdirectories visible by both nodes. To create it run:
+
+.. code-block:: console
+
+ # mkdir /srv/archip/
+ # cd /srv/archip/
+ # mkdir -p {maps,blocks,locks}
+
+Currently Archipelago is the only one that needs to have access to the
+backing store. We could have the whole NFS isolated from Synnefo (owned
+by ``archipelago:archipelago`` with ``640`` access permissions) but we
+choose not to (e.g. some future extension could require access to the
+backing store directly from Synnefo). Thus we set the ownership to
+``archipelago:synnefo`` and access permissions to ``g+ws``.
+
+.. code-block:: console
+
+ # cd /srv/archip
+ # chown archipelago:synnefo {maps,blocks,locks}
+ # chmod 770 {maps,blocks,locks}
+ # chmod g+s {maps,blocks,locks}
+
+In order to install the NFS server, run:
.. code-block:: console
@@ -339,7 +375,7 @@ Now edit ``/etc/exports`` and add the following line:
.. code-block:: console
- /srv/pithos/ 203.0.113.2(rw,no_root_squash,sync,subtree_check)
+ /srv/archip/ 203.0.113.2(rw,no_root_squash,sync,subtree_check)
Once done, run:
@@ -347,6 +383,46 @@ Once done, run:
# /etc/init.d/nfs-kernel-server restart
+Archipelago setup
+~~~~~~~~~~~~~~~~~
+
+To install Archipelago, run:
+
+.. code-block:: console
+
+ root@node1:~ # apt-get install archipelago archipelago-ganeti
+ root@node1:~ # apt-get install blktap-archipelago-utils blktap-dkms
+
+Now edit ``/etc/archipelago/archipelago.conf`` and tweak the following settings:
+
+* ``USER``: Let Archipelago run as ``archipelago`` user (default)
+
+* ``GROUP``: Let Archipelago run as ``synnefo`` group (archipelago by default)
+
+* ``SEGMENT_SIZE``: Adjust shared memory segment size according to your machine's
+ RAM. The default value is 2GB which in some situations might exceed your
+ machine's physical RAM. Consult also with `Archipelago administrator's guide
+ `_ for an
+ appropriate value.
+
+Adjust the following settings of ``blockerb`` and ``blockerm`` to point to
+their corresponding directories.
+
+In section ``blockerb`` set:
+
+* ``archip_dir``: ``/srv/archip/blocks``
+
+In section ``blockerm`` set:
+
+* ``archip_dir``: ``/srv/archip/maps``
+* ``lock_dir``: ``/srv/archip/locks``
+
+Finally, start Archipelago:
+
+.. code-block:: console
+
+ root@node1:~ # /etc/init.d/archipelago start
+
DNS server setup
~~~~~~~~~~~~~~~~
@@ -358,7 +434,7 @@ In order to set up a dns server using dnsmasq do the following:
# apt-get install dnsmasq
-Then edit your ``/etc/hosts/`` file as follows:
+Then edit your ``/etc/hosts`` file as follows:
.. code-block:: console
@@ -417,6 +493,8 @@ General Synnefo dependencies
* gevent
* certificates
* dnsmasq (DNS server)
+ * NFS directory mount
+ * Archipelago
You can install the above by running:
@@ -534,6 +612,60 @@ Copy the certificate you created before on node1 (`ca.crt`) under the directory
to update the records.
+Installation of Archipelago
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To install Archipelago, run:
+
+.. code-block:: console
+
+ root@node2:~ # apt-get install archipelago archipelago-ganeti
+ root@node2:~ # apt-get install blktap-archipelago-utils blktap-dkms
+
+In order to configure Archipelago, the shared data directory must be set up.
+Make sure to mount the Archipelago directory after installing the Archipelago
+package.
+
+First install the package nfs-common by running:
+
+.. code-block:: console
+
+ root@node2:~ # apt-get install nfs-common
+
+Now create the directory /srv/archip/ and mount the remote directory to it:
+
+.. code-block:: console
+
+ root@node2:~ # mkdir /srv/archip/
+ root@node2:~ # mount -t nfs 203.0.113.1:/srv/archip/ /srv/archip/
+
+Now edit ``/etc/archipelago/archipelago.conf`` and tweak the following settings:
+
+* ``SEGMENT_SIZE``: Adjust shared memory segment size according to your machine's
+ RAM. The default value is 2GB which in some situations might exceed your
+ machine's physical RAM. Consult also with `Archipelago administrator's guide
+ `_ for an
+ appropriate value.
+
+Adjust the following settings of ``blockerb`` and ``blockerm`` to point to
+their corresponding directories.
+
+In section ``blockerb`` set:
+
+* ``archip_dir``: ``/srv/archip/blocks``
+
+In section ``blockerm`` set:
+
+* ``archip_dir``: ``/srv/archip/maps``
+* ``lock_dir``: ``/srv/archip/locks``
+
+Finally, start Archipelago:
+
+.. code-block:: console
+
+ root@node2:~ # /etc/init.d/archipelago start
+
+
DNS Setup
~~~~~~~~~
@@ -561,7 +693,8 @@ described previously), by running:
.. code-block:: console
- # apt-get install snf-astakos-app snf-pithos-backend
+ # apt-get install snf-astakos-app
+
.. _conf-astakos:
@@ -576,17 +709,17 @@ Copy the file ``/etc/gunicorn.d/synnefo.example`` to
.. code-block:: console
- # mv /etc/gunicorn.d/synnefo.example /etc/gunicorn.d/synnefo
+ # cp /etc/gunicorn.d/synnefo.example /etc/gunicorn.d/synnefo
.. warning:: Do NOT start the server yet, because it won't find the
- ``synnefo.settings`` module. Also, in case you are using ``/etc/hosts``
- instead of a DNS to get the hostnames, change ``--worker-class=gevent`` to
- ``--worker-class=sync``. We will start the server after successful
- installation of Astakos. If the server is running::
-
+ ``synnefo.settings`` module. We will start the server after successfully
+ installing of Astakos. If the server is running::
# /etc/init.d/gunicorn stop
+.. Also set ``--config=/etc/synnefo/gunicorn-hooks/gunicorn-archipelago.py``.
+
+
Conf Files
----------
@@ -692,18 +825,19 @@ notify administrators with a notice that a new account has just been verified.
More specifically Astakos sends emails in the following cases
- An email containing a verification link after each signup process.
-- An email to the people listed in ``ADMINS`` setting after each email
- verification if ``ASTAKOS_MODERATION`` setting is ``True``. The email
- notifies administrators that an additional action is required in order to
- activate the user.
-- A welcome email to the user email and an admin notification to ``ADMINS``
- right after each account activation.
-- Feedback messages submited from Astakos contact view and Astakos feedback
- API endpoint are sent to contacts listed in ``HELPDESK`` setting.
-- Project application request notifications to people included in ``HELPDESK``
- and ``MANAGERS`` settings.
+- An email to the people listed in ``ACCOUNT_NOTIFICATIONS_RECIPIENTS``
+ setting after each email verification if ``ASTAKOS_MODERATION`` setting is
+ ``True``. The email notifies administrators that an additional action is
+ required in order to activate the user.
+- A welcome email to the user email and a notification to
+ ``ACCOUNT_NOTIFICATIONS_RECIPIENTS`` right after each account activation.
+- Feedback messages submitted from Astakos contact view and Astakos feedback
+ API endpoint are sent to contacts listed in
+ ``FEEDBACK_NOTIFICATIONS_RECIPIENTS`` setting.
+- Project application request notifications to people included in
+ ``PROJECT_NOTIFICATIONS_RECIPIENTS`` setting.
- Notifications after each project members action (join request, membership
- accepted/declinde etc.) to project members or project owners.
+ accepted/declined etc.) to project members or project owners.
Astakos uses the Django internal email delivering mechanism to send email
notifications. A simple configuration, using an external smtp server to
@@ -740,11 +874,13 @@ Refer to
`Django documentation `_
for additional information on available email settings.
-As refered in the previous section, based on the operation that triggers
-an email notification, the recipients list differs. Specifically, for
-emails whose recipients include contacts from your service team
-(administrators, managers, helpdesk etc) synnefo provides the following
-settings located in ``00-snf-common-admins.conf``:
+As referred in the previous section, based on the operation that triggers an
+email notification, the recipients list differs. For convenience (and backward
+compatibility), Astakos defines three service teams (administrators, managers
+and helpdesk) and send the above notifications to these teams in a
+preconfigured way (ie. project notifications are sent to the members of
+managers and helpdesk teams). These settings are located in
+``00-snf-common-admins.conf``:
.. code-block:: python
@@ -904,15 +1040,17 @@ Notice that in this installation astakos and cyclades are in node1 and pithos is
Setting Default Base Quota for Resources
----------------------------------------
-We now have to specify the limit on resources that each user can employ
-(exempting resources offered by projects). When specifying storage or
-memory size limits you can append a unit to the value, i.e. 10240 MB,
-10 GB etc. Use the special value ``inf``, if you don't want to restrict a
-resource.
+All resources are registered with unlimited quota. We now have to restrict
+the limit on the resources we wish to control. We can set the default quota
+a new user is offered by the system (`system default`) with
.. code-block:: console
- # snf-manage resource-modify --default-quota-interactive
+ # snf-manage resource-modify --system-default
+
+When specifying storage or memory size limits you can append a unit to the
+value, i.e. 10240 MB, 10 GB etc. Use the special value ``inf``, if you don't
+want to restrict a resource.
Setting Resource Visibility
---------------------------
@@ -992,8 +1130,8 @@ Now we need to activate that user. Return to a command prompt at node1 and run:
root@node1:~ # snf-manage user-list
This command should show you a list with only one user; the one we just created.
-This user should have an id with a value of ``1`` and flag "active" and
-"verified" set to False. Now run:
+This user should have an id with a value of ``1`` and flag "active"
+set to False. Now run:
.. code-block:: console
@@ -1036,7 +1174,6 @@ This package provides the standalone Pithos web client. The web client is the
web UI for Pithos and will be accessible by clicking "Pithos" on the Astakos
interface's cloudbar, at the top of the Astakos homepage.
-
.. _conf-pithos:
Configuration of Pithos
@@ -1055,10 +1192,8 @@ Copy the file ``/etc/gunicorn.d/synnefo.example`` to
.. warning:: Do NOT start the server yet, because it won't find the
- ``synnefo.settings`` module. Also, in case you are using ``/etc/hosts``
- instead of a DNS to get the hostnames, change ``--worker-class=gevent`` to
- ``--worker-class=sync``. We will start the server after successful
- installation of Astakos. If the server is running::
+ ``synnefo.settings`` module. We will start the server after successful
+ installation of Pithos. If the server is running::
# /etc/init.d/gunicorn stop
@@ -1080,7 +1215,6 @@ this options:
PITHOS_BASE_URL = 'https://node2.example.com/pithos'
PITHOS_BACKEND_DB_CONNECTION = 'postgresql://synnefo:example_passw0rd@node1.example.com:5432/snf_pithos'
- PITHOS_BACKEND_BLOCK_PATH = '/srv/pithos/data'
PITHOS_SERVICE_TOKEN = 'pithos_service_token22w'
@@ -1091,11 +1225,6 @@ find the Pithos backend database. Above we tell Pithos that its database is
``example_passw0rd``. All those settings where setup during node1's "Database
setup" section.
-The ``PITHOS_BACKEND_BLOCK_PATH`` option tells to the Pithos app where to find
-the Pithos backend data. Above we tell Pithos to store its data under
-``/srv/pithos/data``, which is visible by both nodes. We have already setup this
-directory at node1's "Pithos data directory setup" section.
-
The ``ASTAKOS_AUTH_URL`` option informs the Pithos app where Astakos is.
The Astakos service is used for user management (authentication, quotas, etc.)
@@ -1156,10 +1285,29 @@ context. This means adding the following lines at the top of your
from synnefo.lib.db.psyco_gevent import make_psycopg_green
make_psycopg_green()
-Furthermore, add the ``--worker-class=gevent`` (or ``--worker-class=sync`` as
-mentioned above, depending on your setup) argument on your
-``/etc/gunicorn.d/synnefo`` configuration file. The file should look something
-like this:
+
+.. _conf-pithos-gunicorn:
+
+Pithos gunicorn configuration
+-----------------------------
+
+We also need to adjust Pithos gunicorn configuration in order to integrate with
+Archipelago. The file, as mentioned above, is located at
+``/etc/gunicorn.d/synnefo``.
+
+As of version 0.16 Pithos is backed by Archipelago. Pithos integrates with
+Archipelago via a shared memory segment that is used to communicate with the
+various Archipelago components. For more information regarding the Archipelago
+internal architecture consult with the `Archipelago administrator's guide
+`_
+
+Furthermore, we have to set the ``--config=/etc/synnefo/gunicorn-hooks/gunicorn-archipelago.py`` option.
+
+.. Furthermore, add the ``--worker-class=gevent`` (or ``--worker-class=sync`` as
+ mentioned above, depending on your setup) argument on your
+ ``/etc/gunicorn.d/synnefo`` configuration file.
+
+The file should look something like this:
.. code-block:: console
@@ -1169,17 +1317,19 @@ like this:
'DJANGO_SETTINGS_MODULE': 'synnefo.settings',
},
'working_dir': '/etc/synnefo',
- 'user': 'www-data',
- 'group': 'www-data',
+ 'user': 'synnefo',
+ 'group': 'synnefo',
'args': (
'--bind=127.0.0.1:8080',
'--workers=4',
'--worker-class=gevent',
+ '--config=/etc/synnefo/gunicorn-hooks/gunicorn-archipelago.py',
'--log-level=debug',
'--timeout=43200'
),
}
+
Stamp Database Revision
-----------------------
@@ -1195,22 +1345,6 @@ the migration history.
root@node2:~ # pithos-migrate stamp head
-Mount the NFS directory
------------------------
-
-First install the package nfs-common by running:
-
-.. code-block:: console
-
- root@node2:~ # apt-get install nfs-common
-
-now create the directory /srv/pithos/ and mount the remote directory to it:
-
-.. code-block:: console
-
- root@node2:~ # mkdir /srv/pithos/
- root@node2:~ # mount -t nfs 203.0.113.1:/srv/pithos/ /srv/pithos/
-
Servers Initialization
----------------------
@@ -1294,7 +1428,7 @@ Cyclades Prerequisites
Before proceeding with the Cyclades installation, make sure you have
successfully set up Astakos and Pithos first, because Cyclades depends on
them. If you don't have a working Astakos and Pithos installation yet, please
-return to the :ref:`top ` of this guide.
+return to the :ref:`top ` of this guide.
Besides Astakos and Pithos, you will also need a number of additional working
prerequisites, before you start the Cyclades installation.
@@ -1409,37 +1543,44 @@ Ganeti nodes:
# apt-get install qemu-kvm
-It's time to install Ganeti. To be able to use hotplug (which will be part of
-the official Ganeti 2.10), we recommend using our Ganeti package version:
+It's time to install Ganeti. We recommend using our Ganeti package version:
-``2.8.4+snap1+b64v1+kvm2+ext1+lockfix1+ipfix1+ifdown1+backports5-1~wheezy``
+``2.10.7+snap1+b64v1+ext1+lockfix1+ifdown1+qmp1+bpo1-1~wheezy``
Let's briefly explain each patch set:
- * snap adds snapshot support for ext disk template
+ * snap extends snapshot support for the ext disk template (separate LU)
* b64 saves networks' bitarrays in a more compact representation
- * kvm adds migration_caps hypervisor param
* ext
- * exports logical id in hooks
* allows arbitrary params to reach kvm command (i.e. cache overrides
disk_cache hvparam, heads and secs define the disk's geometry)
* lockfix is a workaround for Issue #621
- * ipfix does not require IP if mode is routed (needed for IPv6 only NICs)
* ifdown cleans up node's configuration upon instance migration/shutdown
- * backports is a set of patches backported from stable-2.10
+ * qmp replace HMP with QMP commands during hotplug
+ * bpo is a set of patches backported from later branches
+
+ * Make name and uuid Disk attributes reach bdev (2.11)
+ * IDiskParams fixes (2.11)
+ * Proper support for the --cdrom option (2.12)
+ * Add migration capabilities as an hvparam (2.13)
+ * Convert hv_kvm to a package (2.12)
+ * Extend QMP support (2.12)
+ * Add access to IDiskParams (2.13)
+ * Support userspace access for ExtStorage (2.13)
+ * Allow NICs with routed mode and no IP (2.13)
+ * Add support for KVM multiqueue virtio-net (2.12)
+ * Support Snapshot() for the ExtStorage interface (2.13)
+ * Support disk hotplug even with chroot or SM (2.13)
+ * Some refactor wrt NICs at the HV level (2.12)
- * Hotplug support
- * Better networking support (NIC configuration scripts)
- * Change IP pool to support NAT instances
- * Change RAPI to accept depends body argument and shutdown_timeout
To install Ganeti run:
.. code-block:: console
- # apt-get install snf-ganeti ganeti-htools ganeti-haskell ganeti2
+ # apt-get install snf-ganeti ganeti2
Ganeti will make use of drbd. To enable this and make the configuration
permanent you have to do the following :
@@ -1496,29 +1637,39 @@ to handle image files stored on Pithos. It also needs `python-psycopg2` to be
able to access the Pithos database. This is why, we also install them on *all*
VM-capable Ganeti nodes.
+
+You must set the the ``PITHCAT_UMASK`` setting of snf-image to ``007``. On the
+file ``/etc/default/snf-image`` uncomment or create the relevant setting and set
+its value to ``007``.
+
.. warning::
- snf-image uses ``curl`` for handling URLs. This means that it will
- not work out of the box if you try to use URLs served by servers which do
- not have a valid certificate. In case you haven't followed the guide's
- directions about the certificates, in order to circumvent this you should edit the file
- ``/etc/default/snf-image``. Change ``#CURL="curl"`` to ``CURL="curl -k"`` on every node.
+ snf-image uses ``curl`` for handling URLs. This means that it will
+ not work out of the box if you try to use URLs served by servers which do
+ not have a valid certificate. In case you haven't followed the guide's
+ directions about the certificates, in order to circumvent this you should
+ edit the file ``/etc/default/snf-image``. Change ``# CURL="curl"`` to
+ ``CURL="curl -k"`` on every node.
+
+.. warning::
+ If you are using qemu-kvm from wheezy-backports, note that the official
+ 2.1.0 version has a ACPI regression bug (see
+ `here `_).
+ This bug has reached the
+ `Debian qemu-kvm 2.1+dfsg-2~bpo70+2 package `_
+ found in wheezy-backports and is triggered by snf-image. Until a newer package is
+ out, you can workaround it by editing the file ``/etc/default/snf-image``
+ and changing ``# KVM="kvm"`` to ``KVM="qemu-system-x86_64 -enable-kvm -machine pc-i440fx-2.0,accel=kvm"``
+ on every node.
Configuration
~~~~~~~~~~~~~
snf-image supports native access to Images stored on Pithos. This means that
it can talk directly to the Pithos backend, without the need of providing a
-public URL. More details, are described in the next section. For now, the only
-thing we need to do, is configure snf-image to access our Pithos backend.
-
-To do this, we need to set the corresponding variable in
-``/etc/default/snf-image``, to reflect our Pithos setup:
-
-.. code-block:: console
-
- PITHOS_DATA="/srv/pithos/data"
+public URL. More details, are described in the next section.
If you have installed your Ganeti cluster on different nodes than node1 and
-node2 make sure that ``/srv/pithos/data`` is visible by all of them.
+node2 make sure that ``/srv/archip/`` is visible by all of them and
+Archipelago is installed and configured properly.
If you would like to use Images that are also/only stored locally, you need to
save them under ``IMAGE_DIR``, however this guide targets Images stored only on
@@ -1603,27 +1754,42 @@ for this image.
To spawn a VM from a Pithos file, we need to know:
- 1) The hashmap of the file
+ 1) The mapfile name of the file
2) The size of the file
-If you uploaded the file with kamaki as described above, run:
+If you uploaded the file with kamaki as described above, run on the Astakos
+node:
.. code-block:: console
- # kamaki file info pithos:debian_base-6.0-x86_64.diskdump
+ # snf-manage user-list
-else, replace ``pithos`` and ``debian_base-6.0-x86_64.diskdump`` with the
-container and filename you used, when uploading the file.
+to get a list of users. Then run the following:
-The hashmap is the field ``x-object-hash``, while the size of the file is the
-``content-length`` field, that ``kamaki file info`` command returns.
+.. code-block:: console
+
+ # snf-manage user-show 1
+
+where 1 is the id of the user that uploaded the image, as retrieved by the
+previous command. This will output the user's uuid (among others).
+
+Then on the Pithos node run the following:
+
+.. code-block:: console
+
+ # snf-manage file-show pithos debian_base-6.0-x86_64.diskdump
+
+Replace ``pithos`` and ``debian_base-6.0-x86_64.diskdump`` with the
+container and filename you used, when uploading the file.
+This will output the following info (among others): the name of the pithos
+mapfile (``mapfile`` field) and the size of the image (``bytes`` field).
Run on the :ref:`GANETI-MASTER's ` (node1) command line:
.. code-block:: console
# gnt-instance add -o snf-image+default --os-parameters \
- img_passwd=my_vm_example_passw0rd,img_format=diskdump,img_id="pithosmap:///",img_properties='{"OSFAMILY":"linux"\,"ROOT_PARTITION":"1"}' \
+ img_passwd=my_vm_example_passw0rd,img_format=diskdump,img_id="pithosmap:///",img_properties='{"OSFAMILY":"linux"\,"ROOT_PARTITION":"1"}' \
-t plain --disk 0:size=2G --no-name-check --no-ip-check \
testvm1
@@ -1958,7 +2124,7 @@ package by running on node1:
.. code-block:: console
- # apt-get install snf-cyclades-app memcached python-memcache
+ # apt-get install snf-cyclades-app memcached python-memcache snf-pithos-backend
If all packages install successfully, then Cyclades are installed and we
proceed with their configuration.
@@ -2033,12 +2199,10 @@ Edit ``/etc/synnefo/20-snf-cyclades-app-plankton.conf``:
.. code-block:: console
BACKEND_DB_CONNECTION = 'postgresql://synnefo:example_passw0rd@node1.example.com:5432/snf_pithos'
- BACKEND_BLOCK_PATH = '/srv/pithos/data/'
In this file we configure the Image Service. ``BACKEND_DB_CONNECTION``
denotes the Pithos database (where the Image files are stored). So we set that
-to point to our Pithos database. ``BACKEND_BLOCK_PATH`` denotes the actual
-Pithos data location.
+to point to our Pithos database.
Edit ``/etc/synnefo/20-snf-cyclades-app-queues.conf``:
@@ -2068,7 +2232,7 @@ Configure the vncauthproxy settings in
.. code-block:: console
- CYCLADES_VNCAUTHPROXY_OPTS = {
+ CYCLADES_VNCAUTHPROXY_OPTS = [{
'auth_user': 'synnefo',
'auth_password': 'secret_password',
'server_address': '127.0.0.1',
@@ -2076,15 +2240,49 @@ Configure the vncauthproxy settings in
'enable_ssl': False,
'ca_cert': None,
'strict': False,
- }
+ }]
+
Depending on your snf-vncauthproxy setup, you might want to tweak the above
settings. Check the `documentation
`_ of
snf-vncauthproxy for more information.
+You should also provide snf-vncauthproxy with SSL certificates signed by a
+trusted CA. You can either copy them to `/var/lib/vncauthproxy/{cert,key}.pem`
+or inform vncauthproxy about the location of the certificates (via the
+`DAEMON_OPTS` setting in `/etc/default/vncauthproxy`).
+
+::
+
+ DAEMON_OPTS="--pid-file=$PIDFILE --cert-file= --key-file="
+
+Both files should be readable by the `vncauthproxy` user or group.
+
+.. note::
+
+ When installing snf-vncauthproxy on the same node as Cyclades and using the
+ default settings for snf-vncauthproxy, the certificates should be issued to
+ the FQDN of the Cyclades worker. Refer to the :ref:`admin guide
+ `, for more information on how to setup vncauthproxy on a
+ different host / interface.
+
We have now finished with the basic Cyclades configuration.
+Gunicorn configuration
+----------------------
+
+Cyclades uses Pithos backend library to access and store system and
+user-provided images and snapshots.
+
+We need to adjust gunicorn configuration in order to integrate with
+Archipelago. Set the
+``--config=/etc/synnefo/gunicorn-hooks/gunicorn-archipelago.py`` option
+in the gunicorn configuration file located at
+``/etc/gunicorn.d/synnefo``.
+
+
+
Database Initialization
-----------------------
@@ -2123,7 +2321,7 @@ You can see everything has been setup correctly by running:
Enable the new backend by running:
-.. code-block::
+.. code-block:: console
$ snf-manage backend-modify --drained False 1
@@ -2291,7 +2489,7 @@ skipped.
.. code-block:: console
- node1 # snf-manage quota-verify --sync
+ node1 # snf-manage quota-verify --fix
node1 # snf-manage reconcile-resources-astakos --fix
node2 # snf-manage reconcile-resources-pithos --fix
node1 # snf-manage reconcile-resources-cyclades --fix
@@ -2483,5 +2681,51 @@ to state 'Running' and you will be able to use it. Click 'Console' to connect
through VNC out of band, or click on the machine's icon to connect directly via
SSH or RDP (for windows machines).
+
+Installation of Admin on node1
+==============================
+
+This section describes the installation of Admin. Admin is a Synnefo component
+that provides to trusted users the ability to manage and view various different
+Synnefo entities such as users, VMs, projects etc.
+
+We will install Admin on node1. To do so, we install the corresponding
+package by running on node1 the following command:
+
+.. code-block:: console
+
+ # apt-get install snf-admin-app
+
+Once the package is installed, we must configure the ``ADMIN_BASE_URL``
+setting. This setting is located in the ``20-snf-admin-app-general.conf``
+settings file. Uncomment it and assign the following URL to it:
+
+ ``https://node1.example.com/admin``
+
+Now, we can proceed with testing Admin.
+
+Testing of Admin
+================
+
+In order to test the Admin Dashboard, we need a user that belongs to the
+`admin` group. We will use the user that was created in `Testing of Astakos`_
+section:
+
+.. code-block:: console
+
+ root@node1:~ # snf-manage group-add admin
+ root@node1:~ # snf-manage user-modify 1 --add-group=admin
+
+Then, you need to login to the Astakos node by visiting the following URL:
+
+ ``https://node1.example.com/astakos``
+
+Once you login successfully, you can access the Admin Dashboard from this URL:
+
+ ``https://node1.example.com/admin``
+
+This should redirect you to the **Users** table, where there should be an entry
+with this user.
+
Congratulations. You have successfully installed the whole Synnefo stack and
connected all components.
diff --git a/docs/network-api-guide.rst b/docs/network-api-guide.rst
index 5074f003c1589d61e3f2cc5bd475d14af4a8a73f..05adf20fb85f866a677dc4673c6c613b670dbff2 100644
--- a/docs/network-api-guide.rst
+++ b/docs/network-api-guide.rst
@@ -21,16 +21,17 @@ API Operations
==============
.. rubric:: Networks
-===================================== ========================== ====== ======== ======= ==========
-Description URI Method Cyclades/Network OS/Neutron
-===================================== ========================== ====== ================ ==========
-`List <#list-networks>`_ ``/networks`` GET ✔ ✔
-`Get details <#get-network-details>`_ ``/networks/`` GET ✔ ✔
-`Create <#create-network>`_ ``/networks`` POST ✔ ✔
-Bulk creation ``/networks`` POST **✘** ✔
-`Update <#update-network>`_ ``/networks/`` PUT ✔ ✔
-`Delete <#delete-network>`_ ``/networks/`` DELETE ✔ ✔
-===================================== ========================== ====== ================ ==========
+===================================== ================================= ====== ======== ======= ==========
+Description URI Method Cyclades/Network OS/Neutron
+===================================== ================================= ====== ================ ==========
+`List <#list-networks>`_ ``/networks`` GET ✔ ✔
+`Get details <#get-network-details>`_ ``/networks/`` GET ✔ ✔
+`Create <#create-network>`_ ``/networks`` POST ✔ ✔
+Bulk creation ``/networks`` POST **✘** ✔
+`Update <#update-network>`_ ``/networks/`` PUT ✔ ✔
+`Delete <#delete-network>`_ ``/networks/`` DELETE ✔ ✔
+`Reassign <#reassign-network>`_ ``/networks//action`` POST ✔ **✘**
+===================================== ================================= ====== ================ ==========
.. rubric:: Subnets
==================================== ======================== ====== ======== ======= ==========
@@ -57,15 +58,16 @@ Bulk creation ``/ports`` POST **✘**
================================== ==================== ====== ================ ==========
.. rubric:: Floating IPs
-========================================= ================================ ====== ================ ==========
-Description URI Method Cyclades/Network OS/Neutron Extensions
-========================================= ================================ ====== ================ ==========
-`List <#list-floating-ips>`_ ``/floatingips`` GET ✔ ✔
-`Get details <#get-floating-ip-details>`_ ``/floatingips/`` GET ✔ ✔
-`Create <#create-floating-ip>`_ ``/floatingips`` POST ✔ ✔
-Update ``/floatingips/`` PUT **✘** ✔
-`Delete <#delete-floating-ip>`_ ``/floatingips/`` DELETE ✔ ✔
-========================================= ================================ ====== ================ ==========
+========================================= ======================================= ====== ================ ==========
+Description URI Method Cyclades/Network OS/Neutron Extensions
+========================================= ======================================= ====== ================ ==========
+`List <#list-floating-ips>`_ ``/floatingips`` GET ✔ ✔
+`Get details <#get-floating-ip-details>`_ ``/floatingips/`` GET ✔ ✔
+`Create <#create-floating-ip>`_ ``/floatingips`` POST ✔ ✔
+Update ``/floatingips/`` PUT **✘** ✔
+`Delete <#delete-floating-ip>`_ ``/floatingips/`` DELETE ✔ ✔
+`Reassign <#reassign-floating-ip>`_ ``/floatingips//action`` POST ✔ **✘**
+========================================= ======================================= ====== ================ ==========
List networks
-------------
@@ -392,6 +394,48 @@ Return Code Description
network or floating IPs reserved from its pool. The subnets that are
connected to it, though, are automatically deleted upon network deletion.
+Reassign Network
+----------------
+
+Assign a network to a different project.
+
+.. rubric:: Request
+
+================================= ====== ================ ==========
+URI Method Cyclades/Network OS/Neutron
+================================= ====== ================ ==========
+``/networks//action`` POST ✔ **✘**
+================================= ====== ================ ==========
+
+|
+
+============== =========================
+Request Header Value
+============== =========================
+X-Auth-Token User authentication token
+============== =========================
+
+Request body contents::
+
+ reassign: {
+ project:
+ }
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) Request succeeded
+400 (Bad Request) Malformed request
+401 (Unauthorized) Missing or expired user token
+403 (Forbidden) Not allowed to modify this network (e.g. public)
+404 (Not Found) Network not found
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) The service is not currently available
+=========================== =====================
+
List subnets
------------
@@ -1248,6 +1292,49 @@ Return Code Description
404 (itemNoFound) Floating IP not found
=========================== =====================
+Reassign floating ip
+--------------------
+
+Assign a floating IP to a different project.
+
+.. rubric:: Request
+
+======================================= ====== ================ ==========
+URI Method Cyclades/Network OS/Neutron
+======================================= ====== ================ ==========
+``/floatingips//action`` POST ✔ **✘**
+======================================= ====== ================ ==========
+
+|
+
+============== =========================
+Request Header Value
+============== =========================
+X-Auth-Token User authentication token
+============== =========================
+
+Request body contents::
+
+ reassign: {
+ project:
+ }
+
+.. rubric:: Response
+
+=========================== =====================
+Return Code Description
+=========================== =====================
+200 (OK) Request succeeded
+400 (Bad Request) Malformed request
+401 (Unauthorized) Missing or expired user token
+403 (Forbidden) Not allowed to modify this network (e.g. public)
+404 (Not Found) Network not found
+500 (Internal Server Error) The request cannot be completed because of an
+\ internal error
+503 (Service Unavailable) The service is not currently available
+=========================== =====================
+
+
Index of Attributes
-------------------
diff --git a/docs/networks.rst b/docs/networks.rst
index 55c146d15ba601cf76886034575f696937e71ca3..8590e04b9c8c1b3385861a3868f1b06d7267a228 100644
--- a/docs/networks.rst
+++ b/docs/networks.rst
@@ -453,7 +453,7 @@ that he can access the VPN. Then we run in Cyclades:
.. code-block:: console
- # snf-manage network-create --subnet=192.168.1.0/24 --gateway=192.168.1.0/24 --dhcp=True --flavor=CUSTOM --mode=bridged --link=br200 --mac-prefix=bb:00:44 --owner=user@grnet.gr --tags=nfdhcpd,vpn --name=vpn --backend-id=1
+ # snf-manage network-create --subnet=192.168.1.0/24 --gateway=192.168.1.0/24 --dhcp=True --flavor=CUSTOM --mode=bridged --link=br200 --mac-prefix=bb:00:44 --user=user@grnet.gr --tags=nfdhcpd,vpn --name=vpn --backend-id=1
# snf-manage network-list
id name flavor owner mac_prefix dhcp state link vms public IPv4 Subnet IPv4 Gateway
diff --git a/docs/object-api-guide.rst b/docs/object-api-guide.rst
index f00814da22d057e394d78f3e6f578a9b77a61ff7..89d9e613a61e3662b8009eeddf1a785558dc1bd5 100644
--- a/docs/object-api-guide.rst
+++ b/docs/object-api-guide.rst
@@ -19,7 +19,7 @@ The present document is meant to be read alongside the OOS API documentation. Th
Whatever marked as to be determined (**TBD**), should not be considered by implementors.
-More info about Pithos can be found here: https://code.grnet.gr/projects/pithos
+More info about Pithos can be found `here