<p>One may find a guide on writing Lambda Applications on the landing page of the LoD service. The source code exhibited through that example is available on <ahref="https://github.com/grnet/okeanos-LoD/tree/master/example">github</a>. For easy reference, the aforementioned guide is also include in this page.</p>
<p>One may find a guide on writing Lambda Applications on the landing page of the LoD service. The source code exhibited through that example is available on <ahref="https://github.com/grnet/okeanos-LoD/tree/master/example">github</a>. For easy reference, the aforementioned guide is also in included this page.</p>
<p>In order to upload, deploy and start running a Lambda Application on top of a Lambda Instance the user of the service will go through the workflow shown below, again using the Web UI of the LoD service VM.</p>
...
...
@@ -67,12 +67,12 @@
</div><!--row-->
<ol>
<li>Log into the service dashboard using your ~okeanos token (you can find your ~okeanos token <ahref="https://accounts.okeanos.grnet.gr/ui/api_access">here</a>)</li>
<li>Select Applications from the upper left menu of options</li>
<li>On the main window select the button "Upload an Application"</li>
<li>Fill in the form, select the jar file to upload and press "Submit" when ready</li>
<li>Note the progress bar as the application uploads</li>
<li>Upload is completed</li>
<li>On the Applications page select the Application you have just uploaded (click on its name)</li>
<li>Notice the progress bar as the application uploads</li>
<li>Upload has been completed. On the Applications' page, you can select the Application you have just uploaded (click on its name)</li>
</ol>
<divclass="row margin-bottom">
...
...
@@ -126,8 +126,9 @@
</div><!--row-->
<olstart="7">
<li>Select the "Deploy on a Lambda Instance" button</li>
<li>On the list of Lambda Instances available select the one you want to use for running this specific application and</li>
<li>Select Applications from the upper left menu of options</li>
<li>Select the "Deploy on a Lambda Instance" button of the Application you wish to deploy</li>
<li>On the list of Lambda Instances available select the one you want to use for running this specific application</li>
<li>Select to start your application once it has been deployed on the Lambda Instance</li>
</ol>
...
...
@@ -152,12 +153,12 @@ To get more information about Apache Kafka, you can visit <a href="kafka.apache.
<h4>Master Dataset</h4>
The master dataset is an immutable, append-only set of raw data that is stored in the Apache HDFS cluster of your Lambda Instance. As stated in the previous paragraph, any data sent on a topic categorized as input topic, will be appended to the master dataset automatically via Apache Flume.<br>
For more information regarding Apache HDFS and FLume you can visit <ahref="hadoop.apache.org">hadoop.apache.org</a> and <ahref="flume.apache.org">flume.apache.org</a> respectively.
For more information regarding Apache HDFS and Flume you can visit <ahref="hadoop.apache.org">hadoop.apache.org</a> and <ahref="flume.apache.org">flume.apache.org</a> respectively.
<br><br>
<h3>Stream Job</h3>
A Stream Job, is an Apache Flink Job designed to process streams of data, in real time, with high throughput rates and low latency.
You can find the code for the Stream Job inside the <ahref="https://github.com/grnet/okeanos-LoD/tree/master/example/stream_word_count">stream_word_count</a> directory.
You can find the example code for the Stream Job inside the <ahref="https://github.com/grnet/okeanos-LoD/tree/master/example/stream_word_count">stream_word_count</a> directory.
<br><br>
<h4>Input-Output</h4>
...
...
@@ -253,7 +254,7 @@ After this directive is finished, a new directory, named "target" will have been
<h3>Batch Job</h3>
A Batch Job is an Apache Flink Job designed to process big loads of data asynchronously. You can find the code for
A Batch Job is an Apache Flink Job designed to process big loads of data asynchronously. You can find the example code for
out Batch Job inside the <ahref="https://github.com/grnet/okeanos-LoD/tree/master/example/batch_word_count">batch_word_count</a> directory.
<br><br>
...
...
@@ -357,7 +358,7 @@ After this directive is finished, a new directory, named "target" will have been
<li>Login the service dashboard using your ~okeanos token(you can find your ~okeanos token <ahref="https://accounts.okeanos.grnet.gr/ui/api_access">here</a>)</li>
<li>Loginto the service dashboard using your ~okeanos token(you can find your ~okeanos token <ahref="https://accounts.okeanos.grnet.gr/ui/api_access">here</a>)</li>
<li>Select Instances from the upper left menu of options</li>
<li>On the main window select the button "Create New Instance"</li>
<li>Fill in your new Lambda Instance details in the form provided and click "Submit" when ready (all fields marked with * are mandatory)</li>
<li>Note the status of your new Lambda Instance is automatically updated while it is being deployed and configured</li>
<li>Notice the status of your new Lambda Instance is automatically updated while it is being deployed and configured</li>
</ol>
<p><strong>At the end your Lambda Instance will be ready to use. </strong></p>
<p>The Lambda on Demand service runs on top of GRNET ~okeanos IaaS public cloud. Users of the ~okeanos service can use the Lambda on Demand service in order to instantiate and manage their own Lambda Instances (clusters of VMs utilizing the <ahref="http://lambda-architecture.net/">λ (lambda) architectural pattern</a>) with ease and comfort.</p>
<p>The Lambda on Demand service utilizes Apache Flink on a backend YARN/HDFS installation and Apache Kafka as the data ingestion layer. It also utilizes Apache Flume as an Kafka-HDFS connector.</p>
<p>The Lambda on Demand service utilizes Apache Flink on a backend YARN/HDFS installation and Apache Kafka as the data ingestion layer. It also utilizes Apache Flume as a Kafka-HDFS connector.</p>
<p>In order to use the Lambda on Demand (LoD for brevity) service an ~okeanos user will first need to <ahref="https://cyclades.okeanos.grnet.gr">spin up a LoD service VM</a> using the publicly available Lambda Service VM image file.<i> We recommend using at least 4 CPUs, 4GB of RAM and 40GB disk size for the LoD service VM.</i></p>
You can use Apache Kafka topics to send data to a Lambda Application of yours and also
receive the results of the processing of these data. Note that your Application should be {{#link-to'faqs.lambda-application.create'}}configured properly{{/link-to}} in order to send the results to topics, they will <b>NOT</b> be sent there by default. In order send and receive data from an Apache Kafka topic, you can use the <ahref="https://github.com/grnet/okeanos-LoD/tree/devel/example/kafka_producer">kafka producer</a> and <ahref="https://github.com/grnet/okeanos-LoD/tree/devel/example/kafka_consumer">kafka consumer</a> we have created responsively. Follow the README.md file of each program to get information regarding its usage. It goes without saying that you can create your own producers and consumers to send and receive data from Apache Kafka if our implementations doesn't cover your needs.
receive the results of the processing of these data. Note that your Application should be {{#link-to'faqs.lambda-application.create'}}configured properly{{/link-to}} in order to send the results to topics, they will <b>NOT</b> be sent there by default. In order send and receive data from an Apache Kafka topic, you can use the <ahref="https://github.com/grnet/okeanos-LoD/tree/devel/example/kafka_producer">kafka producer</a> and <ahref="https://github.com/grnet/okeanos-LoD/tree/devel/example/kafka_consumer">kafka consumer</a> we have created respectively. Follow the README.md file of each program to get information regarding its usage. It goes without saying that you can create your own producers and consumers to send and receive data from Apache Kafka if our implementations doesn't cover your needs.
