openstack

Happy Software Freedom Day! Find Out How NASA Has Contributed To Open Source Software:

“Back in 2008 and 2009, people were still trying to figure out what ‘cloud’ meant. While lots of people were calling themselves ‘cloud enabled’ or ‘cloud ready,’ there were few real commercial offerings. With so little clarity on the issue, there was an opportunity for us to help fill that vacuum.” - Raymond O’Brien, +NASA Ames Research Center 

Needing a way to standardize web space, a team of researchers at NASA Ames began a 2008 project known as NASA.net. The project offered a way to consolidate web development tools and data resources which heightened efficiency between all facets of the space agency. William Eshagh, another key contributor from NASA.net’s early days, aimed to find a way for web developers to upload code to a platform that was universally utilized.

“The basic idea was that the web developer would write their code and upload it to the website, and the website would take care of everything else,” according to Eshagh.

Still requiring an “infrastructure service” to manage the large quantities of data that NASA accumulates on a daily basis, the scope of the Ames project switched gears, and NASA.net was reorganized as Nebula. Rather than simply setting standards and providing a platform for web developers, the Nebula team would construct an open source compute controller. Early on, the collaborative nature of Nebula benefited development – as anyone with the understanding of the technology and desire could access the code and provide improvements. Raymond O'Brien, who remained on the Nebula team, reiterated the appeal of Nebula’s open source identity.

“From the beginning, we wanted this project to involve a very large community—private enterprises, academic institutions, research labs—that would take Nebula and bring it to the next level. It was a dream, a vision. It was that way from the start,” said O’Brien.

An early obstacle the pure open sourced project had to overcome was a piece of software known as the cloud controller, a pivotal segment of the project if the end users are to access computers or data. At this time, the existing tools were either written in the incorrect programming language or were closed source – not usable due to licensing limitations. However, It only took the Nebula team a matter of days to build a new cloud controller from scratch, and immediately began to attract interest from Rackspace Inc.

“We believed we were addressing a general problem that would have broad interest,” stated Eshagh. “As it turns out, that prediction couldn’t have been more accurate.”

Rackspace, known for providing open source storage, was set to begin construction of a similar cloud controller to what Nebula just released. Given the technical similarities between the two teams, Rackspace and Nebula began a partnership known as OpenStack – and a community of developers around the world would contribute towards the construction of what would become one of the most successful open source cloud operating systems.

The future of OpenStack, and other open source projects, are bright due the early efforts of the NASA.net team at Ames. Due to the initial devotion to keeping the project open source in those early days, a large majority of contributions to the OpenStack code came from community efforts outside of NASA. Today, on Software Freedom Day, be sure to checkout the following resources related to the OpenStack cloud, a NASA Spinoff.

Sources:
1. Web Solutions Inspire Cloud Computing Software
http://spinoff.nasa.gov/Spinoff2012/it_2.html
2. Nebula, NASA, and OpenStack
http://open.nasa.gov/blog/2012/06/04/nebula-nasa-and-openstack/
3. Software Freedom Day
http://softwarefreedomday.org/

Pythonista なら OpenStack プロジェクトが使っているパッケージは要チェック

プログラミングでベストプラクティスを見つけるには、他人のソースコードを読むのが手っ取り早いと思う。 特に、活動が活発な OSS は素晴らしい勉強材料になる。 その中でも OpenStack は世界中の Pythonista が寄ってたかって弄っているプロジェクトなので、なかなか良い感じ。 コンポーネント毎にちぐはぐ感はあっても間違いは少ないと思う。 そこで使われているパッケージを眺めているだけでも、結構な発見があるんじゃないかな。

とりあえず下準備としてディレクトリを一段掘ってソースコードをチェックアウトしておく。 OpenStack プロジェクトは本当にコンポーネントが多いので主要なものだけに絞った。

$ mkdir openstack
$ git clone https://github.com/openstack/nova.git openstack/nova
$ git clone https://github.com/openstack/neutron.git openstack/neutron
$ git clone https://github.com/openstack/horizon.git openstack/horizon
$ git clone https://github.com/openstack/keystone.git openstack/keystone
$ git clone https://github.com/openstack/glance.git openstack/glance
$ git clone https://github.com/openstack/cinder.git openstack/cinder
$ git clone https://github.com/openstack/swift.git openstack/swift

コンポーネントの動作に必要なパッケージは各プロジェクト直下にある requirements.txt に記述されている。 動作以外でテストに必要なものは test-requirements.txt にある。 ちなみに、これらはパッケージ管理システム PIP に読み取らせるためのフォーマットで書かれている。 pip install -r <requirements-file> みたいにして使えるわけ。

各コンポーネント毎に眺めるならこんな感じ。 ファイルにはパッケージのインストールに必要なバージョン情報とかコメントが入ってて邪魔なので取り除く。
$ cat openstack/nova/requirements.txt | cut -d ">" -f 1 | cut -d "=" -f 1 | cut -d "<" -f 1| sed -e "/^#/d" -e "/^$/d"
$ cat openstack/nova/test-requirements.txt | cut -d ">" -f 1 | cut -d "=" -f 1 | cut -d "<" -f 1| sed -e "/^#/d" -e "/^$/d"

個別に見ていっても良いけど、使っているパッケージを一覧したいのでシェルスクリプトを用意してみる。
$ cat openstack-unique-libs.sh
#!/bin/sh

LIBS=""
for i in `ls openstack`
do
  LIBS+=`cat openstack/$i/requirements.txt`
  LIBS+=`cat openstack/$i/test-requirements.txt`
done
echo "$LIBS" | cut -d ">" -f 1 | cut -d "=" -f 1 | cut -d "<" -f 1| sed -e "/^#/d" -e "/^$/d" | sort | uniq

実行してみる。
$ sh openstack-unique-libs.sh
-f http://pysendfile.googlecode.com/files/pysendfile-2.0.0.tar.gz
Babel
Django
Jinja2
MySQL-python
Paste
PasteDeploy
Routes
SQLAlchemy
WebOb
WebTest
alembic
amqplib
anyjson
argparse
boto
cliff
configobj
coverage
discover
django-nose
django_compressor
django_openstack_auth
docutils
dogpile.cache
eventlet
feedparser
fixtures
greenlet
hacking
hp3parclient
hplefthandclient
httplib2
iso8601
jsonrpclib
jsonschema
keyring
kombu
lesscpy
lockfile
lxml
mock
mox
netaddr
netifaces
nose
nose-exclude
nosehtmloutput
nosexcover
oauthlib
openstack.nose_plugin
ordereddict
oslo.config
oslo.messaging
oslo.rootwrap
oslo.sphinx
oslo.vmware
oslosphinx
oslosphinx# Horizon Core Requirements
oslosphinxpbr
oslotest
paramiko
passlib
pastedeploy
pbr
psutil
psycopg2
pyOpenSSL
pyasn1
pycadf
pycrypto
pylint
pymongo
pysendfile
pysqlite
python-ceilometerclient
python-cinderclient
python-glanceclient
python-heatclient
python-keystoneclient
python-ldap
python-memcached
python-neutronclient
python-novaclient
python-saharaclient
python-subunit
python-swiftclient
python-troveclient
pytz
qpid-python
requests
rtslib-fb
selenium
simplejson
six
sphinx
sqlalchemy-migrate
stevedore
suds
taskflow
testrepository
testscenarios
testtools
websockify
wsgiref
xattr

補足しておくと oslo.* は OpenStack プロジェクトが内製しているパッケージ。 あとは OpenStack のコンポーネント名が付くものも基本そうかな。

こういった活動が活発な OSS を幾つか知っておくと、良さげなライブラリへの感度は高まると思う。 ちなみに前述した「ちぐはぐ感」というのは、例えばあるコンポーネントは DB マイグレーションに alembic を使っているのに、別のコンポーネントでは sqlalchemy-migrate を使っている、とかそういうの。 上のリストを見ると他にはテストダブル用の mock と mox とかも、同様に被ってるよね。 ちなみに二つから選ぶなら、ぼくのオススメは alembic と mock かな。

Beyond the Horizon

Presented by Cindy Lu, David Lyle, Richard Jones, Thai Tran, Travis Tripp

Why AngularJS?

Doesn’t require your web app to be SPA out of the box.

Directives: Semantic HTML Behaviors

Component Layers

  • Angular application
  • Angular JS service Utilities
  • Angular JS REST clients
  • Horizon REST API
  • Horizon API

Django API Endpoint

from openstack_dashboard import api
from openstack_dashboard.api.rest import urls
from openstack_dashboard.api.rest import utils as rest_utils

@urls.register
class Keypairs(generic.View):
  url_regex = r'nova/keypairs/$'
  
  @rest_utils.ajax
  def get(self, request):
    result = api.nova.keypair_list(request)
    return {
      'items': [u.to_dict() for u in result]}
    }

Today we found out, that beam is very useful for interviews. Some people prefer to talk to the robot than to speak with the “human” reporter. But before using this device for interviews you should double-check the connectivity. Minimum 500 kbps are required. You will be on trouble if you lose the connectivity in process of interviewing. And your interlocutor will feel uncomfortable. The idea of using remotely controlled device for the interview is to have two cameras instead of one. Or, if you do not have additional camera you can use the device itself to record the content via Wirecast or some software working like Wirecast. Also remotely controlled device is a feature, so it will work also for your marketing needs.  

Learn you some Ansible for Great Good!

Presented by Juergen Brendel and David Lapsley from Cisco/Metacloud

Ansible Overview

  • orchestration engine for CM and deployment
  • written in Python
  • uses YAML
  • playbooks
  • config specs or explicit commands

Ansible simplicity

Key points

  • no central config server
  • no key management
  • no agent to install on target machine
  • explicit order

Requirements

  • need SSH
  • needs Python installed on target hosts

How does Ansible work?

It copies a Python module to the target machine, executes it, and then deletes it.

Details

Inventory and groups

Define hosts organized by groups - location - function or role (eg. frontend or backend) - hosting provider

Adhoc commands

Single commands, applied to groups ansible -i hosts frontend -a 'service httpd restart' -f 3

Best practice for project layout

  • best practices layout for arranging Ansible playbooks, variables, templates, metadata, etc.
  • better suited for larger projects
  • more extensible
/
  ansible.cfg
  deploy_hosts
  staging_hosts
  group_vars/
    all
    frontend
    backend
    europe
    north-america
  host_Vars/
    server1.co.uk
    host-b.serverhost.com
  site.yml # entry point
  roles/
    common/
      tasks/
        main.yml
      handlers/
        main.yml
      templates/
        sshd_config.j2
      files/
        my_script.sh
      vars/
        main.yml
    web/
      ...
    db/
      ...

CMS Managed Infrastructure

Example: Create instances

- locaiton action:
  register: ec2results
  
- local_action:
    module: add_host
    hostname: {{ item.public_ip }}
    groupname: my-server-group
  with_items: ec2results.instances

Key takeaways

  • same Ansible playbooks can be used to provision application locally or in the cloud
  • with cloud APIs and Ansible moudles playbooks can be used to resource infra

References

IBM and OpenStack

Ron Miller at TechCrunch is one of those reporting on an IBM press release from earlier this week.

The words in the release are broadly positive, and IBM has engaged with various OpenStack projects for a number of years. But the extent to which IBM’s SoftLayer IaaS really uses/ supports/ integrates with OpenStack remains something of a mystery.

Disclaimer: the OpenStack Foundation is meeting the cost of my attendance in Vancouver this week.

Clouds, open source, and new network models: Part 3

by James Urquhart  October 28, 2011

In part 1 of this series, I described what is becoming an increasingly ubiquitous model for cloud computing networks, namely the use of simple abstractions delivered by network systems of varying sophistication. In part 2, I then described OpenStack’s Quantum network service stack and how it reflected that model. Software defined networking (SDN) is an increasingly popular–but extremely nascent–model for network control, based on the idea that network traffic flow can be made programmable at scale, thus enabling new dynamic models for traffic management. Because it can create “virtual” networks in the form of custom traffic flows, it can be confusing to see how SDN and cloud network abstractions like Quantum’s are related.

http://news.cnet.com/8301-19413_3-20126245-240/clouds-open-source-and-new-network-models-part-3/

youtube

Scobleizer interviews Gluster CEO outside Facebook’s Palo Alto DC.