Soo Hwan's Blog

Find an installed Ruby Gem contents folder

2011-09-03T03:14:00.000-07:00

 #!/usr/bin/env ruby  

 def find_path gem_name  

   `gem list #{gem_name}` =~ /(.*)\s\((.*)\)/  

   version = $2.split(",")[0]  

   "#{Gem.default_dir}/gems/#{gem_name}-#{version}"  

 end  

 puts find_path ARGV[0]

CouchDB with Jsonp

2011-07-07T16:45:00.000-07:00

Download jquery.jsonp.js and ejs.js.

Templates "list.ejs"

 <ul>  
 <% $.each(list, function(i, element) {%>  
   <li><%=element%></li>  
 <% }); %>  
 </ul>

test.html

 <!DOCTYPE html>  
 <html>  
  <head>  
   <title>CouchDB Demo</title>  
   <%= stylesheet_link_tag 'jquery-ui', 'develop_demo' %>  
   <%= javascript_include_tag :defaults, 'jquery.couch', 'jquery.jsonp-2.1.4.min', 'ejs' %>  
   <%= csrf_meta_tag %>  
   <script type="text/javascript" charset="utf-8">  
   $(document).ready(function() {  
     var url_all_dbs = "http://127.0.0.1:5984/_all_dbs";  
     function all_dbs() {  
       $.jsonp({  
         url: url_all_dbs + "?callback=?",  
         success: function(data) {  
           render_all_dbs(data);  
         }  
       });  
     }  
     function render_all_dbs(data) {  
       var template = new EJS({url: '/templates/list.ejs'})  
       var div_html = template.render({list: data});  
       $("#couch_all_dbs").html(div_html);  
     };  
     all_dbs();  
   });  
   </script>  
   </head>  
   <body>  
    <h1>Development Test</h1>  
    <div id="content">  
      Content...  
      <h1>all_dbs</h1>  
      <div id="couch_all_dbs"></div>  
    </div>  
   </body>  
 </html>

HTML Audio Tag with Duration

2011-03-24T23:46:00.000-07:00

Here is the code ..

 <html>  
   <header>  
     <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.0/jquery.min.js"></script>  
     <script>  
     $(document).ready(function(){  
       var audioElement = document.getElementById('audio');  
       audioElement.addEventListener("durationchange", function() {  
         var duration_div = document.getElementById('duration');  
         var duration = parseInt(audioElement.duration);  
         var min = parseInt(duration / 60);  
         var sec = duration % 60;  
         var str = "" + min + ":" + sec;   
         duration_div.innerHTML = str;  
       }, true);  
       audioElement.addEventListener("timeupdate", function() {  
         var current_div = document.getElementById('current');  
         var cur_time = parseInt(audioElement.currentTime);  
         var min = parseInt(cur_time / 60);  
         var sec = cur_time % 60;  
         var str = "" + min + ":" + sec;   
         current_div.innerHTML = str;  
       }, true);  
     });  
     </script>  
   </header>  
   <body>  
   <audio id="audio" controls preload="auto" autobuffer>   
     <source src="test.mp3" />  
   </audio>  
   <span id="current"></span> / <span id="duration"></span>  
   </body>  
 </html>

Ruby's mDNS (dnssd)

2011-02-27T15:38:00.000-08:00

I'd like to have Zero conf services. So, I was looking for those Discovery libraries and found this.. Let's try

Here is the service side.

 require 'dnssd'  
 domains = []  
 enumerator = DNSSD.enumerate_domains do |reply|  
  domains << reply.domain  
  next if reply.flags.more_coming?  
  puts "Found domains:\n#{domains.join "\n"}"  
  break  
 end  
 DNSSD.register 'agent', '_agent._tcp', nil, 6464  
 puts "registered agent at 6464"  
 loop do  
   sleep 1  
   puts "agent running.."  
 end

And here is the client side..

 require 'dnssd'  
 browser = DNSSD::Service.new  
 puts "Browsing for Agent service"  
 browser.browse '_agent._tcp' do |reply|  
   puts "Time: #{Time.new.to_f} reply: #{reply.fullname}"  
   puts reply    
   break  
 end

3. Asynchronous Browse and Resolve

 require 'dnssd'  
 require 'socket'  
 browser = DNSSD::Service.new  
 puts "Browsing for Agent service"  
 # Make it Asynchronous ..  
 Thread.new do  
  browser.browse '_agent._tcp' do |reply|  
    puts "Time: #{Time.new.to_f} reply: #{reply.fullname}"  
    puts reply.inspect    
    puts reply.service_name  
    target = nil  
    resolver = DNSSD::Service.new  
    resolver.resolve reply do |r|  
     target = r.target  
     puts "#{r.name} on #{r.target}:#{r.port}"  
     # puts "\t#{r.text_record.inspect}" unless r.text_record.empty?  
     break unless r.flags.more_coming?  
    end  
    resolver.stop  
    puts "target: #{target}"  
    addrinfo = Socket.getaddrinfo target, nil, Socket::AF_INET  
    puts "Addresses for #{target}"  
    addrinfo.each do |addr|  
     puts addr.inspect  
    end  
    break  
  end   
 end  
 sleep 1

Enjoy!

Fix EventMachine::PeriodicTimer

2011-02-25T00:06:00.000-08:00

EventMachine::PeriodicTimer doesn't give an accurate timing. For example, if I add a periodic timer with 1 second interval, in my Mac, it looks like this.

1. the test code.

 require 'eventmachine'  
 puts 'Start'  
 EM.run do  
   EM::add_periodic_timer 1.0 do   
     puts "time: #{Time.new.to_f.round(1)}"  
   end  
 end

2. output in my MacBook Pro.

 time: 1298620726.9  
 time: 1298620728.0  
 time: 1298620729.1  
 time: 1298620730.1  
 time: 1298620731.2  
 time: 1298620732.3  
 time: 1298620733.4  
 time: 1298620734.5  
 time: 1298620735.5  
 time: 1298620736.6  
 time: 1298620737.7  
 time: 1298620738.8  
 time: 1298620739.9  
 time: 1298620741.0  
 time: 1298620742.0  
 time: 1298620743.1  
 time: 1298620744.2  
 time: 1298620745.3  
 time: 1298620746.4  
 time: 1298620747.4  
 time: 1298620748.5  
 time: 1298620749.6

As you can see, it drifted 100ms every time. Which is not very good when you want to do some job every 1 second. You might omit some seconds. So, with using the same method that I used for the Periodic ruby service, I made a change to PeriodicTimer. It looks like this.

 # Override PeriodicTimer  
 module EventMachine  
   class PeriodicTimer  
     alias :old_initialize :initialize  
     def initialize interval, callback=nil, &block  
       # Added two additional instance variables to compensate difference.   
       @start = Time.now  
       @fixed_interval = interval  
       old_initialize interval, callback, &block  
     end  
     alias :old_schedule :schedule  
     def schedule  
       # print "Started at #{@start}..: "  
       compensation = (Time.now - @start) % @fixed_interval   
       @interval = @fixed_interval - compensation  
       # Schedule   
       old_schedule  
     end  
   end  
 end

So, here is the result (this is not that accurate like the simple ruby service, but reasonably useable.

 time: 1298620683.4  
 time: 1298620684.4  
 time: 1298620685.4  
 time: 1298620686.4  
 time: 1298620687.4  
 time: 1298620688.4  
 time: 1298620689.4  
 time: 1298620690.4  
 time: 1298620691.3  
 time: 1298620692.3  
 time: 1298620693.4  
 time: 1298620694.4  
 time: 1298620695.4  
 time: 1298620696.4  
 time: 1298620697.4  
 time: 1298620698.4  
 time: 1298620699.4

That's it. :-)

Ruby - Periodic Process

2011-02-24T22:41:00.000-08:00

Ruby's sleep is not very accurate and Eventmachine's timer is also not very good. So, If you need to process one periodic job with some interval, this approach might be helpful. There is a compensation for sleep and you just need to define your process.

Here is the Service class.

 class Service  
   def initialize  
     @interval = 1.0  
     @start_time = Time.now   
   end  
   def start   
     # Update the start-time  
     @start_time = Time.now   
     # run loop  
     loop do  
       t_start = Time.now  
       # process the job  
       process  
       elapsed = Time.now - t_start  
       if elapsed > @interval  
         puts "Error: Job is bigger than Interval.. #{elapsed}"  
       end  
       # apply some compensation.   
       compensation = (Time.now - @start_time) % @interval  
       sleep (@interval - compensation)  
     end  
   end  
   def process  
     raise NotImplementedError  
   end  
 end

And then, define your jobs in process method.

 class MyService < Service   
   def initialize  
     super  
     @interval = 2.0  
   end  
   def process  
     puts "time: #{Time.now.to_f.round(3)}"  
     # do the random job .. it takes upto 1.5 seconds.   
     sleep rand(150) * 0.01  
   end  
 end

Try to run it..

 service = MyService.new  
 service.start

And you will get a very accurate intervals

 time: 1298615010.271  
 time: 1298615011.271  
 time: 1298615012.271  
 time: 1298615013.271  
 time: 1298615014.271  
 time: 1298615015.271  
 time: 1298615016.271  
 time: 1298615017.271  
 time: 1298615018.271  
 time: 1298615019.271  
 time: 1298615020.271  
 time: 1298615021.271  
 time: 1298615022.271  
 time: 1298615023.271  
 time: 1298615024.271  
 time: 1298615025.271  
 time: 1298615026.271  
 time: 1298615027.271  
 time: 1298615028.271  
 time: 1298615029.271  
 time: 1298615030.271  
 time: 1298615031.271  
 time: 1298615032.271  
 time: 1298615033.271  
 time: 1298615034.271  
 time: 1298615035.271  
 time: 1298615036.271  
 time: 1298615037.271  
 time: 1298615038.271  
 time: 1298615039.271  
 time: 1298615040.271  
 time: 1298615041.271  
 time: 1298615042.271

Next time, will try to fix Eventmachine timer..

NodeJS - HTTP Request

2011-02-13T21:23:00.000-08:00

1. GET

 var http = require('http');  
 // GET.   
 var options = {  
           host: 'localhost',   
           port: 3000,   
           path: '/user/suwanny'  
      };   
 var req = http.get(options, function(res) {  
      console.log("Got response: " + res.statusCode);   
      res.on('data', function(chunk) {  
           console.log("Body: " + chunk);   
      });   
 }).on('error', function(e) {  
      console.log("Got error: " + e.message);   
 });

2. POST

 // POST.   
 var http = require('http');
 var options = {  
           host: 'localhost',   
           port: 3000,   
           path: '/load/debug',  
           method: 'POST'  
      };   
var req = http.request(options, function(res) {       
      console.log("Got response: " + res.statusCode);   
      console.log('HEADERS: ' + JSON.stringify(res.headers));  
      res.on('data', function(chunk) {  
           console.log("Body: \n" + chunk);   
      });   
 });   
 req.end();

NodeJS - Socket.IO

2011-02-13T17:12:00.000-08:00

1. ex_socketio.js

 var express = require('express'),   
      server = express.createServer(),  
  io = require('socket.io');  
 server.configure(function(){  
      server.use(express.logger());   
      server.use(express.staticProvider(__dirname + '/www/public'));  
 });  
 server.listen(3000);   
 // socket.io   
 var socket = io.listen(server),   
      buffer = [];   
 socket.on('connection', function(client){   
      client.send({buffer: buffer});   
      client.broadcast({announcement: client.sessionId + ' connected'});  
  // new client is here!   
  client.on('message', function(message){  
           client.send({ message: "got your message: " + message});   
           client.broadcast({ message: client.sessionId + " sent message: " + message});   
      });  
  client.on('disconnect', function(){  
           client.broadcast({announcement: client.sessionId + ' disconnected'});  
      });   
 });

2. www/public/index.html

 <html>  
   <head>  
   <title> Test </title>  
   <script src="/socket.io/socket.io.js"></script>  
   </head>  
   <body>  
 <h1> Hello NodeJS </h1>  
 <p> This is a static folder </p>   
 <p id="ann">empty now</p>  
 <p id="chat">empty now</p>  
   <script>  
     var socket = new io.Socket(null, {port: 3000});   
     socket.connect();   
     socket.on("message", function(obj) {  
       console.dir(obj);   
       if('buffer' in obj) {  
         document.getElementById('chat').innerHTML = '';  
       }  
       else {  
         if('announcement' in obj)  
           document.getElementById('ann').innerHTML = obj.announcement;   
         else  
           document.getElementById('chat').innerHTML = obj.message;   
       }  
     });   
     socket.send("this is the test message");   
   </script>  
   </body>  
 </html>

NodeJS - Express

2011-02-13T16:00:00.000-08:00

This is an example which uses Express and Ejs with many configuration.
Here is the full guide http://expressjs.com/guide.html

1. Static files ..

 app.use(express.staticProvider(__dirname + '/www/public'));

2. View files (EJS)

      app.set('views', __dirname + '/www/views');  
     app.set('view engine', 'ejs');

layout.ejs

 <html> 
<head><title>EJS Layout</title></head> 
<body> 
  <%- body %> 
</body> 
</html>

hello.js

 <h1>Hello</h1> 
  <h2><%= api %></h2>

3. nodejs application

 // Production: NODE_ENV=production node ex_express.js 
// Development: NODE_ENV=development node ex_express.js 
var express = require('express'),  
     app = express.createServer(); 
app.configure(function(){ 
     app.use(express.logger());  
 app.use(express.methodOverride()); 
 app.use(express.bodyDecoder()); 
 app.use(app.router); 
 app.use(express.staticProvider(__dirname + '/www/public')); 
     app.set('views', __dirname + '/www/views');  
     app.set('view engine', 'ejs'); 
}); 
app.configure('development', function(){ 
 app.use(express.errorHandler({ dumpExceptions: true, showStack: true })); 
}); 
app.configure('production', function(){ 
 app.use(express.errorHandler()); 
}); 
// Application.  
app.get('/user/:id', function(req, res) { 
     res.render('hello.ejs', {locals: {"api": "user"}});  
});  
app.post('/load/:mode', function(req, res) { 
     // console.dir(req);  
     var body = "<h1>Load</h1>";  
     body += req.params.mode + "\n" 
     body += req.param("workers") + "\n" 
     res.send(body);  
}); 
app.listen(3000);  
console.log("Public Dir: " + __dirname + '/public');  
console.log("Service Started at " + 3000);

NodeJS- Useful Modules

2011-02-13T10:02:00.000-08:00

1. express (RESTful routing, http://expressjs.com/ )
http://expressjs.com/guide.html

High performance, high class web development for Node.js

 var express = require('express');
var app = express.createServer();
app.get('/', function(req, res){
   res.send('Hello World');
});
app.listen(3000);

Features

Robust routing
Redirection helpers
Dynamic view helpers
Application level view options
Content negotiation
Focus on high performance
View rendering and partials support
Environment based configuration
Session based flash notifications
Built on Connect
Executable for generating applications quickly
High test coverage

2. socket.io (Websocket)
http://soohwan.blogspot.com/2011/02/nodejs-socketio.html

3. couch_client (CouchDB)

4. node_redis (redis)

5. mDNS/Zeroconf/Bonjour (https://github.com/agnat/node_mdns)
Hmm unfortunately this is not useable in my Appliance, so I gave up to use this even though I need this feature.

 npm install mdns

Service Discovery

node_mdns – service discovery for node.js
node_mdns adds support for multicast DNS service discovery, also known as zeroconf or bonjour to node.js. It provides an object-oriented interface to announce and browse services on the local network.

Internally, it uses the mDNSResponder API which is available on all major platforms.

On Linux and other systems using the avahi daemon the avahi dns_sd compat library and its header files are required.

 var mdns = require('mdns'); 
// Browser.  
var browser = mdns.createBrowser('http'); 
browser.on('serviceUp', function(info, flags) { 
     console.log("up"); 
     console.dir(info);  
});  
browser.on('serviceDown', function(info, flags) { 
     console.log("down"); 
     console.dir(info); 
}); 
browser.on('serviceChanged', function(info, flags) { 
     console.log("changed"); 
     console.dir(info); 
}); 
browser.start();  
// Advertisement.  
// mdns.createAdvertisement('http', 4321).start();

6. msgpack (gave up because it is hard to compile in my centos/mac)
A space-efficient object serialization library for NodeJS

NodeJS- Routing / Redis

2011-02-12T14:31:00.000-08:00

This is an example for multiple listening ports.

 var http = require('http');  
 var connect = require('connect');  
 http.createServer(function(request, response) {  
      response.writeHead(200, {'Content-Type': 'text/plain'});  
      response.end('Hello World from 8124\n');  
 }).listen(8124);  
 var server = connect.createServer(function(req, res) {  
  res.writeHead(200, { 'Content-Type': 'text/plain' });  
  res.end('Hello World from 3000');  
 }).listen(3000);  
 console.log('Server running at http://127.0.0.1:8124 & 3000');

2. Redis Example

 var redis = require('redis'),  
      redis_client = redis.createClient();   
 redis_client.on('error', function(err) {  
      console.log("Error: " + err);  
 });   
 redis_client.set("Hello", "World", redis.print);  
 redis_client.hset("Hash", "field1", "value1", redis.print);  
 redis_client.hset(["Hash", "field2", "value2"], redis.print);  
 redis_client.hkeys("Hash", function(err, replies) {  
      console.log(replies.length + " replies:");  
      replies.forEach(function(reply, i) {  
           console.log("  " + i + ": " + reply);  
      });  
      // redis_client.quit();   
 });   
 var instruction = {  
      "id": "test1",  
      "musl": [ "1.mus", "2.mus", "3.mus"],  
      "hosts": {  
           "host_0": "a1",  
           "host_1": "a2"  
      }  
 };   

/// --> this is not supported. (no nested object.)  
 redis_client.hmset("Inst1", instruction); 


 redis_client.hgetall("Inst1", function(err, obj) {  
      console.dir(obj);   
 });   
 var str_instruction = JSON.stringify(instruction);  
 console.log("str: " + str_instruction);   
 redis_client.set("instruction", str_instruction);   
 redis_client.get("instruction", function(err, value) {  
      console.log("get: " + value);   
      console.dir(JSON.parse(value));   
 });   
 // Publish/Subscribe  
 redis_client.on('message', function(channel, message) {  
      console.log("client1 channel " + channel + ": " + message);  
 });  
 redis_client.subscribe('testengine');

3. Redis PubSub Example

 var redis = require("redis"),  
   client1 = redis.createClient(), client2 = redis.createClient(),  
   msg_count = 0;  
 client1.on("subscribe", function (channel, count) {  
   client2.publish("a nice channel", "I am sending a message.");  
   client2.publish("a nice channel", "I am sending a second message.");  
   client2.publish("a nice channel", "I am sending my last message.");  
 });  
 client1.on("message", function (channel, message) {  
   console.log("client1 channel " + channel + ": " + message);  
   msg_count += 1;  
   if (msg_count === 3) {  
     client1.unsubscribe();  
     client1.end();  
     client2.end();  
   }  
 });  
 client1.incr("did a thing");  
 client1.subscribe("a nice channel");

A Simple Bash script to see the login information.

2011-02-10T00:14:00.000-08:00

This is the script.

 #!/bin/bash  
 for user in `ls -al /home | awk '{ print $9 }'`  
 do  
   echo $user  
   last -f /var/log/wtmp.1 -1 $user | head -1  
   last -1 $user | head -1  
 done

And, this is the output.

 bxxxx  
 bxxxx  pts/0    wireless1x............ Thu Jan 27 15:56 - 16:06 (00:09)    
 bxxxx  pts/0    wireless1x-15-20 Tue Feb 1 11:44 - 12:19 (00:35)

CouchDB - Practice 01

2011-02-08T01:04:00.000-08:00

With Curl

 curl http://127.0.0.1:5984/  
 curl -X GET http://127.0.0.1:5984/_all_dbs  
 curl -X PUT http://127.0.0.1:5984/baseball  
 curl -X PUT http://127.0.0.1:5984/plankton
 curl -X DELETE http://127.0.0.1:5984/plankton

Javascript MVC

2011-02-07T12:05:00.000-08:00

This a practice to use Javascript MVC.

1. Download javascriptmvc from http://www.javascriptmvc.com/.

2. Move/Unzip and cd to your jmvc folder

3. Updates every sub libraries.

  ./js documentjs/update
./js funcunit/update
./js jquery/update
./js steal/update

4. Generates your app

 ./js jquery/generate/app cookbook
   cookbook/cookbook.css
   cookbook/cookbook.html
   cookbook/cookbook.js
   cookbook/controllers
   cookbook/docs
   cookbook/fixtures
   cookbook/funcunit.html
   cookbook/models
   cookbook/qunit.html
   cookbook/resources
   cookbook/scripts
   cookbook/scripts/build.html
   cookbook/scripts/build.js
   cookbook/scripts/clean.js
   cookbook/scripts/docs.js
   cookbook/test
   cookbook/test/funcunit
   cookbook/test/funcunit/cookbook_test.js
   cookbook/test/funcunit/funcunit.js
   cookbook/test/qunit
   cookbook/test/qunit/cookbook_test.js
   cookbook/test/qunit/qunit.js
   cookbook/views

5. Make a Create-Read-Update-Delete (CRUD) scaffold.

 $ ./js jquery/generate/scaffold Cookbook.Models.Recipe 
  cookbook/controllers
  cookbook/controllers/recipe_controller.js
  cookbook/fixtures
  cookbook/fixtures/recipes.json.get
  cookbook/models
  cookbook/models/recipe.js
  cookbook/test
  cookbook/test/funcunit
  cookbook/test/funcunit/recipe_controller_test.js
  cookbook/test/qunit
  cookbook/test/qunit/recipe_test.js
  cookbook/views
  cookbook/views/recipe
  cookbook/views/recipe/edit.ejs
  cookbook/views/recipe/init.ejs
  cookbook/views/recipe/list.ejs
  cookbook/views/recipe/show.ejs

Here's what each part does:

recipe_controller.js: Cookbook.Controllers.Recipe, like all Controllers, respond to events such as click and manipulate the DOM.
edit.ejs,init.ejs,list.ejs,show.ejs: Views are JavaScript templates for easily creating HTML.
recipe_controller_test.js: Tests the CRUD functionality of the user interface.
recipe.js: Cookbook.Models.Recipe model performs AJAX requests by manipulating services.
recipes.get: Fixtures simulate AJAX responses. This fixture responds to GET '/recipes'.
recipe_test.js: A unit test that tests Recipe model.

6. Including Scripts
After generating the scaffolding files, you must steal them in your application file. Open cookbook/cookbook.js and modify the code to steal your recipe controller and model as follows:

 steal.plugins(    
   'jquery/controller',        
   'jquery/controller/subscribe',   
   'jquery/view/ejs',   
   'jquery/controller/view',        
   'jquery/model',           
   'jquery/dom/fixture',        
   'jquery/dom/form_params')      
   .css('cookbook')    
   .resources()            
   .models('recipe')              
   .controllers('recipe')           
   .views();

cookbook/test/qunit/qunits.js

 steal  
  .plugins("funcunit/qunit", "cookbook")  
  .then("cookbook_test","recipe_test")

cookbook/test/funcunit/funcunit.js

 steal  
  .plugins("funcunit")  
  .then("cookbook_test","recipe_controller_test")

7. That's it. open cookbook/cookbook.html

CSS tutorial

2011-02-03T19:42:00.000-08:00

Syntax

Two main parts:
- A selector
- Declarations

 selector   declarations   
 h1 {color: blue; font-size:12px; }  
 p {color:red;text-align:center;}

Selectors

1. ID selector
The id selector uses the id attribute of the HTML element, and is defined with a "#". The style rule below will be applied to the element with id="para1":

 #para1  
 {  
 text-align:center;  
 color:red;  
 }

2. Class Selector

The class selector is used to specify a style for a group of elements. Unlike the id selector, the class selector is most often used on several elements. This allows you to set a particular style for any HTML elements with the same class. The class selector uses the HTML class attribute, and is defined with a "."

Studying jQuery

2011-02-02T23:02:00.000-08:00

I'm reading Essential JavaScript & jQuery Design Patterns by Addy Osmani. He and many other gurus mentioned about "A Pattern Language". Should I read the book? It was published in 1977. Sounds old, but still referred. It means it's kind of Bible and people are interested "Pattern". I don't know how they are related. There is the main and most important book, "Design Patterns: Elements of Reusable Object-Oriented Software", which introduced and made "Design Patterns" popular to the world. It defines 23 core Object-Oriented design patterns.

Addy talked about Design Patterns in jQuery and Javascript. Here are Lists.

jQuery

Lazy Initialization
Composite
Wrapper
Facade
Observer
Iterator
Strategy
Proxy
Builder
Prototype
Flyweight

Javascript

Creational
Constructor
Singleton
Module
Revealing Module
Prototype
DRY
Facade
Factory
Decorator

What is a pattern?

A: A pattern is a reusable solution that can be applied to commonly occurring problems in software design. There are three main benefits from Design Patterns.

1. Patterns are proven solutions.

2. Patterns can be easily re-used.

3. Patterns can be expressive.

However, Patterns are NOT an exact solution. It provides an idea and prototype. It can be building blocks and you can customize it for your project.

"Reusing patterns assists in preventing minor issues that can cause major problems in the application development process". - So, you can focus on your problem more. These days nobody starts from the scratch unless it is required.

"Patterns can provide generalized solutions which are documented in a fashion that doesn't require them to be tied to a specific problem."

"Certain patterns can actually decrease the overall file-size footprint of your code by avoiding repetition." - DRY (Don't Repeat Yourself)

A Good Pattern does these.

- Solves a particular problem.

- The solution to this problem cannot be obvious.

- The concept described must have been proven.

- It must describe a relationship.

Here is the rule of three for a valid pattern.

1. Fitness of purpose. - How?

2. Usefulness. - Why?

3. Applicability. - Wide?

----

The structure of a design pattern.

A design pattern must have a

Pattern Name and a description
Context outline
Problem Statement
Solution
Design
Implementation
Illustrations.
Examples
Co-requisites
Relations
Known usage
Discussions

Writing Design Patterns

Bare in mind practicability
Ensure that you draw upon best practices.
Your design patterns should be transparent to the user.
Remember that originality is not key in pattern design.
Know the differences between patterns and design.
Your pattern needs to have a strong set of examples.

Installing Zope and Plone from source on Ubuntu in 10 easy steps

2009-07-28T22:18:00.000-07:00

from : http://www.paolocorti.net/2008/01/02/installing-zope-and-plone-from-source-on-ubuntu-in-10-easy-steps/

Installing Zope and Plone from source on Ubuntu in 10 easy steps

Posted by Paolo Corti on January 2, 2008

Here are my notes for installing Zope and Plone from source in Ubuntu. I took this notes installing on 7.10 (Gutsy) but should work without problems for earlier Ubuntu versions, and for other Linux platforms

-1- install libraries needed for build (build-essential) and Zope

sudo apt-get install build-essential python2.4-dev python-lxml python-elementtree python-imaging

-2- download zope 2.10.5 and plone 3.0.4

wget https://launchpad.net/plone/3.0/3.0.4/+download/Plone-3.0.4.tar.gz wget http://www.zope.org/Products/Zope/2.10.5/Zope-2.10.5-final.tgz

-3- unzip the downloaded files

tar -xvzf Zope-2.10.5-final.tgz tar -xvzf Plone-3.0.4.tar.gz

-4- create the makefile and set the prefix

cd Zope-2.10.5-final

way1:

./configure gedit makefile  and set  prefix=/opt/Zope-2.10.5

or way2 (simpler):

 ./configure --prefix=/opt/Zope-2.10.5

-5- Install Zope (at the location set in prefix)

sudo make install

-6- create an user for zope

not run as suggested: “Now run ‘/opt/Zope-2.10.5/bin/mkzopeinstance.py’” but create an user for zope

sudo adduser zope

-7- create one ore more Zope instances (here we will create just one single instance)

create a directory where to place each Zope instance

sudo mkdir /srv/zope/inst_2_10_5

give to this directory ownership to zope user:

sudo chown zope.zope /srv/zope/inst_2_10_5

run mkzopeinstance.py as the zope user:

sudo su zope /opt/Zope-2.10.5/bin/mkzopeinstance.py Directory: /srv/zope/inst_2_10_5 Username: admin Password: ****

-8- Run Zope and test Zope installation

If you want to run Zope on a port different from 8080:

sudo gedit /srv/zope/inst_2_10_5/etc/zope.conf

go to server directive:

>   # valid keys are "address" and "force-connection-close"   address 8080   # force-connection-close on >

and change 8080 to whatever port you want to run the Zope process

Now let’s start Zope: you need to run the Zope instance as the zope user

sudo su zope /srv/zope/inst_2_10_5/bin/zopectl fg

fg is the foreground option for the Zope process. Without using it you will enter an interactive mode, where you can directly send commands (like start, stop, fg, debug).

Now if you go to the address: http://localhost:8080 (change 8080 to the port you are running the Zope process) you will be promptet for the user and password of the administrative user of ZMI.

Now to stop the zope fg process (in order to install Plone) just type ctrl+Z and kill the python process:

sudo killall -9 python2.4

-9- Install Plone

9.a) For Plone 3.x

Copy Plone products and Python libs to the zope instance (products directory, base Zope is without products)

sudo cp -r Plone-3.0.4/Products/* /srv/zope/inst_2_10_5/Products/ sudo cp -r Plone-3.0.4/lib/python/* /srv/zope/inst_2_10_5/lib/python/

give permission to zope user:

sudo chown -R zope.zope /srv/zope/inst_2_10_5

9.b) For Plone 2.x

you just need to copy the Plone products

sudo cp -r Plone-3.0.4/* /srv/zope/inst_plone2/Products/

note: for any Plone (2.x, 3.x) if you copy the Products under the /opt/Zope-2.10.5/skell/Products directory, each instance that will be created will already have all the products. Same thing for the lib/python directory.
Or, if you copy the products in /opt/Zope-2.10.5/lib/python/Products directory, all the zope instances will use that directory without copying them in each instance.

-10- Run Zope and test Plone

Now if you run the Zope process and you go to the address: http://localhost:8080 (change 8080 to the port you are running the Zope process) you will access the ZMI.
There you will create your first Plone site (by selecting “Plone Site” and then clicking the “add” button)

If you named the Plone Site “test” then you can go there by typing on the browser: http://localhost:8080/test. You are now on a new ajaxifed Plone 3.0 web site: congrats!

-extra step- debugging Zope

Run zope without the fg option and enter in the debug command prompt:

/srv/zope/inst_2_10_5/bin/zopectl zopectl> start --will start zope zopectl> debug --will debug zope zopectl> stop --will stop zope

Now under the Python shell you can debug Zope (and Plone). To exit from debug type ctrl+d.

>>> dir() ....many resources >>> dir(app) ...many resources >>> dir(app.test) --if you named your Plone site as "test" >>> print app.test

Transactional Datastore

2009-07-17T11:56:00.000-07:00

Chordless

(http://chordless.wiki.sourceforge.net/Chordless is on the lowest level a distributed hash table modeled on Chord and DHash.)

It implements most features talked about in Robust and Efficient Data Management for a Distributed Hash Table and Chord: A Scalable Peer-to-peer Lookup Protocol for Internet Applications with a few exceptions (mostly the lack of erasure codes for storage - if those interest you, look at archipelago, where the rubygem oneliner is a C/ruby implementation of online codes) and a few extras such as support for ACID transactions, remote execution, scalable data structures and a nifty graphical user interface.

scalaris

Distributed Transactional Key-Value Store

Scalaris is a distributed key-value store.

Scalaris is a scalable, transactional, distributed key-value store. It can be used for building scalable Web 2.0 services.

Scalaris uses a structured overlay with a non-blocking Paxos commit protocol for transaction processing with strong consistency over replicas. Scalaris is implemented in Erlang.

Mailing list: http://groups.google.com/group/scalaris

Documentation:

GettingStarted
APIs
FAQ
Users and developers guide in the SVN: trunk/user-dev-guide/main.pdf

Related projects:

http://code.google.com/p/mochiweb-scalaris/

The Scalaris project was initiated by Zuse Institute Berlin and onScale solutions GmbH and is partly funded by the EU projects Selfman and XtreemOS. More information (papers, videos) can be found at http://www.zib.de/CSR/Projects/scalaris and http://www.onscale.de/scalaris.html.

Using JConsole to Monitor Applications

2009-07-06T16:31:00.001-07:00

Using jconsole

Jconsole is a JMX-compliant monitoring tool. It uses the extensive JMX instrumentation of the Java virtual machine to provide information on performance and resource consumption of applications running on the Java platform.

Starting jconsole

The jconsole executable is in JDK_HOME/bin, where JDK_HOME is the directory where the JDK is installed. If this directory is on your system path, you can start the tool by simply typing jconsolein a command (shell) prompt. Otherwise, you have to type the full path to the executable file.

Command Syntax

You can use jconsole to monitor both local applications (those running on the same system as jconsole) and remote applications (those running on other systems).

Note: Using jconsole to monitor a local application is useful for development and prototyping, but is not recommended for production environments, because jconsole itself consumes significant system resources. Remote monitoring is recommended to isolate the jconsole application from the platform being monitored.

For a complete reference on jconsole command syntax, see jconsole - Java Monitoring and Management Console.

Local Monitoring

To monitor a local application, it must be running with the same user ID as jconsole. The command syntax to start jconsole for local monitoring is:

jconsole [processID]

where processID is the application's process ID (PID). To determine an application's PID:

On Unix or Linux systems, use the ps command to find the PID of java.
On Windows systems, use Task Manager to find the PID of java or javaw.

You can also use the jps command-line utility to determine PIDs.

For example, if you determined that the process ID of the Notepad application is 2956, then you would start jconsole as follows:

jconsole 2956

Both jconsole and the application must by executed by the same user name. The management and monitoring system uses the operating system's file permissions.

If you don't specify a process ID, jconsole will automatically detect all local Java applications, and display a dialog box that lets you select the one you want to monitor (see the next section).

For more information, see Local JMX Monitoring and Management.

Remote Monitoring

To start jconsole for remote monitoring, use this command syntax:

jconsole [hostName:portNum]

where hostName is the name of the system running the application and portNum is the port number you specified when you enabled the JMX agent when you started the JVM. For more information, see Remote JMX Monitoring and Management.

If you do not specify a host name/port number combination, then jconsole will display a connection dialog box (see the next section) enabling you to enter a host name and port number.

Connecting to a JMX Agent

If you start jconsole with arguments specifying a JMX agent to which to connect, it will automatically start monitoring the specified JVM. You can connect to a different host at any time by choosingConnection | New Connection. and entering the necessary information.

Otherwise, if you do not provide any arguments when you start jconsole, the first thing you see is the connection dialog box. This dialog box has three tabs:

Local
Remote
Advanced

Local Tab

The local tab lists any JVMs running on the local system started with the same user ID as jconsole, along with their process ID and class/argument information. Select the application you want to monitor, then click Connect.

Remote Tab

To monitor a remote JVM, enter:

Host name: name of the machine on which the JVM is running.
Port number: the JMX agent port number you specified when you started the JVM.
User name and password: the user name and password to use (required only if monitoring a JVM through a JMX agent that requires password authentication).

For information on setting the port number of the JMX agent, see Enabling the JMX Management Agent. For information on user names and passwords, see Using Password and Access Files.

To monitor the JVM running jconsole, simply click Connect, using host localhost and the port zero (0).

Advanced Tab

The advanced tab enables you to connect to other JMX agents (MBean servers) by specifying their JMX URL, and the user name and password. The syntax of a JMX URL is described in the API documentation for javax.management.remote.JMXServiceURL.

Note: If the JMX agent is using in a connector which is not included in the Java platform, you need to add the connector classes to the classpath when running jconsole as follow:

jconsole -J-Djava.class.path=JAVA_HOME/lib/jconsole.jar:JAVA_HOME/lib/tools.jar:connector-path

where JAVA_HOME is the directory containing the JDK and connector-path is the directory or the jar file containing the classes not included in JDK 5.0 to be used by jconsole for connecting to a JMX agent.

The jconsole interface

The jconsole interface is composed of six tabs:

Summary tab: displays summary information on the JVM and monitored values.
Memory tab: displays information on memory use.
Threads tab: displays information on thread use.
Classes tab: displays information on class loading
MBeans tab: displays information on MBeans
VM tab: displays information on the JVM

The following sections provide information on each tab.

Viewing Summary Information

The Summary tab displays some key monitoring information on thread usage, memory consumption, and class loading, plus information on the JVM and operating system.

Summary

Uptime: how long the JVM has been running
Total compile time: the amount of time spent in just-in-time (JIT) compilation.
Process CPU time: the total amount of CPU time consumed by the JVM

Threads

Live threads: Current number of live daemon threads plus non-daemon threads
Peak: Highest number of live threads since JVM started.
Daemon threads: Current number of live daemon threads
Total started: Total number of threads started since JVM started (including daemon, non-daemon, and terminated).

Memory

Current heap size: Number of Kbytes currently occupied by the heap.
Committed memory: Total amount of memory allocated for use by the heap.
Maximum heap size: Maximum number of Kbytes occupied by the heap.
Objects pending for finalization: Number of objects pending for finalization.
Garbage collector information: Information on GC, including the garbage collector names, number of collections performed, and total time spent performing GC.

Classes

Current classes loaded: Number of classes currently loaded into memory.
Total classes loaded: Total number of classes loaded into memory since the JVM started, included those subsequently unloaded.
Total classes unloaded: Number of classes unloaded from memory since the JVM started.

Operating System

Total physical memory: Amount of random-access memory (RAM) that the OS has.
Free physical memory: Amount of free RAM the OS has.
Committed virtual memory: Amount of virtual memory guaranteed to be available to the running process.

Monitoring Memory Consumption

The Memory tab provides information on memory consumption and memory pools.

The chart shows the JVM's memory use versus time, for heap and non-heap memory, and for specific memory pools. The memory pools available depend on the JVM being used. For the HotSpot JVM, the pools are:

Eden Space (heap): pool from which memory is initially allocated for most objects.
Survivor Space (heap): pool containing objects that have survived GC of eden space.
Tenured Generation (heap): pool containing objects that have existed for some time in the survivor space.
Permanent Generation (non-heap): holds all the reflective data of the virtual machine itself, such as class and method objects. With JVMs that use class data sharing, this generation is divided into read-only and read-write areas.
Code Cache (non-heap): HotSpot JVM also includes a "code cache" containing memory used for compilation and storage of native code.

For more information on these memory pools, see Garbage Collection.

The Details area shows several current memory metrics:

Used: the amount of memory currently used. Memory used includes the memory occupied by all objects including both reachable and unreachable objects.
Committed: the amount of memory guaranteed to be available for use by the JVM. The amount of committed memory may change over time. The Java virtual machine may release memory to the system and committed could be less than the amount of memory initially allocated at startup. Committed will always be greater than or equal to used.
Max: the maximum amount of memory that can be used for memory management. Its value may change or be undefined. A memory allocation may fail if the JVM attempts to increase the used memory to be greater than committed memory, even if the amount used is less than or equal to max (for example, when the system is low on virtual memory).

The bar chart at the lower right shows memory consumed by the memory pools in heap and non-heap memory. The bar will turn red when the memory used exceeds the memory usage threshold. You can set the memory usage threshold through an attribute of the MemoryMXBean.

Heap and Non-heap Memory

The JVM manages two kinds of memory: heap and non-heap memory, both created when it starts.

Heap memory is the runtime data area from which the JVM allocates memory for all class instances and arrays. The heap may be of a fixed or variable size. The garbage collector is an automatic memory management system that reclaims heap memory for objects.

Non-heap memory includes a method area shared among all threads and memory required for the internal processing or optimization for the JVM. It stores per-class structures such as a runtime constant pool, field and method data, and the code for methods and constructors. The method area is logically part of the heap but, depending on implementation, a JVM may not garbage collect or compact it. Like the heap, the method area may be of fixed or variable size. The memory for the method area does not need to be contiguous.

In addition to the method area, a JVM implementation may require memory for internal processing or optimization which also belongs to non-heap memory. For example, the JIT compiler requires memory for storing the native machine code translated from the JVM code for high performance.

Memory Pools and Memory Managers

Memory pools and memory managers are key aspects of the JVM memory system.

A memory pool represents a memory area that the JVM manages. The JVM has at least one memory pool and it may create or remove memory pools during execution. A memory pool can belong to either heap or non-heap memory.

A memory manager manages one or more memory pools. The garbage collector is a type of memory manager responsible for reclaiming memory used by unreachable objects. A JVM may have one or more memory managers. It may add or remove memory managers during execution. A memory pool can be managed by more than one memory manager.

Garbage Collection

Garbage collection (GC) is how the JVM frees memory occupied by objects that are no longer referenced. It is common to think of objects that have active references as being "alive" and un-referenced (or unreachable) objects as "dead." Garbage collection is the process of releasing memory used by the dead objects. The algorithms and parameters used by GC can have dramatic effects on performance.

The HotSpot VM garbage collector uses generational garbage collection. Generational GC takes advantage of the observation that, in practice, most programs create:

many objects that have short lives (for example, iterators and local variables).
some objects that have very long lifetimes (for example, high level persistent objects)

So, generational GC divides memory into several generations, and assigns each a memory pool. When a generation uses up its allotted memory, the VM performs a partial garbage collection (also called a minor collection) on that memory pool to reclaim memory used by dead objects. This partial GC is usually much faster than a full GC.

The HotSpot VM defines two generations: the young generation (sometimes called the "nursery") and the old generation. The young generation consists of an "eden space" and two "survivor spaces." The VM initially assigns all objects to the eden space, and most objects die there. When it performs a minor GC, the VM moves any remaining objects from the eden space to one of the survivor spaces. The VM moves objects that live long enough in the survivor spaces to the "tenured" space in the old generation. When the tenured generation fills up, there is a full GC that is often much slower because it involves all live objects. The permanent generation holds all the reflective data of the virtual machine itself, such as class and method objects.

The default arrangement of generations looks something like this:

As explained in the following documents, if the garbage collector has become a bottleneck, you can improve performance by customizing the generation sizes. Using jconsole, explore the sensitivity of your performance metric to the garbage collector parameters. For more information, see:

Tuning Garbage collection with the 5.0 HotSpot VM

Monitoring Thread Use

The Threads tab provides information on thread use.

The Threads list in the lower left lists all the active threads. If you enter a string in the Filter field, the Threads list will show only those threads whose name contains the string you enter. Click on the name of a thread in the Threads list to display information about that thread to the right, including the thread name, state, and stack trace.

The chart shows the number of live threads versus time. Three lines are shown:

Magenta: total number of threads
Red: peak number of threads
Blue: number of live threads.

See java.lang.Thread for more information about threads and daemon threads.

Monitoring Class Loading

The Classes tab displays information on class loading.

The graph plots the number of classes loaded versus time:

Red line is the total number of classes loaded (including those subsequently unloaded).
Blue line is the current number of classes loaded.

The Details section at the bottom of the tab displays the total number of classes loaded since the JVM started, the number currently loaded and the number unloaded.

Monitoring and Managing MBeans

The MBean tab displays information on all the MBeans registered with the platform MBean server.

The tree on the left shows all the MBeans, organized according to their objectNames. When you select an MBean in the tree, its attributes, operations, notifications and other information is displayed on the right.

You can set the value of attributes, if they are writeable (the value will be displayed in blue). You can also invoke operations displayed in the Operations tab.

Displaying a Chart

You can display a chart of an attribute's value versus time by double-clicking on the attribute value. For example, if you click on the value of the CollectionTime property of java.lang.GarbageCollector.Copy MBean, you will see a chart that looks something like this:

Viewing VM Information

The VM tab provides information on the JVM.

The information includes:

Uptime: Total amount of time since the JVM was started.
Process CPU Time: Total amount of CPU time that the JVM has consumed since it was started.
Total Compile Time: Total accumulated time spent in just-in-time (JIT) compilation. The JVM implementation determines when JIT compilation occurs. The Hotspot VM uses adaptive compilation, in which the VM launches an application using a standard interpreter, but then analyzes the code as it runs to detect performance bottlenecks, or "hot spots".

Apache ActiveMQ

2009-07-06T16:16:00.000-07:00

Apache ActiveMQ is the most popular and powerful open source Enterprise Messaging andIntegration Patterns provider.

Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License

Grab yourself a Download, try our Getting Started Guide, surf our FAQ or start Contributing and join us on our Discussion Forums.

Features

Supports a variety of Cross Language Clients and Protocols from Java, C, C++, C#, Ruby, Perl, Python, PHP
- OpenWire for high performance clients in Java, C, C++, C#
- Stomp support so that clients can be written easily in C, Ruby, Perl, Python, PHP, ActionScript/Flash, Smalltalk to talk to ActiveMQ as well as any other popular Message Broker
full support for the Enterprise Integration Patterns both in the JMS client and the Message Broker
Supports many advanced features such as Message Groups, Virtual Destinations, Wildcards and Composite Destinations
Fully supports JMS 1.1 and J2EE 1.4 with support for transient, persistent, transactional and XA messaging
Spring Support so that ActiveMQ can be easily embedded into Spring applications and configured using Spring's XML configuration mechanism
Tested inside popular J2EE servers such as Geronimo, JBoss 4, GlassFish and WebLogic
- Includes JCA 1.5 resource adaptors for inbound & outbound messaging so that ActiveMQ should auto-deploy in any J2EE 1.4 compliant server
Supports pluggable transport protocols such as in-VM, TCP, SSL, NIO, UDP, multicast, JGroups and JXTA transports
Supports very fast persistence using JDBC along with a high performance journal
Designed for high performance clustering, client-server, peer based communication
REST API to provide technology agnostic and language neutral web based API to messaging
Ajax to support web streaming support to web browsers using pure DHTML, allowing web browsers to be part of the messaging fabric
CXF and Axis Support so that ActiveMQ can be easily dropped into either of these web service stacks to provide reliable messaging
Can be used as an in memory JMS provider, ideal for unit testing JMS

Starting ActiveMQ

There now follows instructions on how to run the ActiveMQ Message Broker.

On Windows:

From a console window, change to the installation directory and run ActiveMQ:

cd [activemq_install_dir]

where activemq_install_dir is the directory in which ActiveMQ was installed, e.g., c:\Program Files\ActiveMQ-4.x.
Then type:

bin\activemq

NOTE: Working directories get created relative to the current directory. To create working directories in the proper place, ActiveMQ must be launched from its home/installation directory.

On Unix:

From a command shell, change to the installation directory and run ActiveMQ:

cd [activemq_install_dir]

where activemq_install_dir is the directory in which ActiveMQ was installed, e.g., /usr/local/activemq-4.x.
Then type:

bin/activemq  OR  bin/activemq > /tmp/smlog  2>&1 &; Note: /tmp/smlog may be changed to another file name.

NOTE: Working directories get created relative to the current directory. To create working directories in the proper place, ActiveMQ must be launched from its home/installation directory.

More help

For other ways of running the broker see Here. For example you can run an embedded broker inside your JMS Connection to avoid starting a separate process.

Testing the Installation

If ActiveMQ is up and running without problems, the Window's console window or the Unix command shell will display information similar to the following log line:

INFO  ActiveMQ JMS Message Broker (ID:apple-s-Computer.local-51222-1140729837569-0:0) has started

ActiveMQ's default port is 61616. From another window run netstat and search for port 61616.

From a Windows console, type:

netstat -an|find "61616"

From a Unix command shell, type:

netstat -an|grep 61616

Monitoring ActiveMQ

There are various ways to monitor ActiveMQ. If you are on version 4.2 or later of ActiveMQ you can then monitor ActiveMQ using the Web Console by pointing your browser at

http://localhost:8161/admin

Or you can use the JMX support to view the running state of ActiveMQ.

Stopping ActiveMQ

For both Windows and Unix installations, terminate ActiveMQ by typing "CTRL-C" in the console or command shell in which it is running.

If ActiveMQ was started in the background on Unix, the process can be killed, with the following:

ps -ef|grep activemq kill [PID]   where [PID] is the process id of the ActiveMQ process.

Configuring ActiveMQ

The ActiveMQ broker should now run. You can configure the broker by specifying an Xml Configuration file as a parameter to theactivemq command. An alternative is to use the Broker Configuration URI to configure things on the command line in a concise format (though the configuration options are not as extensive as if you use Java or XML code). You can also

Also see Configuring Transports to see how you can configure the various connection, transport and broker options using the connection URL in the ActiveMQConnectionFactory.

See the Initial Configuration for details of which jars you need to add to your classpath to start using ActiveMQ in your Java code

If you want to use JNDI to connect to your JMS provider then please view the JNDI Support. If you are a Spring user you should read about Spring Support

After the installation, ActiveMQ is running with a basic configuration. For details on configuring options, please see refer to theConfiguration section.

Additional Resources

If you are new to using ActiveMQ, running the Web Samples or the Examples is a good next step to learn more about ActiveMQ.

FUSE Source ActiveMQ Getting Started Guide (Note: FUSE Message Broker is an open source Apache licenced enterprise version of ActiveMQ)

The Grinder, a Java Load Testing Framework

2009-07-06T16:14:00.000-07:00

The Grinder, a Java Load Testing Framework

What is The Grinder?

The Grinder is a Java^TM load testing framework that makes it easy to run a distributed test using many load injector machines. It is freely available under a BSD-style open-source license.

The latest news, downloads, and mailing list archives can be found on SourceForge.net.

Key features

Generic Approach Load test anything that has a Java API. This includes common cases such as HTTP web servers, SOAP and REST web services, and application servers (CORBA, RMI, JMS, EJBs), as well as custom protocols.
Flexible Scripting Tests are written in the powerful Jython scripting language.
Distributed Framework A graphical console allows multiple load injectors to be monitored and controlled, and provides centralised script editing and distribution.
Mature HTTP Support Automatic management of client connections and cookies. SSL. Proxy aware. Connection throttling. Sophisticated record and replay of the interaction between a browser and a web site.

See the longer features list for further details.

History

The Grinder was originally developed for the book Professional Java 2 Enterprise Edition with BEA WebLogic Serverby Paco Gómez and Peter Zadrozny. Philip Aston took ownership of the code, reworked it to create The Grinder 2, and shortly after began work on The Grinder 3. The Grinder 3 provides many new features, the most significant of which is Jython scripting. Philip continues to enhance and maintain The Grinder.

In 2003, Peter, Philip and Ted Osborne published the book J2EE Performance Testing which makes extensive use of The Grinder 2.

Capabilities of The Grinder

Load Testing	Load Testing determines if an application can support a specified load (for example, 500 concurrent users) with specified response times. Load Testing is used to create benchmarks.
Capacity Testing	Capacity Testing determines the maximum load that an application can sustain before system failure.
Functional Testing	Functional Testing proves the correct behaviour of an application.
Stress Testing	Stress Testing is load testing over an extended period of time. Stress Testing determines if an application can meet specified goals for stability and reliability, under a specified load, for a specified time period.

Open Source

BSD style license	The Grinder is distributed under a BSD style license.
Dependencies	The Grinder depends on a number of other open source products including Jython HTTPClient JEdit Syntax Apache XMLBeans PicoContainer

Standards

100% Pure Java	The Grinder works on any hardware platform and any operating system that supports J2SE 1.4 and above.
Web Browsers	The Grinder can simulate web browsers and other devices that use HTTP, and HTTPS.
Web Services	The Grinder can be used to test Web Service interfaces using protocols such as SOAP and XML-RPC.
Database	The Grinder can be used to test databases using JDBC.
Middleware	The Grinder can be used to test RPC and MOM based systems using protocols such as IIOP, RMI/IIOP, RMI/JRMP, and JMS.
Other Internet protocols	The Grinder can be used to test systems that utilise other protocols such as POP3, SMTP, FTP, and LDAP.

The Grinder Architecture

Goal	Minimize system resource requirements while maximizing the number of test contexts ("virtual users").
Multi-threaded, multi-process	Each test context runs in its own thread. The threads can be split over many processes depending on the requirements of the test and the capabilities of the load injection machine.
Distributed	The Grinder makes it easy to coordinate and monitor the activity of processes across a network of many load injection machines from a central console.
Scalable	The Grinder typically can support several hundred HTTP test contexts per load injection machine. (The number varies depending on the type of test client). More load injection machines can be added to generate bigger loads.

Console

Graphical Interface	100% Java Swing user interface.
Process coordination	Worker processes can be started, stopped and reset from one central console.
Process monitoring	Dynamic display of current worker processes and threads.
Internationalised and Localised	English, French, Spanish, and German translations are supplied. Users can add their own translations.
Script editing	Central editing and management of test scripts.

Statistics, Reports, Charts

Test monitoring	Pre-defined charts for response time, test throughput. Display the number of invocations, test result (pass/fail), average, minimum and maximum values for response time and tests per second for each test.
Data collation	Collates data from worker processes. Data can be saved for import into a spreadsheet or other analysis tool.
Instrument anything	The Grinder records statistics about the number of times each test has been called and the response times achieved. Any part of the test script can be marked as a test.
Statistics engine	Scripts can declare their own statistics and report against them. The values will appear in the console and the data logs. Composite statistics can be specified as expressions involving other statistics.

Script

Record real users	Scripts can be created by recording actions of a real user using the TCP Proxy. The script can then be customised by hand.
Powerful scripting in Python	Simple to use but powerful, fully object-oriented scripting.
Multiple scenarios	Arbitrary looping and branching allows the simulation of multiple scenarios. Simple scenarios can be composed into more complex scenarios. For example, you might allocate 10% of test contexts to a login scenario, 70% to searching, 10% to browsing, and 10% to buying; or you might have different workloads for specific times of a day.
Access to any Java API	Jython allows any Java-based API to be used directly from the test script.
Parameterization of input data	Input data (e.g. URL parameters, form fields) can be dynamically generated. The source of the data can be anything including flat files, random generation, a database, or previously captured output.
Content Verification	Scripts have full access to test results. In the future, The Grinder will include support for enhanced parsing of common results such as HTML pages.

The Grinder Plug-ins

HTTP	The Grinder has special support for HTTP that automatically handles cookie and connection management for test contexts.
Custom	Users can write their own plug-ins to a documented interface; although this is rarely necessary due to the powerful scripting facilities.

HTTP Plug-in

HTTP 1.0, HTTP 1.1	Support for both HTTP 1.0 and HTTP 1.1 is provided.
HTTPS	The Grinder supports HTTP over SSL.
Cookies	Full support for Cookies is provided.
Multi-part forms	The Grinder supports multi-part forms.
Connection throttling	Low bandwidth client connections can be simulated.

TCP Proxy

TCP proxy	A TCP proxy utility is supplied that can be used to intercept system interaction at the protocol level. It is useful for recording scripts and as a debugging tool.
HTTP Proxy	The TCP proxy can be configured as an HTTP/HTTPS proxy for easy integration with web browsers.
SSL Support	The TCP proxy can simulate SSL sessions.
Filter-based architecture	The TCP proxy has a pluggable filter architecture. Users can write their own filters.

Documentation

User Guide	http://grinder.sourceforge.net/g3/whats-new.html
FAQs	http://grinder.sourceforge.net/faq.html
Tutorial	http://grinder.sourceforge.net/g3/tutorial-perks.html
Script Gallery	http://grinder.sourceforge.net/g3/script-gallery.html
Articles	http://grinder.sourceforge.net/links.html
Commercial books	Professional Java 2 Enterprise Edition with BEA WebLogic Server

Apache ZooKeeper

2009-07-06T16:12:00.000-07:00

Welcome to Apache ZooKeeper!

What is ZooKeeper?

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them ,which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

Learn more about Zookeeper on the ZooKeeper Wiki

ZooKeeper: A Distributed Coordination Service for Distributed Applications

ZooKeeper is a distributed, open-source coordination service for distributed applications. It exposes a simple set of primitives that distributed applications can build upon to implement higher level services for synchronization, configuration maintenance, and groups and naming. It is designed to be easy to program to, and uses a data model styled after the familiar directory tree structure of file systems. It runs in Java and has bindings for both Java and C.

Coordination services are notoriously hard to get right. They are especially prone to errors such as race conditions and deadlock. The motivation behind ZooKeeper is to relieve distributed applications the responsibility of implementing coordination services from scratch.

Design Goals

ZooKeeper is simple. ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchal namespace which is organized similarly to a standard file system. The name space consists of data registers - called znodes, in ZooKeeper parlance - and these are similar to files and directories. Unlike a typical file system, which is designed for storage, ZooKeeper data is kept in-memory, which means ZooKeeper can acheive high throughput and low latency numbers.

The ZooKeeper implementation puts a premium on high performance, highly available, strictly ordered access. The performance aspects of ZooKeeper means it can be used in large, distributed systems. The reliability aspects keep it from being a single point of failure. The strict ordering means that sophisticated synchronization primitives can be implemented at the client.

ZooKeeper is replicated. Like the distributed processes it coordinates, ZooKeeper itself is intended to be replicated over a sets of hosts called an ensemble.

ZooKeeper Service

The servers that make up the ZooKeeper service must all know about each other. They maintain an in-memory image of state, along with a transaction logs and snapshots in a persistent store. As long as a majority of the servers are available, the ZooKeeper service will be available.

Clients connect to a single ZooKeeper server. The client maintains a TCP connection through which it sends requests, gets responses, gets watch events, and sends heart beats. If the TCP connection to the server breaks, the client will connect to a different server.

ZooKeeper is ordered. ZooKeeper stamps each update with a number that reflects the order of all ZooKeeper transactions. Subsequent operations can use the order to implement higher-level abstractions, such as synchronization primitives.

ZooKeeper is fast. It is especially fast in "read-dominant" workloads. ZooKeeper applications run on thousands of machines, and it performs best where reads are more common than writes, at ratios of around 10:1.

Data model and the hierarchical namespace

The name space provided by ZooKeeper is much like that of a standard file system. A name is a sequence of path elements separated by a slash (/). Every node in ZooKeeper's name space is identified by a path.

ZooKeeper's Hierarchical Namespace

Nodes and ephemeral nodes

Unlike is standard file systems, each node in a ZooKeeper namespace can have data associated with it as well as children. It is like having a file-system that allows a file to also be a directory. (ZooKeeper was designed to store coordination data: status information, configuration, location information, etc., so the data stored at each node is usually small, in the byte to kilobyte range.) We use the term znode to make it clear that we are talking about ZooKeeper data nodes.

Znodes maintain a stat structure that includes version numbers for data changes, ACL changes, and timestamps, to allow cache validations and coordinated updates. Each time a znode's data changes, the version number increases. For instance, whenever a client retrieves data it also receives the version of the data.

The data stored at each znode in a namespace is read and written atomically. Reads get all the data bytes associated with a znode and a write replaces all the data. Each node has an Access Control List (ACL) that restricts who can do what.

ZooKeeper also has the notion of ephemeral nodes. These znodes exists as long as the session that created the znode is active. When the session ends the znode is deleted. Ephemeral nodes are useful when you want to implement [tbd].

Conditional updates and watches

ZooKeeper supports the concept of watches. Clients can set a watch on a znodes. A watch will be triggered and removed when the znode changes. When a watch is triggered the client receives a packet saying that the znode has changed. And if the connection between the client and one of the Zoo Keeper servers is broken, the client will receive a local notification. These can be used to [tbd].

Guarantees

ZooKeeper is very fast and very simple. Since its goal, though, is to be a basis for the construction of more complicated services, such as synchronization, it provides a set of guarantees. These are:

Sequential Consistency - Updates from a client will be applied in the order that they were sent.
Atomicity - Updates either succeed or fail. No partial results.
Single System Image - A client will see the same view of the service regardless of the server that it connects to.

Reliability - Once an update has been applied, it will persist from that time forward until a client overwrites the update.

Timeliness - The clients view of the system is guaranteed to be up-to-date within a certain time bound.

For more information on these, and how they can be used, see [tbd]

Simple API

One of the design goals of ZooKeeper is provide a very simple programming interface. As a result, it supports only these operations:

create: creates a node at a location in the tree
delete: deletes a node
exists: tests if a node exists at a location
get data: reads the data from a node
set data: writes data to a node
get children: retrieves a list of children of a node
sync: waits for data to be propagated

For a more in-depth discussion on these, and how they can be used to implement higher level operations, please refer to [tbd]

Implementation

ZooKeeper Components shows the high-level components of the ZooKeeper service. With the exception of the request processor, each of the servers that make up the ZooKeeper service replicates its own copy of each of components.

ZooKeeper Components

The replicated database is an in-memory database containing the entire data tree. Updates are logged to disk for recoverability, and writes are serialized to disk before they are applied to the in-memory database.

Every ZooKeeper server services clients. Clients connect to exactly one server to submit irequests. Read requests are serviced from the local replica of each server database. Requests that change the state of the service, write requests, are processed by an agreement protocol.

As part of the agreement protocol all write requests from clients are forwarded to a single server, called the leader. The rest of the ZooKeeper servers, calledfollowers, receive message proposals from the leader and agree upon message delivery. The messaging layer takes care of replacing leaders on failures and syncing followers with leaders.

ZooKeeper uses a custom atomic messaging protocol. Since the messaging layer is atomic, ZooKeeper can guarantee that the local replicas never diverge. When the leader receives a write request, it calculates what the state of the system is when the write is to be applied and transforms this into a transaction that captures this new state.

Uses

The programming interface to ZooKeeper is deliberately simple. With it, however, you can implement higher order operations, such as synchronizations primitives, group membership, ownership, etc. Some distributed applications have used it to: [tbd: add uses from white paper and video presentation.] For more information, see

rsync

2009-07-06T16:10:00.000-07:00

from: http://www.cyberciti.biz/tips/linux-use-rsync-transfer-mirror-files-directories.html

How to use rsync for transferring files under Linux or UNIX

How do you install and use rsync to synchronize files and directories from one location (or one server) to another location? - A common question asked by new sys admin.

rsync is a free software computer program for Unix and Linux like systems which synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction.

So what is unique about rsync?

It can perform differential uploads and downloads (synchronization) of files across the network, transferring only data that has changed. The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection.

How do I install rsync?

Use any one of the following commands to install rsync.

If you are using Debian or Ubuntu Linux, type the following command

# apt-get install rsync
OR
$ sudo apt-get install rsync

If you are using Red Hat Enterprise Linux (RHEL), type the following command

# up2date rsync

If you are using CentOS/Fedora Core Linux, type the following command

# yum install rsync

Always use rsync over ssh

Since rsync does not provide any security while transferring data it is recommended that you use rsync over ssh . This allows a secure remote connection. Now let us see some examples of rsync.

rsync command common options

--delete : delete files that don't exist on sender (system)
-v : Verbose (try -vv for more detailed information)
-e "ssh options" : specify the ssh as remote shell
-a : archive mode
-r : recurse into directories
-z : compress file data

Task : Copy file from a local computer to a remote server

Copy file from /www/backup.tar.gz to a remote server called openbsd.nixcraft.in
$ rsync -v -e ssh /www/backup.tar.gz jerry@openbsd.nixcraft.in:~Output:

Password: sent 19099 bytes  received 36 bytes  1093.43 bytes/sec total size is 19014  speedup is 0.99

Please note that symbol ~ indicate the users home directory (/home/jerry).

Task : Copy file from a remote server to a local computer

Copy file /home/jerry/webroot.txt from a remote server openbsd.nixcraft.in to a local computer /tmp directory:
$ rsync -v -e ssh jerry@openbsd.nixcraft.in:~/webroot.txt /tmp
Password

Task: Synchronize a local directory with a remote directory

$ rsync -r -a -v -e "ssh -l jerry" --delete openbsd.nixcraft.in:/webroot/ /local/webroot

Task: Synchronize a remote directory with a local directory

$ rsync -r -a -v -e "ssh -l jerry" --delete /local/webroot openbsd.nixcraft.in:/webroot

Task: Synchronize a local directory with a remote rsync server

$ rsync -r -a -v --delete rsync://rsync.nixcraft.in/cvs /home/cvs

Task: Mirror a directory between my "old" and "new" web server/ftp

You can mirror a directory between my "old" (my.old.server.com) and "new" web server with the command (assuming that ssh keys are set for password less authentication)
$ rsync -zavrR --delete --links --rsh="ssh -l vivek" my.old.server.com:/home/lighttpd /home/lighttpd

Read related previous articles

Other options - rdiff and rdiff-backup

There also exists a utility called rdiff, which uses the rsync algorithm to generate delta files Using rdiff. A utility called rdiff-backup has been created which is capable of maintaining a backup mirror of a file or directory over the network, on another server. rdiff-backup stores incremental rdiff deltas with the backup, with which it is possible to recreate any backup point. Next time I will write about these Utilities :)

rsync for Windows server/XP

Please note if you are using Windows, try any one of the program:

The Ten Minute Guide to diff and patch

2009-06-04T15:11:00.000-07:00

from: http://stephenjungels.com/jungels.net/articles/diff-patch-ten-minutes.html

The Ten Minute Guide to diff and patch

Situation one: you are trying to compile a package from source and you discover that somebody has already done the work for you of modifying it slightly to compile on your system. They have made their work available as a "patch", but you're not sure how to make use of it. The answer is that you apply the patch to the original source code with a command line tool called, appropriately, patch.

Situation two: you have downloaded the source code to an open source package and after an hour or so of minor edits, you manage to make it compile on your system. You would like to make your work available to other programmers, or to the authors of the package, without redistributing the entire modified package. Now you are in a situation where you need to create a patch of your own, and the tool you need is diff.

This is a quick guide to diff and patch which will help you in these situations by describing the tools as they are most commonly used. It tells you enough to get started right away. Later, you can learn the ins and outs of diff and patch at your leisure, using the man pages.

Applying patches with patch

To apply a patch to a single file, change to the directory where the file is located and call patch:

patch <>

These instructions assume the patch is distributed in unified format, which identifies the file the patch should be applied to. If not, you can specify the file on the command line:

patch foo.txt <>

Applying patches to entire directories (perhaps the more common case) is similar, but you have to be careful about setting a "p level". What this means is that, within patch files, the files to be patched are identified by path names which may be different now that the files are located on your computer rather than on the computer where the patch was created. The p level instructs patch to ignore parts of the path name so that it can identify the files correctly. Most often a p level of one will work, so you use:

patch -p1 <>

You should change to the top level source directory before running this command. If a patch level of one does not correctly identify any files to patch, inspect the patch file for file names. If you see a name like

/users/stephen/package/src/net/http.c

and you are working in a directory that contains net/http.c, use

patch -p5 <>

In general, count up one for each path separator (slash character) that you remove from the beginning of the path, until what's left is a path that exists in your working directory. The count you reach is the p level.

To remove a patch, use the -R flag, ie

patch -p5 -R <>

`Creating patches with diff`

Using diff is simple whether you are working with single files or entire source directories. To create a patch for a single file, use the form:

diff -u original.c new.c > original.patch

To create a patch for an entire source tree, make a copy of the tree:

cp -R original new

Make any changes required in the directory new/. Then create a patch with the following command:

diff -rupN original/ new/ > original.patch

That's all you need to get started with diff and patch. For more information use:

man diff man patch

`Translations`

This article has been translated and republished in the following languages:

Spanish: Guia de 10 minutos de diff y patch



Jabber Client with Flex & PureMVC
2009-05-16T13:53:00.000-07:00
From: http://www.actionscriptdeveloper.co.uk/puremvc-tutorial-flex-puremvc-jabber-and-xiff-3-conclusion-demo-downloads/
& http://actionscriptnotes.com/blog/2008/07/21/jabber-client-with-flex-puremvc/

PUREMVC TUTORIAL - FLEX, PUREMVC, JABBER AND XIFF 3: CONCLUSION, DEMO & DOWNLOADS
07.19.08 | 34 Comments

Introduction
Part 1 - Frameworks
Part 2 - Directory structure
Part 3 - Application and ApplicationFacade
Part 4 - Notifications, Commands & Use Cases
Part 5 - Model & Proxy
Part 6 - The Application View & Mediator
Part 7 - The Login View & Mediator
Part 8 - The Roster View & Mediator
Part 9 - The Chat View & Mediator
Conclusion, Demo & Downloads
Here is a working demo of the application!


libketama - a consistent hashing algo for memcache clients
2009-05-15T20:13:00.000-07:00
libketama - a consistent hashing algo for memcache clientsfrom: http://www.last.fm/user/RJ/journal/2007/04/10/rz_libketama_-_a_consistent_hashing_algo_for_memcache_clients

We wrote ketama to replace how our memcached clients mapped keys to servers. Previously, clients mapped keys->servers like this:

server = serverlist[hash(key)%serverlist.length];

This meant that whenever we added or removed servers from the pool, everything hashed to different servers, which effectively wiped the entire cache. We add (and sometimes remove) servers from the memcached pool often enough to warrant writing this - if your memcached pool never changes, you can probably stop reading now :)

Ketama is an implementation of a consistent hashing algorithm, meaning you can add or remove servers from the memcached pool without causing a complete remap of all keys.

Here's how it works:

* Take your list of servers (eg: 1.2.3.4:11211, 5.6.7.8:11211, 9.8.7.6:11211)
* Hash each server string to several (100-200) unsigned ints
* Conceptually, these numbers are placed on a circle called the continuum. (imagine a clock face that goes from 0 to 2^32)
* Each number links to the server it was hashed from, so servers appear at several points on the continuum, by each of the numbers they hashed to.
* To map a key->server, hash your key to a single unsigned int, and find the next biggest number on the continuum. The server linked to that number is the correct server for that key.
* If you hash your key to a value near 2^32 and there are no points on the continuum greater than your hash, return the first server in the continuum.

If you then add or remove a server from the list, only a small proportion of keys end up mapping to different servers.


The majority of the code is a C library (libketama) and a php4 extension that wraps it. I've also included a class from our Java client. (Java Collections makes it rather easy). We use a single-server memcache client wrapped with a native php class to make it multi-server capable, so we just replaced the hashing method with a ketama_find_server call. (should be easy enough to plug this into libmemcache if need be)

http://static.last.fm/rj/ketama.tar.bz2
http://static.last.fm/ketama/ketama-0.1.1.tar.bz2
svn://svn.audioscrobbler.net/misc/ketama/

We've been using this in production for all our php installs and java services at Last.fm for around 10 days now. We deployed it just in time to smooth over moving loads of webservers between datacenters.
Accepted Submissions
Programmers - Anyone who writes code




DRBD
2009-05-15T19:12:00.000-07:00
from: http://www.drbd.org/ and http://oss.linbit.com/drbd/
DRBD^® refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based raid-1.
In the illustration above, the two orange boxes represent two servers that form an HA cluster. The boxes contain the usual components of a Linux™ kernel: file system, buffer cache, disk scheduler, disk drivers, TCP/IP stack and network interface card (NIC) driver. The black arrows illustrate the flow of data between these components.
The orange arrows show the flow of data, as DRBD mirrors the data of a high availably service from the active node of the HA cluster to the standby node of the HA cluster.
→ Continue with What is HA
DRBD^® and the DRBD^® logo are trademarks or registered trademarks of LINBIT^® in Austria, the United States and other countries. 

DRBD is a block device which is designed to build high availability clusters.    This is done by mirroring a whole block device via (a dedicated) network. You    could see it as a network raid-1.        DRBD takes over the data, writes it to the local disk and sends it to the other    host. On the other host, it takes it to the disk there.        The other components needed are a cluster membership service, which is supposed    to be heartbeat, and some kind of application that works on top of a block    device.        Each device (DRBD provides more than one of these devices) has a state, which    can be 'primary' or 'secondary'. On the node with the primary device the    application is supposed to run and to access the device (/dev/drbdX; used to be    /dev/nbX). Every write is sent to the local 'lower level block device' and to    the node with the device in 'secondary' state. The secondary device simply    writes the data to its lower level block device. Reads are always carried out    locally.        If the primary node fails, heartbeat is switching the secondary device into    primary state and starts the application there. (If you are using it with a    non-journaling FS this involves running fsck)        If the failed node comes up again, it is a new secondary node and has to    synchronise its content to the primary. This, of course, will happen whithout    interruption of service in the background.         And, of course, we only will resynchronize those parts of the device that    actually have been changed. DRBD has always done intelligent resynchronization    when possible. Starting with the DBRD-0.7 series, you can define an "active    set" of a certain size. This makes it possible to have a total resync time of    1--3 min, regardless of device size (currently up to 4TB), even after a hard    crash of an active node.         The ChangeLogs can be found here:    http://git.drbd.org/?p=drbd-8.0.git;a=blob;f=ChangeLog;hb=HEAD    http://git.drbd.org/?p=drbd-8.2.git;a=blob;f=ChangeLog;hb=HEAD    http://git.drbd.org/        The DRBD Homepage is http://www.drbd.org/.



Nagios
2009-05-14T14:26:00.000-07:00
http://www.nagios.org/
About Nagios
Home > About
Get proactive.Save time, money, and your sanity.
Nagios is the industry standard in enterprise-class monitoring for good reason. It allows you to gain insight into your network and fix problems before customers know they even exist. It's stable, scalable, supported, and extensible. Most importantly, it works.
What does Nagios provide?
Comprehensive Network Monitoring

Windows
Linux/Unix
Routers, Switches, Firewalls
Printers
Services
Applications
Immediate Awareness and Insight

Receive immediate notifications of problems via email, pager and cellphone
Multi-user notification escalation capabilities
See detailed status information through the Nagios web interface
Problem Remediation

Acknowledge problems through the web interface
Automatically restart failed applications, services and hosts with event handlers
Proactive Planning

Schedule downtime for anticipated host, service, and network upgrades
Capacity planning capabilites through usage monitoring
Reporting Options

SLA availability reports
Alert and notification history reports
Trending reports through integration with Cacti and RRD-based addons
Multi-Tenant/Multi-User Capabilites

Multiple users can access the web interface
Each user can have their own unique, restricted view
Integration With Your Existing Applications

Trouble ticket systems
Wikis
Easily Extendable Architecture

Over 200 community addons are available to enhance Nagios
Stable, Reliable, and Respected Platform

10 years in development
Scales to monitor 100,000+ nodes
Failover protection capabilities
Winner of multiple awards
Constant media coverage
Huge Community

250,000+ users worldwide
Active mailing lists
Extensive community website network
Customizable Code

Open Source Software
Full access to source code
Released under the GPL license



Ganglia
2009-05-14T14:00:00.001-07:00
http://ganglia.info/

What is Ganglia?
Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRDtool for data storage and visualization. It uses carefully engineered data structures and algorithms to achieve very low per-node overheads and high concurrency. The implementation is robust, has been ported to an extensive set of operating systems and processor architectures, and is currently in use on thousands of clusters around the world. It has been used to link clusters across university campuses and around the world and can scale to handle clusters with 2000 nodes.
Ganglia is an open-source project that grew out of the University of California, Berkeley Millennium Project which was initially funded in large part by the National Partnership for Advanced Computational Infrastructure (NPACI) and National Science Foundation RI Award EIA-9802069. NPACI is funded by the National Science Foundation and strives to advance science by creating a ubiquitous, continuous, and pervasive national computational infrastructure: the Grid. Current support comes from Planet Lab: an open platform for developing, deploying, and accessing planetary-scale services.

Ganglia (http://ganglia.sourceforge.net/) takes a stab at answering all of these questions. Ganglia is comprised of several components: gmond, gmetad, and gmetric (pronounced gee-mon-dee and so forth), which perform the roles of sending data, aggregating data, and collecting data, respectively. gmond, the Ganglia-monitoring daemon, runs on each host we want to monitor. It collects core statistics like processor, memory, and network usage and deals with sending them to the spokesperson for the server cluster. The spokesperson for a cluster collects all of the data for the machine in a cluster (such as web servers or database servers) and then sends them on to the central logging host. gmetad is used for aggregating together cluster data at intermediate nodes. gmetric allows you to inject extra metric data into the Ganglia multicast channel, sending your statistics to the central repository.
One of the really useful abilities of the Ganglia system is how easy it is to extend the monitoring with custom metrics. We can write a program or script that runs periodically via cron to check on our statistic and then feed it into the Ganglia aggregation and recording framework by using the gmetric program. Our custom scripts can collect the data using whatever means necessary and then inject a single value (for each sampling run) along with the metric they're collecting for, its format, and units:
/usr/bin/gmetric -tuint32 -nmemcachced_hitratio -v92 -u% 

This example feeds data into Ganglia for a statistic called memcached_hitratio. If Ganglia hasn't seen this statistic before, it will create a new RRDTool database for storing it, using the data type specified by the -t flag (unsigned 32-bit integer in this example). The value 92 is then stored for this sampling period. Once two sampling periods have been completed, we can see a graph of the memcached_hitratio statistic on the Ganglia status page for that node. The final -uflag lets us tell Ganglia what units the statistic is being gathered in, which then displays on the output graphs, as shown in Figure 10-8.
Figure 10-8. Output graph with units




RRDtool
2009-05-14T13:58:00.001-07:00
http://oss.oetiker.ch/rrdtool/
What RRDtool does
RRDtool is the OpenSource industry standard, high performance data logging and graphing system for time series data. Use it to write your custom monitoring shell scripts or create whole applications using its Perl, Python, Ruby, TCL or PHP bindings.



Tobi Oetiker's Toolbox
2009-05-14T13:51:00.000-07:00
From: http://tobi.oetiker.ch/hp/
I work as an IT-Specialist for OETIKER+PARTNER AG a consultancy company based in Olten, Switzerland.
We use OpenSource Software extensively in all of our projects. So I made it a habit to open source code we write whenever possible. Below is a list of notable projects I have created or been involved with over the years.
Monitoring
RRDtool
A industry standard database for logging and graphing time-series data.
MRTG
The first widely used, long term network monitoring tool on the Internet. I created it in 1995 and still update it regularly as people send in patches.
SmokePing
Monitor latency on your network. By default it uses ping to track the latency on your network links.
du2rrd
Running a large storage server. This highly configurable tool lets you keep track of disk usage per directory. It comes with a Ajax front end and needs classic du for input.
Perl Win32::Monitoring Modules
A set of perl modules helping to write user interactivity monitoring systems.
Database Development
Gedafe the Generic Database Frontend
Developed with David Schweikert, this neat tool lets you write database centric web applications by simple setting up a postgresql database while following a few simple design and naming rules.
System Management
O+P Insights
A website where me and my colleagues document our insights connected with all things system management.
Windows System Management: Real Men Don't Click
A website about windows system management setup I helped setup while I was working for ETH Zurich.
OP SEPP
A software deployment system for large Unix systems. It deals with multi architecture issues, multiple versions of the same package installed in parallel.
ISG Toolchest
A system for keeping track of all the little scripts you write while sysadmining. Also on this page you find a large collection of tools developed at ETH Zurich using the system.
SamFS Support for Samba
This patch teaches Samba to handle SamFS offline files.
Miscellaneous
The not so Short Introduction to LaTeX2e
LaTeX is a system for writing documents with perfect looks while fully concentrating on the content. This document shows you how to do this.



TUX
2009-05-14T13:46:00.000-07:00
http://www.redhat.com/docs/manuals/tux/
Red Hat Content Accelerator Manuals ?
Red Hat Content Accelerator is a kernel-based, multi-threaded, extremely high performance HTTP server with the ability to serve both static and dynamic data. TUX is also an architecture for kernel-accelerated network services.
Red Hat Content Accelerator 2.2

Reference Manual
PDF (208 KB)    |    HTML Tarball (28 KB)
Red Hat Content Accelerator 2.1

Reference Manual
PDF (144 KB)   |   HTML Tarball (24 KB)
Red Hat Content Accelerator 2.0

Reference Manual
PDF (76 KB)   |   HTML Tarball (22 KB)



MogileFS is our open source distributed filesystem
2009-05-14T13:45:00.000-07:00
http://www.danga.com/mogilefs/
About MogileFS
MogileFS is our open source distributed filesystem. Its properties and features include:
Application level -- no special kernel modules required.
No single point of failure -- all three components of a MogileFS setup (storage nodes, trackers, and the tracker's database(s)) can be run on multiple machines, so there's no single point of failure. (you can run trackers on the same machines as storage nodes, too, so you don't need 4 machines...) A minimum of 2 machines is recommended.
Automatic file replication -- files, based on their "class", are automatically replicated between enough different storage nodes as to satisfy the minimum replica count as requested by their class. For instance, for a photo hosting site you can make original JPEGs have a minimum replica count of 3, but thumbnails and scaled versions only have a replica count of 1 or 2. If you lose the only copy of a thumbnail, the application can just rebuild it. In this way, MogileFS (without RAID) can save money on disks that would otherwise be storing multiple copies of data unnecessarily.
"Better than RAID" -- in a non-SAN RAID setup, the disks are redundant, but the host isn't. If you lose the entire machine, the files are inaccessible. MogileFS replicates the files between devices which are on different hosts, so files are always available.
Flat Namespace -- Files are identified by named keys in a flat, global namespace. You can create as many namespaces as you'd like, so multiple applications with potentially conflicting keys can run on the same MogileFS installation.
Shared-Nothing -- MogileFS doesn't depend on a pricey SAN with shared disks. Every machine maintains its own local disks.
No RAID required -- Local disks on MogileFS storage nodes can be in a RAID, or not. It's cheaper not to, as RAID doesn't buy you any safety that MogileFS doesn't already provide.
Local filesystem agnostic -- Local disks on MogileFS storage nodes can be formatted with your filesystem of choice (ext3, XFS, etc..). MogileFS does its own internal directory hashing so it doesn't hit filesystem limits such as "max files per directory" or "max directories per directory". Use what you're comfortable with.
MogileFS is not:
POSIX Compliant -- you don't run regular Unix applications or databases against MogileFS. It's meant for archiving write-once files and doing only sequential reads. (though you can modify a file by way of overwriting it with a new version) Notes:Yes, this means your application has to specifically use a MogileFS client library to store and retrieve files. The steps in general are 1) talk to a tracker about what you want to put or get, 2) read/write to one of the places it told you you could (it'll pick storage node(s) for you as part of its load balancing), using HTTP GET/PUT
We've prototyped a FUSE binding, so you could use MogileFS without application support, but it's not production-ready.
Completely portable ... yet -- we have some Linux-isms in our code, at least in the HTTP transport code. Our plan is to scrap that and make it portable, though.
Please see one of these Wiki pages for more information:
High-level overview of MogileFS
Obtaining MogileFS
Setup and Configuration
Using MogileFS from Perl
Need help? Have Questions?
Join the mailing list! We'd love to help you out.
MogileFS List Archives - For postings until June 2008
Check out the MogileFS Wiki.



squid-cache.org: Optimising Web Delivery
2009-05-14T13:44:00.000-07:00
http://www.squid-cache.org/
Squid: Optimising Web Delivery
Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing frequently-requested web pages. Squid has extensive access controls and makes a great server accelerator. It runs on most available operating systems, including Windows and is licensed under the GNU GPL.
Making the most of your Internet Connection
Squid is used by hundreds of Internet Providers world-wide to provide their users with the best possible web access. Squid optimises the data flow between client and server to improve performance and caches frequently-used content to save bandwidth. Squid can also route content requests to servers in a wide variety of ways to build cache server hierarchies which optimise network throughput.
Website Content Acceleration and Distribution
Thousands of web-sites around the Internet use Squid to drastically increase their content delivery. Squid can reduce your server load and improve delivery speeds to clients. Squid can also be used to deliver content from around the world - copying only the content being used, rather than inefficiently copying everything. Finally, Squid's advanced content routing configuration allows you to build content clusters to route and load balance requests via a variety of web servers.
 [The Squid systems] are currently running at a hit-rate of approximately 75%, effectively quadrupling the capacity of the Apache servers behind them. This is particularly noticeable when a large surge of traffic arrives directed to a particular page via a web link from another site, as the caching efficiency for that page will be nearly 100%.  - Wikemedia Deployment Information.
Want to learn more?
The Squid project provides a number of resources to assist users design, implement and support Squid installations. Please browse the Documentation and Support sections for more information.



The Spread Toolkit
2009-05-14T13:42:00.000-07:00
http://www.spread.org/  
Welcome to the Spread Toolkit!
Spread is an open source toolkit that provides a high performance messaging service that is resilient to faults across local and wide area networks. Spread functions as a unified message bus for distributed applications, and provides highly tuned application-level multicast, group communication, and point to point support. Spread services range from reliable messaging to fully ordered messages with delivery guarantees.
Spread can be used in many distributed applications that require high reliability, high performance, and robust communication among various subsets of members. The toolkit is designed to encapsulate the challenging aspects of asynchronous networks and enable the construction of reliable and scalable distributed applications.
Spread consists of a library that user applications are linked with, a binary daemon which runs on each computer that is part of the processor group, and various utility and demonstration programs.
Some of the services and benefits provided by Spread:
Reliable and scalable messaging and group communication.
A very powerful but simple API simplifies the construction of distributed architectures.
Easy to use, deploy and maintain.
Highly scalable from one local area network to complex wide area networks.
Supports thousands of groups with different sets of members.
Enables message reliability in the presence of machine failures, process crashes and recoveries, and network partitions and merges.
Provides a range of reliability, ordering and stability guarantees for messages.
Emphasis on robustness and high performance.
Completely distributed algorithms with no central point of failure.



Bazaar
2009-04-22T17:03:00.000-07:00
Introduction

Bazaar (or bzr) is a project of Canonical to develop an open source distributed version control system that is powerful, friendly, and scalable. ... It is used by Launchpad for managing revisions with packages. If you have never created a Launchpad project because bzr scared you, fear no more!

To get started, you should install the bzr application by running:
Code:
sudo apt-get install bzr
Bazaar relies on SSH keys to transfer files to your Launchpad account. If you don't have a SSH key, you can follow the next steps to get your SSH key on Launchpad. If you already have your SSH key on Launchpad, you can skip this section to "Using Bazaar".

Launchpad / SSH Key

To create your SSH key, open a terminal and run:
Code:
ssh-keygen -t dsa
Leave the first prompt (for the key location) at default, by pressing Enter, then enter a passphrase for your private SSH key. When complete, run:
Code:
cat ~/.ssh/id_dsa.pub
Copy your public key, from the previous command, then go to Launchpad to edit your SSH key:
https://launchpad.net/~username/+editsshkeys

Paste the key into "Add an SSH key" then click "Import Public Key". You should be all set now!


Using Bazaar

Suppose you found a project on Launchpad, that you wanted to help out with, or wanted to download their project source to tweak it to your own likings. I'll give my perlbot project as a quick example. If you go to:
https://code.launchpad.net/~drsmall/perlbot/trunk

You can view the trunk, and revisions for that project. If you wanted to download a copy of this trunk to your local system, you would issue this command:
Code:
bzr pull lp:perlbot
This would then download the source files of perlbot to ~/perlbot on your system. You can then execute, run, or send the revisions back to the trunk (with proper permission).

Ok. So let's say you want to start your own branch, where you can host your own edited version of some software, or it could be something you created by yourself that you want to be worked on by a team. Gather up all of the files you want to placed in your launchpad branch, and place them in one directory. cd to this directory, and then run:
Code:
bzr init
This makes the directory into a version branch. If you take the time to notice, there is now a hidden directory called .bzr in here. This is where all the revisions, and files are stored to be used by bzr. Now, add all of the files to the branch:
Code:
bzr add *
It is a good idea to get in the habit of running the next command for checking the difference between your last revision and the current one. You shouldn't have to do this on your first time around, though.
Code:
bzr diff
With this next step, we are committing our edits into the current revision. It is a good idea to label your revisions with meaningful comments.
Code:
bzr commit -m "Revision 1 Comment"
You may now upload your revision to your Launchpad branch. If the branch does not exist, it will be created. You can have multiple branches, so name them accordingly. This command may take a few minutes, but it is creating the branch, uploading your files, creating revisions and alot of other things.
Code:
bzr push lp:~user/projectname/branchname

Commands

Make directory a bzr branch:
Code:
bzr init
Download a branch:
Code:
bzr pull 
Update a branch:
Code:
bzr push 
Add files to branch:
Code:
bzr add 
Check the difference between revisions:
Code:
bzr diff
Commit the revision:
Code:
bzr commit -m "Revision Comment"
(These are just the basic commands. You can find the complete list of commands by running 'man bzr' in your terminal.)




Performance Timer ..
2009-04-19T19:14:00.001-07:00
From http://java.sun.com/developer/JDCTechTips/2002/tt0305.html

class MyTimer {         
private final long start;              
public MyTimer() {             
start = System.currentTimeMillis();         
}              
public long getElapsed() {             
return System.currentTimeMillis() - start;         
}     
}






Summary of techniques used in Dynamo and their advantages.
2009-04-18T15:41:00.001-07:00
From "Amazon Dynamo" paper ..

Table 1: Summary of techniques used in Dynamo and their advantages.
Problem Technique Advantage
Partitioning Consistent Hashing Incremental Scalability
High Availability for writes Vector clocks with reconciliation during reads Version size is decoupled from update rates.
Handling temporary failures Sloppy Quorum and hinted handoff Provides high availability and durability guarantee when some of the replicas are not available.
Recovering from permanent failures Anti-entropy using Merkle trees Synchronizes divergent replicas in the background.
Membership and failure detection Gossip-based membership protocol and failure detection. Preserves symmetry and avoids having a centralized registry for storing membership and node liveness information.




Object versioning
2009-04-18T15:14:00.000-07:00
From (http://ez.no/doc/ez_publish/technical_manual/3_9/concepts_and_basics/content_management/object_versioning)
Object versioning
eZ publish comes with a built in versioning system which is implemented at the object level. This mechanism makes it possible to have several versions of the contents (attributes) of an object. It basically provides a generic, out-of-the-box version control framework that can be used with any kind of content. The different versions are encapsulated by the object itself. The following illustration shows a more detailed example of the object structure seen from the outside world.
Example of a content object that consists of two versions.
Every time an object is edited, a new version of the object's contents will be created. It is always the new version that will be edited, the old version(s) remains untouched. This is how eZ publish keeps track of changes made by various users. An accidental or unwanted change can thus be undone by simply reverting an object back to the previous version.
Version limitations
Since every edit procedure results in the creation of a new version (unless the new version is discarded), the database can be quickly filled up by different versions of the same content. In order to prevent this problem, the versioning system can be limited to a certain number of versions per object. It is possible to assign different version limitations for different object types (different classes). The default limitation is 10, which means that every object can have a maximum number of 10 versions of its content. If the maximum count is reached, the oldest version will be automatically deleted and thus a free slot will be available for the new one. This is the default behavior. An alternative setting can be used to disallow the creation of new versions until an existing version is manually deleted by a user.
Version structure
A version consist of the following elements:
Version number
Creation time
Modification time
Creator
Status
Translations
Version number
Every version has a unique version number. This number is used by the system to organize and keep track of the different versions of an object. The version number is automatically increased for each version that is created inside an object.
Creation time
The creation time contains a timestamp pinpointing the exact date and time when the version was initially created. This information is set by the system and will remain the same regardless of what happens to the version.
Modification time
The modification time contains a timestamp revealing the exact date and time when the version was last modified. This information is set by the system every time the version is stored and when the version is finally published. When a version is published, the modification time of the object itself will be updated (it will simply be set to the same value as modification time of the version that was published).
Creator
The version's creator contains a reference to the user that created the version. Although a content object can only belong to a single user (revealed by the "Owner" field), each version may belong different users. The creator reference is set by the system when the version is created. It can not be manipulated and will not change even if the user who created the version is removed from the system.
Status
The state of a version is determined by its status. There are five possibilities:
Draft (0)
Published (1)
Pending (2)
Archived (3)
Rejected (4)
In eZ Publish versions 3.8 and later, there is an additional possibility: if a version of a content object is created but not modified (for example, if someone clicked an "Add comments" button but didn't actually post anything), the status of the version will be "Internal draft (5)". In the administration interface, status "5" drafts are called "untouched drafts". From 3.9, you can set the number of days, hours, minutes and seconds before an internal draft is considered old and removed by the "internal_drafts_cleanup.php" cronjob script. Another cronjob script called "old_drafts_cleanup.php" can be configured to remove status "0" drafts that have been in the system for a specified period of time.
A newly created version is a draft. This status will remain until that version becomespublished. Although an object can have many versions, there can only be one published version (the others are usually drafts and archived versions). The published version can be considered as the "current" version and it is the one that is accessed when the object is viewed. A published version can not become a draft. However, it will become archived as soon as another version is published. The following illustration shows how the versioning system actually works.
Overview of the object states.
The illustration above shows the most common states of a content object. When a new object is created (step 1), eZ publish will also create a new draft version. Because the object has not been published yet, its status is set to draft and the current version is unknown. Storing the draft (steps 2a and 2b) will not change the state of the object. The only thing that will happen is that the contents of the draft will be stored in version 1. If the draft (which is the only existing version) is discarded, the object is completely removed from the system (step 2c). When the draft is published (step 2), both the draft and the object's states will be set to published. In addition, the current version will be set to 1, which reveals the currently published version of the object. When published, the contents of the object can be viewed by others. A published object can be removed/deleted from the system (step 3a). When removed, the object's state will be set to "Archived" and thus it will be in the trash. The object can be recovered from the trash to its previous state. Among other things, this involves the status field being set to "Published" again. When a published object is edited (step 4), the current version (version 1 in this case) will remain untouched and a completely new version will be created. The contents of the new version (version 2 in this case) will be a copy of the contents of the current version. Again, storing the draft (steps 4b and 4c) will not change the state of the object. If the draft is discarded (step 4a), it will be completely removed from the system and thus the object will be in the exact same state as it was in before it was edited. If the newly created and edited draft is published, it will become the current version of the object and thus the previous version (version 1 in this case) will be set to "Archived". Step 5a illustrates what would happen if the object (now with two versions) would be removed.
The pending and the rejected states are used by the collaboration system. When a version is waiting to be approved by an editor, the status is set to pending. If the version is approved, it will be automatically published and thus the status will be set to published. On the other hand, if a pending version is rejected by the editor, the status will be set to rejected.
A version can only be edited if it is a draft and it can only be edited by the same user who initially created it. In addition, rejected versions can also be edited. When a rejected version is edited, it will become a draft. Published and archived versions can not be edited. However, it is possible to make copies of them. When a published or an archived version is copied, the status of the copy is set to draft and thus it becomes editable. When/if the new draft is published, the system automatically sets the status of the previously published version to archived and the new draft will become the published version.
Translations
The actual contents of a version is stored inside different translations. A translation is a representation of the information in a specific language. In other words, the translation layer allows a version of the object's actual contents to exist in different languages. A version always has at least one translation of the content (which represents the data in the default/standard language).


Consistent Hashing
2009-04-18T15:11:00.000-07:00
FRom Tom White's Blog (http://www.lexemetech.com/2007/11/consistent-hashing.html) 

Consistent Hashing
I've bumped into consistent hashing a couple of times lately. The paper that introduced the idea (Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web by David Karger et al) appeared ten years ago, although recently it seems the idea has quietly been finding its way into more and more services, from Amazon'sDynamo to memcached (courtesy of Last.fm). So what is consistent hashing and why should you care?

The need for consistent hashing arose from limitations experienced while running collections of caching machines - web caches, for example. If you have a collection of n cache machines then a common way of load balancing across them is to put object o in cache machine number hash(o) mod n. This works well until you add or remove cache machines (for whatever reason), for then nchanges and every object is hashed to a new location. This can be catastrophic since the originating content servers are swamped with requests from the cache machines. It's as if the cache suddenly disappeared. Which it has, in a sense. (This is why you should care - consistent hashing is needed to avoid swamping your servers!)

It would be nice if, when a cache machine was added, it took its fair share of objects from all the other cache machines. Equally, when a cache machine was removed, it would be nice if its objects were shared between the remaining machines. This is exactly what consistent hashing does - consistently maps objects to the same cache machine, as far as is possible, at least.

The basic idea behind the consistent hashing algorithm is to hash both objects and caches using the same hash function. The reason to do this is to map the cache to an interval, which will contain a number of object hashes. If the cache is removed then its interval is taken over by a cache with an adjacent interval. All the other caches remain unchanged.
Demonstration
Let's look at this in more detail. The hash function actually maps objects and caches to a number range. This should be familiar to every Java programmer - the hashCode method on Object returns an int, which lies in the range -2³¹ to 2³¹-1. Imagine mapping this range into a circle so the values wrap around. Here's a picture of the circle with a number of objects (1, 2, 3, 4) and caches (A, B, C) marked at the points that they hash to (based on a diagram from Web Caching with Consistent Hashing by David Karger et al):



To find which cache an object goes in, we move clockwise round the circle until we find a cache point. So in the diagram above, we see object 1 and 4 belong in cache A, object 2 belongs in cache B and object 3 belongs in cache C. Consider what happens if cache C is removed: object 3 now belongs in cache A, and all the other object mappings are unchanged. If then another cache D is added in the position marked it will take objects 3 and 4, leaving only object 1 belonging to A.



This works well, except the size of the intervals assigned to each cache is pretty hit and miss. Since it is essentially random it is possible to have a very non-uniform distribution of objects between caches. The solution to this problem is to introduce the idea of "virtual nodes", which are replicas of cache points in the circle. So whenever we add a cache we create a number of points in the circle for it.

You can see the effect of this in the following plot which I produced by simulating storing 10,000 objects in 10 caches using the code described below. On the x-axis is the number of replicas of cache points (with a logarithmic scale). When it is small, we see that the distribution of objects across caches is unbalanced, since the standard deviation as a percentage of the mean number of objects per cache (on the y-axis, also logarithmic) is high. As the number of replicas increases the distribution of objects becomes more balanced. This experiment shows that a figure of one or two hundred replicas achieves an acceptable balance (a standard deviation that is roughly between 5% and 10% of the mean).


Implementation
For completeness here is a simple implementation in Java. In order for consistent hashing to be effective it is important to have a hash function thatmixes well. Most implementations of Object's hashCode do not mix well - for example, they typically produce a restricted number of small integer values - so we have a HashFunction interface to allow a custom hash function to be used. MD5 hashes are recommended here.


import java.util.Collection;
import java.util.SortedMap;
import java.util.TreeMap;

public class ConsistentHash {

 private final HashFunction hashFunction;
 private final int numberOfReplicas;
 private final SortedMap circle =
   new TreeMap();

 public ConsistentHash(HashFunction hashFunction,
   int numberOfReplicas, Collection nodes) {

   this.hashFunction = hashFunction;
   this.numberOfReplicas = numberOfReplicas;

   for (T node : nodes) {
     add(node);
   }
 }

 public void add(T node) {
   for (int i = 0; i < numberOfReplicas; i++) {
     circle.put(hashFunction.hash(node.toString() + i),
       node);
   }
 }

 public void remove(T node) {
   for (int i = 0; i < numberOfReplicas; i++) {
     circle.remove(hashFunction.hash(node.toString() + i));
   }
 }

 public T get(Object key) {
   if (circle.isEmpty()) {
     return null;
   }
   int hash = hashFunction.hash(key);
   if (!circle.containsKey(hash)) {
     SortedMap tailMap =
       circle.tailMap(hash);
     hash = tailMap.isEmpty() ?
            circle.firstKey() : tailMap.firstKey();
   }
   return circle.get(hash);
 }

}


The circle is represented as a sorted map of integers, which represent the hash values, to caches (of type T here).
When a ConsistentHash object is created each node is added to the circle map a number of times (controlled by numberOfReplicas). The location of each replica is chosen by hashing the node's name along with a numerical suffix, and the node is stored at each of these points in the map.

To find a node for an object (the get method), the hash value of the object is used to look in the map. Most of the time there will not be a node stored at this hash value (since the hash value space is typically much larger than the number of nodes, even with replicas), so the next node is found by looking for the first key in the tail map. If the tail map is empty then we wrap around the circle by getting the first key in the circle.
Usage
So how can you use consistent hashing? You are most likely to meet it in a library, rather than having to code it yourself. For example, as mentioned above, memcached, a distributed memory object caching system, now has clients that support consistent hashing. Last.fm's ketama by Richard Jones was the first, and there is now a Java implementation by Dustin Sallings (which inspired my simplified demonstration implementation above). It is interesting to note that it is only the client that needs to implement the consistent hashing algorithm - the memcached server is unchanged. Other systems that employ consistent hashing include Chord, which is a distributed hash table implementation, and Amazon's Dynamo, which is a key-value store (not available outside Amazon).
Posted by Tom White at 17:26
Labels: Distributed Systems, Hashing

Problem	Technique	Advantage
Partitioning	Consistent Hashing	Incremental Scalability
High Availability for writes	Vector clocks with reconciliation during reads	Version size is decoupled from update rates.
Handling temporary failures	Sloppy Quorum and hinted handoff	Provides high availability and durability guarantee when some of the replicas are not available.
Recovering from permanent failures	Anti-entropy using Merkle trees	Synchronizes divergent replicas in the background.
Membership and failure detection	Gossip-based membership protocol and failure detection.	Preserves symmetry and avoids having a centralized registry for storing membership and node liveness information.