lets share the knowledge

Installing MySQL-python in Snow Leopard

2010-08-10T22:22:00.000-07:00

Here are the steps for installing MySQL-python in Snow leopard:

Untar downloaded Mysql-Python

cd [Mysql-Python path]
export PATH=/usr/local/mysql:$PATH
sudo ARCHFLAGS='-arch x86_64' python setup.py build
sudo ARCHFLAGS='-arch x86_64' python setup.py install

FYI: I have tested the installation in Python2.6.5

Enjoy

Installing Scipy, Numpy and Matplotlib in Snow Leopard

2010-08-03T00:10:00.000-07:00

I had trouble installing Scipy in my Macbook(SnowLeopard). After googling, I endup into the site http://bit.ly/baCJUY .It solved my problem. Here is how i did it in simple way.

Actual Source: http://bit.ly/bGSgBa

To install Matplotlib download matplotlib from here. Install the downloaded package and enjoy python scientific toolkits :) .

Experssion, Term and Item

2010-07-09T03:34:00.000-07:00

Every time i study about Parse tree (Compiler deisgn/Theory of compuatation), I get confused what expression, term and item actually means. So i am blogging for my reference and hope it will be useful to others too.

An expression is a sum (with + or -) of terms.
A term is a product (with * or /) of items.
An item is either a number, or a variable name, or an expression enclosed in parentheses.

Generalized Producer-Consumer solution using Semaphore

2008-12-31T22:59:00.000-08:00

The producer-consumer problem is a classical example of a multi-process synchronization problem. It illustrates the need for synchronization in systems where many processes share a resource. The problem describes two processes, the producer and the consumer, who share a common, fixed-size buffer. The producer's job is to generate a piece of data, put it into the buffer and start again. At the same time the consumer is consuming the data (i.e. removing it from the buffer) one piece at a time. The problem is to make sure that the producer won't try to add data into the buffer if it's full and that the consumer won't try to remove data from an empty buffer.

Here, the problem is generalized to have multiple producers and consumers. The solution to these type of problem must assure three constraints

Mutual exclusion
Free from Deadlock
Free from Starvation

Pseudo-code for the solution to multiple producer- consumer problem is given below.

Semaphore fullBuffer = 0; // Initially, no item in buffer
Semaphore emptyBuffers = numBuffers; // Initially, num empty buffer
Semaphore mutex = 1;      // No thread updating the buffer
Producer(item) {
  emptyBuffers.P(); // Wait until space
  mutex.P();        // Wait until buffer free
  Enqueue(item);
  mutex.V();
  fullBuffers.V();  // Tell consumers there is data in buffer
}
Consumer() {
  fullBuffers.P();
  mutex.P();
  item = Dequeue();
  mutex.V();
  emptyBuffers.V();
  return item;
}

The mutex lock is used to ensure that only one process will be accessing the buffer at a time,this protect the Queue data structure (a shared buffer) from being damaged. If the mutex lock is not used then the concurrent process can manipulate the head and tail pointer associated with the buffer in undesired way. The emptyBuffers and fullBuffer are the semaphore specifying the no of the empty buffers and fullbuffer available respectively.

The code implementing the given pseudo-code is shown below:

/*
 * main.c
 *
 *  Created on: Dec 31, 2008
 *      Author: Suvash Sedhain
 *  Multiple producer and Multiple consumer solution using Semaphores
 *      Note: semaphore.P() is actually sem_wait() and semaphore.V() is sem_post()
 *      as defined in semaphore.h
 *
 *      Ref:
 *      - Operating System Concepts : Abraham Silberschatz, Peter Baer Galvin, Greg Gagne
 *      - Operating Systems, William Stallings
 *      - Professor John D. Kubiatowicz lectures notes, University of California, Berkeley
 *        website: http://inst.eecs.berkeley.edu/~cs162
 *
 */

#include <stdlib.h>
#include <pthread.h>
#include <semaphore.h>
#include <unistd.h>
#include <stdio.h>
#define BUFFER_SIZE 5
const int MAX_PRODUCER = 20;
const int MAX_CONSUMER = 20;

// information to maintain the circular queue data structure
int head = 0;
int tail = 0;

//shared buffer
int buffer[BUFFER_SIZE];
// mutex lock for buffer
pthread_mutex_t mutex;  
//semaphores for specifying the empty and full
sem_t emptyBuffers;
sem_t fullBuffer;

//initialze the locks
void initialize_locks()
{
    pthread_mutex_init(&mutex,NULL);
    sem_init(&emptyBuffers,0,5);
    sem_init(&fullBuffer,0,0);
}

// Produce random value to shared buffer
void *producer(void *param)
{
  int item;
  while(1)
  {
      item = rand();
      sem_wait(&emptyBuffers);
      pthread_mutex_lock(&mutex);
      buffer[tail] = item ;
      tail = (tail+1) % BUFFER_SIZE;
      printf ("producer: inserted %d \n", item);
      fflush (stdout);
      pthread_mutex_unlock(&mutex);
      sem_post(&fullBuffer);

  }
  printf ("producer quiting\n");  fflush (stdout);
}

//consume values from the shared buffer
void *consumer(void *param)
{
   int item;
   while (1)
   {
       sem_wait(&fullBuffer);
       pthread_mutex_lock(&mutex);
       item = buffer[head];
       head = ( head + 1) % BUFFER_SIZE;
       printf ("consumer: removed %d \n", item);
       fflush (stdout);
       pthread_mutex_unlock(&mutex);
       sem_post(&emptyBuffers);
   }
}

int main( int argc, char *argv[])
{
   int i, sleep_time = 100, no_of_consumer_threads = 10, no_of_producer_threads = 2;
   pthread_t producer_id[MAX_PRODUCER], consumer_id[MAX_CONSUMER];
   initialize_locks();
   // create producer threads
   for(i = 0; i < no_of_producer_threads; i++)
   {
       if (pthread_create(&producer_id[i],NULL,producer,NULL) != 0)
       {
           fprintf (stderr, "Unable to create producer thread\n");
           return -1;
       }
   }
   // create consumer threads
   for(i = 0; i < no_of_consumer_threads; i++)
   {
       if (pthread_create(&consumer_id[i],NULL,consumer,NULL) != 0)
       {
           fprintf (stderr, "Unable to create consumer thread\n");
       }
   }
   sleep(sleep_time);
   return 0;
}

Simple RSS feed reader in python

2008-09-12T04:07:00.000-07:00

Writing a simple RSS feed reader is pretty simple in python.Here i have presented a simple snippet that prints the "title" element of the RSS feed


# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.


from urllib2 import urlopen
from xml.sax import make_parser, ContentHandler
import sys

__author__ = "Suvash Sedhain"
__date__ = "Fri Sep 12 16:46:44 NPT 2008"

class RSSHandler (ContentHandler):

  def __init__(self):
      ContentHandler.__init__(self)
      self.__inItem = False
      self.__inTitle = False

  def characters(self,data):
      if self.__inTitle:
          sys.stdout.write(data)
       
  def startElement(self, tag, attrs):
      if tag == "item":
          self.__inItem = True
      if tag == "title" and self.__inItem:
          self.__inTitle = True

  def endElement(self, tag):
      if tag == "title" and self.__inTitle:
          sys.stdout.write("\n")
          self.__inTitle = False
      if tag == "item":
          self.__inItem = False

def listFeedTitle(url):
  infile = urlopen(url)
  parser = make_parser()
  parser.setContentHandler(RSSHandler())
  parser.parse(infile)

#extract all the python related links from dzone
listFeedTitle("http://www.dzone.com/links/feed/search/python/rss.xml")

Code Explanation:

The class RSSHandler is derived from the class ContentHandler, which is a interface that provides functionality of Parsing XML. we override methods characters, startElement and endElement to customize the parser as per our need.

The method invocation can be summarized as:

"characters" method is called when stream of text is encountered

"startElement" method is called when any starting tag is encountered

"endElement" method is called when any ending tag is encountered

The function make_parser is a factory function that returns the generic XML parser.

IMAGE VERIFICATION WITH PHP

2008-05-25T09:53:00.002-07:00

Image verification has become an integral part in any online registration process. It is important to know whether the other side that is filling the form is really a human or any robot or automatic scripts so, its time to look at how it really works.

To achieve this with PHP, you need to have the GD library installed. The whole procedure is quite simple. You generate a random number and then stamp it over a picture and there you go, you have your own verification number on a nice picture.

The coding is also quite simple as the algorithm mentioned above. Enough of words and let’s take a look at the coding to make the things go clear.

//verification.php

$alphanum = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";

// generate the5 digit verification code

$rand = substr(str_shuffle($alphanum), 0, 5);

//create image from a given jpeg to write our code

$image = imagecreatefromjpeg("background.jpg");

//allocate the text color of the code

$textColor = imagecolorallocate($image, 0, 0, 0);

// write the code on the background image

imagestring ($image, 5, 5, 8, $rand, $textColor);

// send the content type header so the image is //displayed properly

header(‘Content-type: image/jpeg’);

// send the image to the browser

imagejpeg($image);

// destroy the image to free up the memory

imagedestroy($image);

Save the above code as verification.php and preview it in your browser and there you have it.

How it works

In the above program, we created a string of alphanumeric characters. Then the sting was shuffled using str_shuffle(string) function. We then extracted the first five characters of the shuffled string using the substr(string,starting_position,no_of_chars). The extracted characters were then stored as our random code. We then created a image from the given image and wrote our code on the image using the imagestring(image,font,x_pos,y_pos,string,string_color). (Refer the imagettftext() function for better result).

And finally the image was fed to the browser using the imagejpeg(image) function for jpeg image. For a complete reference on the functions used in above codes, you can log on to the official site of php,

http://www.php.net/ and get your copy of php manual for free.

Now you have your verification code, but what about using it to verify the users. They say life is full of options and so does php offers a variety of options to pass a variable between the web pages. You can pass the random code generated to any other page either using sessions or cookies variables, using the form submission method or the rather insecure URL method. Let’s try it using session variables. For this you need to have some modifications on your verification.php file or whatever you named it. first of all initialize a session and then assign the random number to a session variable.

//mod_verification.php

Session_start();

$alphanum = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";

// generate the5 digit verification code

$rand = substr(str_shuffle($alphanum), 0, 5);

//assign session variable the verification code

$_SESSION[‘variable’]=$rand;

//create image from a given jpeg to write our code

$image = imagecreatefromjpeg("background.jpg");

//allocate the text color of the code

$textColor = imagecolorallocate($image, 0, 0, 0);

// write the code on the background image

imagestring ($image, 5, 5, 8, $rand, $textColor);

// send the content type header so the image is //displayed properly

header(‘Content-type: image/jpeg’);

//save the final image as image.jpg

imagejpeg($image,”image.jpg”);

echo “<img src=\”image.jpg\”>”;;

echo "Type the code you see in the image in the box below. (case sensitive)";

<form action=check.php method=post>

<input name=code type=text > //place to type the code

<input type=submit> //submit button

</form>

With that done, we must now create a new file and name it check.php and see if it works.

//check.php

Session_start();

//retrieve the posted variable via forn

$typed=$_POST[‘code’];

//retrieve the session variable

$var=$_SESSION[‘variable’];

echo “The verification code was $var and you typed $typed”;

if($var==$typed)

{

echo “you are verified”;

}

There you are, with your own image verification tool.

5 Reasons why beginner hates web-frame work?

2007-11-07T08:37:00.000-08:00

Working on spring framework and Cake PHP framework make me realize why beginners hate web-framework. I had heard a lot of eulogy about spring so I was eager to use the spring framework. Initially, I found the framework enigmatic. Aspect Oriented programming, Dependency injection, Bean creation, Validation and Hibernate everything was completely new for me. At first, to create a simple “hello world” was also daunting task for me. Fortunately, my team leader was expert in all those things and he was so helpful. So, we keep on learning all those stuffs during the course of the project. Now, I am working with energetic programmers on CakePHP framework. As I am familiar with MVC, Cake PHP is yummy for me. Although my teammates were familiar with PHP but MVC was new for them. So, at the beginning they were arguing “Why we are using any framework?” And they were not convinced about the productivity of the framework too.

As per my personal experience, following are the 5 reasons why beginner hates web-framework

1) Comparing everything with hello world.

I have found in many pages which compares languages by showing in how many line they can display “hello world” and something simple things like this. I have found the good impression of such pages on the beginner. So, when I say you need MODEL, VIEW and CONTROLLER to create a simple app to a beginner, they simply think that frame-work adds overhead because they have to create three pages to create a simple hello world.

2) Coding Convention

While working with the Cake PHP naming convention of the model view and controller have annoyed my team members and it reminds me the days when novice to the spring frame work.

3) Functionality provided by the framework

Beginners prefer to use their own code even though the framework provides it. Once,

One of my teammates argued with me regarding the CakePHP validation. At first he used his own JavaScript based validation. But After working with numbers of pages he realized that it’s effective to use the CakePHP validation and was convinced to use that.

4) MVC is overhead for beginner.

Actually MVC systematize the development process itself. Work of the programmer and designer can be easily separated. So, coding and designing can go hand on hand. Its difficult for the beginner to visualize the program flow with MVC. However, it is difficult to convince the beginner about MVC pattern until and unless they get involved in any large projects.

5) Framework is difficult to understand.

For any one who is new to any framework, s/he has problem on even in simple things. Only difference is that for the people who have already worked in other frameworks can compare the features and quickly learn the things. But, for the novice it is too difficult to grasp a lot of new terminology at a time. So, all these terminology intimidate the novice.

Object Oriented Programming with PHP

2007-10-07T22:16:00.000-07:00

The term object-oriented involves thinking about processes as entities; in other words, the way we think about day-to-day objects. Object oriented programming is widespread today, and many universities teach object-oriented programming in beginning programming classes. Currently, Java and C++ are the most prevalent languages used for object-oriented programming. Object-oriented programming is not just a matter of using different syntax. It’s a different way of analyzing programming problems. In object-oriented programming, the elements of a program are objects. The objects represent the elements of the problem your program is meant to solve. Object-oriented programming developed new concepts and new terminology to represent those concepts.

But what about OOP in web scripts like PHP? Web scripts typically execute quickly and then go away. So you may think do we really need OOP concept in the web scripting? PHP wasn’t developed as an object oriented language. PHP began life as a simple set of scripts. However, PHP couldn’t be left untouched by this growing global phenomenon of OOP and its numerous advantages forced it to be reckoned by the language. Over the course of its life, PHP has evolved, more and more object-oriented features. First, you could define classes, but there were no constructors. Then, constructors appeared, but there were no destructors. Slowly but surely, as more people began to push the limits of PHP's syntax additional features were added to satisfy the demand. Object oriented programming became possible with PHP 4. With the introduction of PHP 5, the PHP developers have really beefed up the object-oriented features of PHP, resulting in both more speed and added features. Much of this improvement is invisible — changes introduced with the Zend 2 engine that powers PHP 5, that make scripts using objects run much faster and more efficiently than they did in PHP 4. PHP typically has a less thoroughgoing implementation of OOP than languages like c++, java, etc. There are still some concepts missing as function overloading and multiple inheritances.

OOP concept has really changed the way we used to program in PHP. Because of OOP, the code redundancy has been greatly reduced and using this technique we are able to make simple programs on the fly. This concept has helped to develop and maintain large scale projects easily. Features like inheritance, encapsulation and abstractions have helped speed up the development process of products using PHP. As of PHP5, it supports single inheritance, constructors and destructors, encapsulations, static functions, object interface, etc. So, OOP is certainly developing as an integral part of PHP programming.

Pdf version of the Django book

2007-10-01T12:09:00.000-07:00

Pdf version of online Django book

If you prefer the pdf format to html then download the pdf format of the django book here:
Djangobook.

The book is actually compiled from the content available in the Djangobook.com

What is Django and why I love it?

Django is the High level python Web framework for the rapid web development. Django is based on the MVC ie Model View Controller (actually the developer of the Django calls it MTV ie Model Template View).

While Ruby on Rails(ROR) was the subject of the discussion among the web developers, Django was developed to make the web development easy with the power of the python. The framework is highly inspired by the features of the Ruby on Rails, but it should be mentioned that it’s not the port of the ROR for python. Django has its own descent architecture and design. As per my experience Django is more Productive and scalable than the RoR. So its true for me too that if ruby is on rails then the Python is on jet. An excellent comparison between the Django and Rails in terms of the performance is available at wiki.rubyonrails.org/rails/pages/Framework+Performance. The best part of the Django is its well maintained site www.Djangoproject.com. This site provides lots of the information about the web development with Django.

An up-to-date list of dozens of sites that use Django is located at code.Djangoproject.com/wiki/DjangoPoweredSites.

Visit the link below for the short but excellent introduction of the Django http://www.ibm.com/developerworks/linux/library/l-Django/?ca=dgr-btw01DjangoAndPython

If you find this stuff useful then don’t forget to digg this post so that it is easily available for others too.

Ajax :Defacto of web 2.0

2007-09-28T10:34:00.001-07:00

AJAX

What is Ajax?

If you are thinking about getting information about Ajax FC or the Ajax city, then you've just hit the wrong page. We are here to talk about the term AJAX which stands for Asynchronous JavaScript & XML. The term Ajax was coined in 2005 by James Garrett and is used to describe a combination of technologies of JavaScript, XML, CSS, XHTML and DOM and the all important XHR (XMLHTTPRequest) object. While the name AJAX has only recently been coined, the underlying technologies (JavaScript, XML,

SOAP, and so on) have been maturing for years .XHR was created by Microsoft in IE5 in March of 1999, as an ActiveX Object and it only worked in IE.XHR was then adopted by Mozilla 1.0 and Safari 1.2.And a new generation of web application was born. Its core technique is centered on communication to the server without a page refresh .It is also known as XMLHTTP technique.

Ajax is all about improving the user experience. Since the beginning of then Web, and until recently, the users have nothing to do but fill some forms, click some links and then wait for reloading the whole page. It was not only becoming monotonous but tedious as well. This is what could be called the Web 1.0 model. Google Maps (http://maps.google.com) really ignited the Ajax fire. Google, through the clever use of XHR callbacks provided the first in-page scrollable map. There was no waiting around for a bunch of ads to refresh. It really opened the new pattern of web UI which we prefer to call web 2.0.Real time Live searches, Web based chatting, Dragging and dropping ,instant feedback etc, have really established AJAX as a tool for making richer application. Google gmail, Google suggest, Google maps, Flickr etc. are the examples of Ajax application that have sparked people's imagination.

The four defining principles of Ajax

1. The browser hosts an application, not content

Classically when a user logs into a site, a session is initialized and several serverside objects are created and pages are served to the browser. The home page is dished up to the browser, in a stream of HTML markup that mixes together standard boilerplate presentation and user-specific data. The browser acted as a dumb terminal and flushed whatever was sent to it. But Ajax moves some application logic to the browser. When the user logs in. a more complex document is delivered to the browser, a large proportion of

which is JavaScript. This document will stay with the user throughout the session, although it will probably alter its appearance considerably while the user is interacting with it. It knows how to respond to user input and is able to decide whether to handle the user input itself or to pass a request on to the web server.

2. The server delivers data, not content

As we noted, the classic web app serves up the same mixture of boilerplate, content, and data at every step. An Ajax based application can behave smartly by sending asynchronous requests to the server so that the server can only send back the relevant data while all other features of the page layout are there already.

3 .User interactions with the application can be fluid and continuous

In classic web applications, the only way to contact the server was either to click a hyperlink or to submit a form and wait for the server to respond. But Ajax completely changes this style of communicating the server. With Ajax technology, we can contact the server without interfering the user's flow. Interactions can be made with the server on the basis of mouse drag or movement or on every keystroke as on Google suggest.

4. This is real coding and requires discipline

An Ajax technology doesn't use JavaScript just to get fancy decorations or designs in the site. It really has to do some serious works. In Ajax, the code that you deliver when users launch the application must run until they close it, without breaking.

What can you do with Ajax

The technology for Ajax has been around since 1998, and a handful of applications (such as Microsoft’s Outlook Web Access) have already put it to use. But Ajax didn’t really catch on until early 2005, when a couple of high-profile Web applications (such as Google Suggest and Google Maps) put it to work, and Jesse James Garrett wrote his article coining the term Ajax and so putting everything under one roof. Google has been the pioneer in developing Ajax applications and has done more than nyone to raise the profile of Ajax applications. Some of the applications of Ajax are:

a. Auto complete

This is the Ajax feature that reacts to keystrokes and tries to make intelligent

guesses on what the user is going to type and then suggest it. For example

suggestions in gmail from the address book on typing email addresses.

b. Searching in real time with live searches

This feature gives result to your queries instantly as you enter them.

Googlesuggest is an example of this feature. You can view an example by logging onto http://www.google.com/webhp?complete=1

c. Chatting

Ajax has enabled us to have a web based chat service. http://iloveim.com is an example of a site based on this technology.

d. Dragging and dropping

As you might have seen in many e-commerce sites, you can drag and drop items to your cart which you prefer to buy. This has been made possible by Ajax. Ajax basically deals with the asynchronous handling of events. By using Ajax a rich application can be achieved without any additional plug-ins whatsoever to the browser as Ajax is supported by all major browsers IE, Firefox / Netscape / Mozilla with slight variations. So, Ajax has been a major tool in making the web UI more attractive and user friendly and is definitely here to stay.

WEB CRAWLER: How It Works?

2007-09-27T10:56:00.000-07:00

In the internet there are hundreds of millions of pages providing the information on an amazing variety of topics. So, retrieving the useful information from the web is really a daunting task. How to obtain the required information from those millions of pages? Of course internet search engine site like google.com, yahoo.com, live.com etc are one and only option. These are special sites on the Web that are designed to help people find information stored on other sites. At the first glance it seems like a magic .These site understand what we intended to search. Really amazing, Search engine can be Crawler-Based Search Engines and Human-powered directories. Crawler-based search engines create their listings automatically. It automatically tracks any changes on the web pages where as a human-powered directory depends on humans for its listings. So, in the rapidly growing web, Crawler-Based Search Engine is better.

Crawler-based search engines have three major steps.

a) Crawling

b) Indexing

c) Searching

Crawling:

Web crawlers are programs that locate and gather information on the web. They recursively follow hyperlinks present in known document to find other document. The usual starting points are lists of heavily used servers and very popular pages. In this way, the spider system quickly begins to travel, spreading out across the most widely used portions of the Web. The spider visits to the site on a regular basis, such as every month or two, to look for changes.

Indexing:

An index helps to find the information as quickly as possible. The index is also known as catalog. If a web page changes, then index is updated with new information. Indexing basically consists of two steps:

a) Parsing

b) Hashing

a. Parsing:

Parser extracts the link for further crawling. It also removes tag, JavaScript, comments etc. from the web pages and convert the html document to plain text. For the automated analysis of the text Regular expressions are extensively used. Parser which is designed to run on the entire Web must handle a huge array of possible errors.

b. Hashing:

After each document is parsed, it is encoded into a number. For hashing, a formula known as hashing function is applied to attach a numerical value to a word. So, every word is converted into a wordID by using hash function. Inverted index is used to maintain the relationship between WordID and DocID which helps to quickly find the document containing the given word.

Searching:

All the documents matching the index are not equally relevant. Among the millions of documents only the most relevant documents have to be listed. In the simplest case, a search engine could just store the word and the URL where it was found. In reality, this would make for an engine of limited use, since there would be no way of telling whether the word was used in an important or a trivial way on the page, whether the word was used once or many times or whether the page contained links to other pages containing the word. So, to provide quality search results efficiently, searching process has to complete following steps

· Parse the query.

· Convert words into wordIDs using hash function.

· Compute the rank of that document for the query.

· Sort the documents by rank.

· List only the top N numbers of documents.

For those who are interested in the implementation of the web crawler, check out any of the open source crawler listed below:

Heritrix is the Internet Archive's archival-quality crawler, designed for archiving periodic snapshots of a large portion of the Web (written in Java).

ht://Dig includes a Web crawler in its indexing engine.(Written in C)

Larbin a simple web Crawler(Written in c)

Nutch is a scalable crawler written in Java and released under an Apache License. It can be used in conjunction with the Lucene text indexing package.

WIRE - Web Information Retrieval Environment (Baeza-Yates and Castillo, 2002) is a web crawler written in C++ and released under the GPL, including several policies for scheduling the page downloads and a module for generating reports and statistics on the downloaded pages so it has been used for Web characterization.

Ruya Ruya is an Open Source, breadth-first, level-based web crawler written in python.

Universal Information Crawler Simple web crawler, written in Python.

DataparkSearch is a crawler and search engine released under the GNU General Public License.

Python Never bites

2007-08-31T10:06:00.000-07:00

Python:

Python is a general purpose object-oriented, high-level interpreted language. Python was originally developed in the early '90s by Guido van Rossum. His original goal was to develop a language that stresses in readability, simplicity and elegance. Python runs on all major hardware platforms and operating systems, so it doesn't constrain your platform choices.

Python offers high productivity for all phases of the software life cycle: analysis, design, prototyping, coding, testing, debugging, tuning, documentation, deployment, and, of course, maintenance. Python is easy to learn, so it is quite suitable to anyone new to programming, yet at the same time it is powerful enough for the sophisticated expert. There are many sophisticated libraries available which make the programming in python even more equipped. The combination of simplicity, power and portability, along with its open-source nature, has made Python extremely popular.

Who uses python?

Python is used extensively for system administration tasks (it is, for example, a vital component of several Linux distributions).It is also used to teach programming to beginners. Here is list of some organization that uses python

Google has used it to implement many components of its Web crawler and search engine. The most interesting thing is that even the originator of Python, Guido van Rossum, is a Google employee.
NASA uses Python for several of its software systems, and has adopted it as the standard scripting language for its Integrated Planning System.
Industrial Light & Magic, Creator of star wars, uses Python in its production of special effects for large-budget feature films.
Yahoo! uses it (among other things) to manage its discussion groups and Yahoo maps.
Video sharing site Youtube uses it.
Disney uses Python for its animation production applications. It has developed a 3D engine “Panda3d” for the development of interactive graphics. Panda3d is developed as the joint venture of Disney and Carnegie Mellon university(CMU)

To learn more about the organization using python, visit the page http://wiki.python.org/moin/OrganizationsUsingPython

Advantage of python

Python is open source software so, it has huge open source community supporting it.
Python is available on an incredibly wide range of hardware and software platforms. This includes the usual suspects: Sun, Intel, IBM, Microsoft Windows variants, Macintosh OS variants and all *nix system.
Python programs require less time to develop than other high-level languages. Because of the elegance and simplicity of the language, Python programs tend to be 3-5 times shorter than their equivalent in Java, and 5-10 times shorter than C++ equivalents.
Since the python code is highly readable, programs are easier to maintain. So it reduces the maintenance cost which is crucial in software development.
Python has loosely typed language.
It exploits the full power of object oriented approach.
Python programs can be extended using C, C++, or Java. SWIG (Simple Wrapper and Interface Generator) helps to create the wrapper for python.
The popular web development framework for python such as Zope and Plone , Django, TurboGears make it popular for web development.

What the python users say?

Google

"Python has been an important part of Google since the beginning, and remains so as the system grows and evolves. Today dozens of Google engineers use Python, and we're looking for more people with skills in this language." said Peter Norvig, director of search quality at Google, Inc.

YouTube.com

"Python is fast enough for our site and allows us to produce maintainable features in record times, with a minimum of developers," said Cuong Do, Software Architect, YouTube.com.

Industrial Light & Magic

"Python plays a key role in our production pipeline. Without it a project the size of Star Wars: Episode II would have been very difficult to pull off. From crowd rendering to batch processing to compositing, Python binds all things together," said Tommy Burnette, Senior Technical Director, Industrial Light & Magic.

"Python is everywhere at ILM. It's used to extend the capabilities of our applications, as well as providing the glue between them. Every CG image we create has involved Python somewhere in the process," said Philip Peterson, Principal Engineer, Research & Development, Industrial Light & Magic.

Visit http://www.python.org/about/quotes/ to read more quotes from others.

Is Python Suitable for me?

Well, it depends on what you are seeking for. If you are completely new in the field of programming then it’s easy to learn and has good learning curve.

For the rapid development this is the best language. However, for the high end computational and simulation software which involves extremely complex graphics and mathematics, C and C++ may be the better choice. I think quotations from the web/software giants like Google, YouTube and Industrial Light & Magic speak more than thousands of my words.So, you can use to python if you want pleasing coding experience. And its true that python never bites.

NOTE: If you are sure that it’s the programming language for you then get the recent copy of the python from www.python.org . Eclipse IDE with Pydev plug-in will give great coding experience.