Rafael Sanches

RSS parsing optimization for bandwidth and processing time with SAX and httpclient – pooling scripts

mufumbo — Sun, 18 Oct 2009 15:55:43 +0000

My server was having a constant income traffic of 1.7mb/s for a service that downloads RSS from the net and process them in order to return the last updates of multiple RSS feeds. It’s a very basic pooling system, but it was downloading too much data for just 5000 active users. The growth wasn’t looking very feasible.

I was using the ROME java library to parse the XML. So far so good, the problem was that it downloads the whole feed and process it all. With my application scope I don’t need to download the whole RSS, just the new entries that i didn’t downloaded yet.

The solution was to use a custom RSS parser, looping through the “” tags and identifying “”. In this way i can parse item per item, and identify if the current item is not updated, so I can abort the http connection and stop the download of the feed. I wish that ROME had an option to do that, like “stop processing when date minor than..”.

The impact on bandwidth usage and in processing time were impressive:

If someone is interested I can post and explain the java class. It’s compatible with com.sun.syndication.feed.synd and uses the SyndEntry and SyndFeed interfaces.

My first public android app – craigslist notification

mufumbo — Thu, 17 Sep 2009 18:16:56 +0000

This application alert the users when new stuff is posted to craigslist. In this way they can get the best deals as soon as they are registered.

The application is totally free and doesn’t use ads either in current or future versions. There is no form of monetization associated with this application. I am using my new server that i am renting from hetzner.
The application does not require registration and does not store information about its usage on the server.

Features

Enable the user to create notifications for certain keywords;
Allow to have all filtering that craigslist.org has;
Has all cities that craigslist.org has;
Preferences menu for configuring the location and network options;
Mark posts as favorite to read at a later time;
The relevant posts are downloaded on the phone, so they can be read later without network;
Faster navigation, since relevant posts are downloaded in batch;

Technical details

It periodically checks for new posts;
In each check it download *only* the updated data, which should be
small after the initial download;
If no update is available it does not download anything at all. This proves to use less bandwidth than a normal navigation app, since:
- Only relevant data is downloaded;
- It never download duplicated data two times;
- It’s not like RSS feeds that it’s the phone downloading and pooling the feeds every time.
  The phone only donwload the updated data one and do it just one time.

Bandwidth usage

For example, a download of 100 new posts takes 50kb of
internet bandwidth.
The daily bandwidth will depend on the number of
notifications that you are monitoring and the number of times that
your notifications get updated.
To reduce bandwidth, try to be specific in your notifications.
For example: try to use “honda civic” instead of “car”

Screenshots

Main screen – notification list

Main screen. This is where your notifications are listed:
Posts listing

When you click on one notification, you go to the post listing.

Notice that you can edit the previously configured notification by clicking on this menu item.
Post detail

Once you click on one post you will see this screen which displays the post details.

You are not required to be connected to the network to see the already downloaded posts.

Notice that you can mark posts as favorite to read them later. The favorite page is accessed from the main menu.
Notification

When there are new posts the app displays a notification on your phone. This can be disabled in the preferences.

reading java-style properties file in PHP

mufumbo — Wed, 05 Aug 2009 04:46:37 +0000

It’s very strange that PHP only support the “parse_ini_string” as configuration function. I don’t like it at all! It has problems handling quotes, new lines, and other caveats.

The only benefit of parse_ini_string against Java Properties file is that it can handle “arrays”, but I don’t think that’s a benefit anyways. I had some trouble because I was wanting to use properties file in php for translations, since I only found buggy versions on the net I had build my own:

      function parse_properties($txtProperties) {
		$result = array();
		$lines = split("\n", $txtProperties);
		$key = "";
		$isWaitingOtherLine = false;
		foreach ($lines as $i => $line) {
			if (empty($line) || (!$isWaitingOtherLine && strpos($line, "#") === 0))
				continue;

			if (!$isWaitingOtherLine) {
				$key = substr($line, 0, strpos($line, '='));
				$value = substr($line, strpos($line, '=')+1, strlen($line));
			}
			else {
				$value .= $line;
			}	

			/* Check if ends with single '\' */
			if (strrpos($value, "\\") === strlen($value)-strlen("\\")) {
				$value = substr($value,0,strlen($value)-1)."\n";
				$isWaitingOtherLine = true;
			}
			else {
				$isWaitingOtherLine = false;
			}

			$result[$key] = $value;
			unset($lines[$i]);
		}

		return $result;
	}

This function can be used to create a php properties class. It should have the same behavior as the Java properties, so it should handle ” quotes and \ for new lines.

Let me know if it have bugs

Retrieve driving directions from google maps with server-side HTTP calls and show results with static maps for WAP

mufumbo — Thu, 23 Jul 2009 08:32:34 +0000

Mobile phones are not famous for supporting javascript. A problem arises when you need to use google API to retrieve the driver directions for a mobile site, or when you just don’t want your site to be overloaded with the complete maps toolkit to show a simple map.

Google provides a good directions-static demo but, unfortunately, it’s javascript-only. They don’t have any example or documentation about how to generate the directions through server side HTTP call.

The driving direction API doesn’t seem to be 100%. I wasn’t able to find examples of how to use the directions API with google maps API-V3 javascript, it seems that it’s not ready yet. I’ve founded this issue tracking in gmaps-api-issues (by the way, the most voted issue) that says that there isn’t support for google maps directions through server side http calls.

Putting all together: I have decided to check what HTTP calls the directions API of google maps API-V2 does and to use it in the server side. I don’t know if it’s forbidden in the terms of service, but it works!

Basically, my PHP server-side is doing the exact same thing that the google directions-static demo is doing in javascript (checkout the source code of the page). It retrieves the answer from the server and write the coordinates information in the static maps URL.

Note that HttpHelper::doGET downloads the string, you can substitute that part with CURL or fopen. Here is the code to make the HTTP calls to get the driving directions and to create the static map image link:

class GoogleGeo {
    public static function buildStaticMap($center, $markers=array(), $width=400, $height=400, $zoom=12, $directions=null) {
        $strMarkers = "";
        foreach($markers as $marker) {
            if (!empty($strMarkers)) $strMarkers .= '|';
            $strMarkers .= urlencode($marker);
        }
        if ($width > 640) $width = 640;
        if (!empty($center)) {
            $center = "¢er=".$center;
        }
        if (!empty($strMarkers)) {
            $strMarkers = "&markers=".$strMarkers;
        }
        if ($zoom > 0) {
            $zoom = "&zoom=$zoom";
        }

        $steps = "";
        if (!empty($directions)) {
            foreach($directions['Directions']['Routes'][0]['Steps'] as $step) {
                $lat = $step['Point']['coordinates'][1];
                $lon = $step['Point']['coordinates'][0];
                if (!empty($steps)) $steps .= "|";
                $steps .= $lat.",".$lon;
            }
            if (!empty($steps)) {
                $steps .= "|".$directions['Directions']['Routes'][0]['End']['coordinates'][1].",".$directions['Directions']['Routes'][0]['End']['coordinates'][0];
                $steps = "&path=rgb:0x0000ff,weight:5|".$steps;
            }
        }

        $staticMap = "http://maps.google.com/staticmap?maptype=mobile&size=".$width."x$height&maptype=roadmap&key=".GOOGLE_MAPS_KEY."&sensor=false$strMarkers$center$zoom$steps";
        return $staticMap;
    }

    public static function retrieveDirections ($from, $to) {
        $params = array('key' => GOOGLE_MAPS_KEY, 'output' => 'json', 'q' => "from: $from to: $to");
        $url = "http://maps.google.com/maps/nav";
        $result = HttpHelper::doGET($url, $params);
        $result = json_decode($result, true);
        return $result;
    }
}

A example how to use this code is:

...
    /* FROM and TO coordinates */
    $markers = array("37.262568,-121.962232,redr", "37.229898,-121.971853,blueg");
    /* Get the driving directions from google api */
    $directions = GoogleGeo::retrieveDirections("485 Alberto Way, Suite 210. Los Gatos, CA 95032", "14109 Winchester Bl, Los Gatos, CA");
    /* Create the map image url with the directions coordinates */
    $staticMap = GoogleGeo::buildStaticMap(null, $markers, 640, 240, null, $directions);
....

In this way you will have $staticMap with a value similar to the image urls in the directions-static demo. In this case it will be:

http://maps.google.com/staticmap?maptype=mobile&size=640×240&maptype=roadmap&key=YOUR_GOOGLE_KEY&sensor=false&markers=37.262568%2C-121.962232%2Credr|37.229898%2C-121.971853%2Cblueg&path=rgb:0×0000ff,weight:5|37.22898,-121.97104|37.22818,-121.97112|37.22597,-121.97231|37.22892,-121.98063|37.23713,-121.97714|37.26301,-121.96088|37.262282,-121.961628

The variable $directions will contain a array with the complete direction steps, so you can easily loop through it and print it out on your WAP application. For example:

Install ffmpeg for alembik transcoding server with 51.40.4 on ubuntu

mufumbo — Thu, 07 May 2009 12:31:07 +0000

I find the alembik transcode server a cool piece of software, but I found lots of caveats when started to use it.

The most difficult part is that alembik supports only two versions of libavcodec for ffmpeg, so you must compile it from the sources, since they are quite old.
That happens because ffmpeg changes the parameters from version to version and the alembik team had choosen both 51.40.4 and 51.48.0 to be supported. I think that in the future it will be possible to have an external “.properties” file, so I hope that it will not be necessary to compile ffmpeg by hand, every time.

I’ve used this references:
http://sourceforge.net/mailarchive/message.php?msg_name=48A176CC.8080300%40kimiasol.com (here i’ve found the rpm of a ffmpeg with libavcodec version 51.40.4 to download)
http://juliensimon.blogspot.com/2008/12/howto-compiling-ffmpeg-x264-mp3-xvid.html (this post teach how to install libamr from mediubuntu repository)

install both amr, you can install also from mediubuntu repo.

wget http://ftp.penguin.cz/pub/users/utx/amr/amrnb-6.1.0.4.tar.bz2
wget http://ftp.penguin.cz/pub/users/utx/amr/amrwb-7.0.0.1.tar.bz2
# apt-get install g++ nasm zlib1g-dev libx264-dev libfaad-dev faad liba52-0.7.4-dev libgsm1-dev libmp3lame-dev \
libtheora-dev libvorbis-dev libxvidcore4-dev libfaac-dev libimlib2-dev libfreetype6-dev

Use an aplication to convert the rpm to tar and decompress it (with alien or something else):

# wget http://dag.wieers.com/rpm/packages/ffmpeg/ffmpeg-0.4.9-0.9.20070530.rf.src.rpm

# ./configure --enable-libmp3lame --enable-libogg --enable-libvorbis --enable-libamr-nb --enable-libfaad --enable-libfaac \
--enable-libgsm --enable-xvid --enable-x264 --enable-liba52 --enable-liba52bin --enable-pp --enable-shared --enable-pthreads \
--enable-libtheora --enable-gpl --disable-strip --enable-libfaadbin --prefix=/usr/local --enable-swscaler --enable-libamr-wb

Install it as a package or with “make && make install” and settup alembik with “ffmpeg.version=51.40.4″

How to correctly make Javascript onclick links and keep in mind SEO and compatibility

mufumbo — Fri, 22 Aug 2008 08:44:06 +0000

Today I have faced a common problem with web developers: How to correctly make Javascript onclick links keep in mind SEO and compatibility.

My solution was very simple, first of all, never use:
Text

This is the worst way of having javascript links. As discussed in this post this is a known issue of IE and why the use of the javascript pseudo-protocol for the value of HREF attributes is actively discouraged.

Suppose that you have an ajax box with tabs that allows you to refresh the content of the tabs with ajax. What would be the best way of assign onclick events to refresh the content with ajax?

What I did was:

When the page is loaded the first tab is loaded in the server side, without ajax. In this way search engines can crawl the contents.
The link on the tabs points to the HREF of the current page with a parameter that permits to change the pre-selected tab in the server side when the page is loaded. Ex: ?sec=onlineusers loads the page with the tab of onlineusers pre-selected and ?sec=lastusersers load the page with lastusers pre-selected.
The link on the tabs have the event onclick associated with a function and it returns false.

Code speaking:Online Users Last registered users

What are the advantages of this solution?

If the user have javascript disabled he can still have a fully functional website.
Search engines and mobile phones can navigate without problem.
It is compatible with all browsers.

For those who have already a website running, the simpliest way of having compatibility is to link the HREF to a page like “/noJavaScriptEnabled.html” and return false in the onclick event. Doing like this seems to be better than use the “javascript:void(0)” or point to the anchor “#”.

new audio toy – terratec dmx 6 fire usb

mufumbo — Fri, 30 May 2008 21:06:41 +0000

Today I have “invested” a reasonable amount of money in the Terratec DMX 6Fire USB witch seems to be a very cool toy! It’s engine runs at 24 bits / 192 kHz and promises crystal-clear sound with a with switched 48V phantom power, a -20 dB pad switch and separated gain controls.

Now I just have to find the best software for starting recording. I am very used to Magix Samplitude, but my last recordings with the open source tool Ardour2 under Linux were not bad at all! I hope I can use it easily with Ubuntu Studio.

database replication tools

mufumbo — Wed, 14 May 2008 21:32:31 +0000

Today I was searching about “replication architectures” and found a very interesting presentation: Portable Scale-Out Benchmarks for MySQL that refers to GORDA – “Open Replication of Databases“. The following tools are the result of my search on that topic:

ESCADA is a opensource implementation of the GORDA replication server interface. It provides a full range of database replication options across a multiple database management systems, in a single inter-operable and evolutive package. Target application scenarios include:

Asynchronous master-slave replication, the no-frills industry standard approach.
Consistent multi-master/update everywhere replication for scalable and high performance shared-nothing clusters.
Zero data-loss inter-cluster replication over WAN for mission critical applications and disaster recovery.

–

SEQUOIA is a database cluster middleware that allows any Java application to transparently access a cluster of databases through JDBC. You do not have to modify client applications, application servers or database server software. You just have to ensure that all database accesses are performed through JDBC.

SEQUOIA allows to achieve scalability, high availability and failover for database tiers. It instantiates the concept of Redundant Array of Inexpensive Databases (RAIDb). The database is distributed and replicated among several nodes and SEQUOIA load balance the queries between these nodes. The server can be accessed from a generic JDBC driver, used by the clients. The client drivers forward the SQL requests to the SEQUOIA controller that balances them on a cluster of replicate d databases (reads are load balanced and writes are broadcasted).

–

Slony-I is a “master to multiple slaves” replication system supporting cascading (e.g. – a node can feed another node which feeds another node…) and failover.

The big picture for the development of Slony-I is that it is a master-slave replication system that includes all features and capabilities needed to replicate large databases to a reasonably limited number of slave systems.

simple script to merge commits from a bugzilla id

mufumbo — Sat, 10 May 2008 21:15:07 +0000

Today i have made my first PERL script!

For me it is very painful when it arrives the time to merge, into another branch, all the commits that i have done in the “trunk”. I have searched a little and did not find anything that could magically solve all my problems. I know that it’s better to create a separated branch when there are lot’s of commits, but there are some cases that a super-simple functionality can explode into a big ball of mud.

Practically the script merge all the commits of a bugzilla id to another branch. If someone knows a standard way to do this; please tell me!

The script take three inputs:

The starting revision ID to filter the search.
The SVN address of the source.
The search string to filter the results. Here you put your bugzilla bug id.

Commands that are executed when you launch the script:

Go to the directory of the destination branch.
To execute the script simply do:
svn_search_merge.pl 0 https://svn.example.com/main/trunk/ “1: “
Note that “1: ” is the bugzilla bug id. What happens next is:
svn log -r 1:HEAD https://svn.example.com/main/trunk/
With that command we get the log of all commits from the revision 1 to the HEAD. After it’s just matter of check if the string “1: ” is inside the log. Then we simply execute:
svn merge -r (ACTUAL_REVISION-1):ACTUAL_REVISION https://svn.example.com/main/trunk/

Source code of the script:

#!/usr/bin/perl

# Simple script to merge commits from a source branch to the current destination directory.
# http://mufumbo.wordpress.com/2008/05/10/simple-script-to-merge-commits-from-a-bugzilla-id/
#
# Example:
# $ cd my-branch-destination/
# $ svn_search_merge.pl 3000 https://svn.example.com/main/trunk/ "bug 673"
# Where 3000 is the starting revision and "bug 673" is the string to match in the comments.
#
use strict;
use warnings;

my $prev_revision = shift;
my $svnHost = shift;
my $searchStr = shift;

print "Starting Revision: $prev_revision\n";
print "SVN addr: $svnHost\n";
print "Search pattern: $searchStr\n";

my $buffer;
$buffer = `svn log -r $prev_revision:HEAD $svnHost`;
my $shouldContinue = "y";
LOGS: foreach my $changelog_entry (split(/----+/m, $buffer)) {
	if($changelog_entry =~ m/($searchStr)/) {
	        #my (undef, $info, undef, $comment) = split(/\n/, $changelog_entry);
	        #next unless $info =~ m/^r/;

		print "\n--------------------------------------------------";
		print $changelog_entry;
		my $revisionId = substr($changelog_entry, 2, 5);
		$revisionId =~ s/^\s+//;
		$revisionId =~ s/\s+$//;

		if ($shouldContinue ne 'a') {
			PROMPT: while(1) {
				print "\nShould continue with merge of revision '$revisionId'? (Yes,Always,Skip,Exit): ";
				$shouldContinue = <>;
				chomp($shouldContinue);

				last PROMPT if $shouldContinue eq 'y';
				last PROMPT if $shouldContinue eq 'a';
				next LOGS if $shouldContinue eq 's';
				die("User requested to stop.") if $shouldContinue eq 'e';
			}
		}
		else {
			print "\nAuto merging '$revisionId'\n";
		}

		my $pRevisionId = $revisionId-1;
		my $mergeBuffer = `svn merge -r $pRevisionId:$revisionId $svnHost`;
		print $mergeBuffer;
	}
}