Appengine – improving cold startup with spring enabled apps

First of all, in 2013 I refuse to work like 1983. Most of the issues with cold startup seem to be related with appengine being not able to guarantee fast disk operations at all times. There are many times, when their servers have an oversubscribed i/o, that booting time become a bottleneck if you’re doing classpath scanning and things like that.

Below are the bottlenecks for appengine cold startup with spring:

1. At the very beginning, before spring is even called, appengine do a nice classpath scanning searching for taglib stuff. Even if you don’t use taglib, jsp or any other 1997 stuff.
2. context:component-scan is slow. Again, because of classpath scanning.
3. WebApplicationContext takes forever to initialize.
4. reduce classpath scanning in general.

Easy fixes:

Set default-lazy-init=”true” as global beans level.

You can read https://developers.google.com/appengine/articles/spring_optimization, but just ignore where it says about component-scan, as that one is easily solvable by caching at maven time. (explained below)

Solving Taglib Classpath Scanning (TLD search of file)

Yes, this happens even if you don’t use taglib at all. Here are the symptoms:

INFO] Jul 11, 2013 11:47:51 PM com.google.apphosting.utils.jetty.JettyLogger debug
[INFO] FINE: TLD search of file:/Users/mufumbo/workspace-sts/allthecooks-gae/target/allthecooks-gae-1/WEB-INF/lib/lucene-core-3.6.2.jar
[INFO] Jul 11, 2013 11:47:51 PM com.google.apphosting.utils.jetty.JettyLogger debug
[INFO] FINE: TLD search of file:/Users/mufumbo/workspace-sts/allthecooks-gae/target/allthecooks-gae-1/WEB-INF/lib/appengine-api-1.0-sdk-1.8.1.jar
[INFO] Jul 11, 2013 11:47:51 PM com.google.apphosting.utils.jetty.JettyLogger debug
[INFO] FINE: TLD search of file:/Users/mufumbo/.m2/repository/com/google/appengine/appengine-java-sdk/1.8.1/appengine-java-sdk/appengine-java-sdk-1.8.1/lib/impl/agent/appengine-agentruntime.jar

…….. 500 more of these lines later…

This issue is not solvable. This is my stackoverflow thread about this issue. By the way, God forbid myself from using taglib, jsp, struts or anything related to that ugly technology. Also, if you still use those kind of technologies, take the time to read how to make yourself free from 1997 programming with: spring-mvc + velocity + dcvm.

Solving context:component-scan slowness

There’s a nifty library called reflections that enables you to pre-cache the classpath scan for your annotations with a maven plugin.

Here’s my spring configuration, that replaces :

<reflections:component-scan base-package="com.yumyumlabs" collect="true" save="false" parallel="false">
		<reflections:exclude-filter type="regex" expression="org.springframework.(?!stereotype).*"/>
    </reflections:component-scan>

Attention to the parameter “collect“, it basically says that it will enable you to use the serialized information generated by maven at packaging time. Here’s my maven conf for the plugin:

<plugin>
				<groupId>org.reflections</groupId>
				<artifactId>reflections-maven</artifactId>
				<version>0.9.9-RC1</version>
				<executions>
					<execution>
						<id>reflections-generator</id>
						<goals>
							<goal>reflections</goal>
						</goals>
						<phase>process-classes</phase>
					</execution>
				</executions>
				<configuration>
					<!-- Set com.yumyumlabs because it's the basePath for spring scanning! -->
					<destinations>${project.build.outputDirectory}/META-INF/reflections/com.yumyumlabs-reflections.xml</destinations>
					<includeExclude>+com.yumyumlabs.*</includeExclude>
					<parallel>true</parallel>
					<!-- <serializer>JavaCodeSerializer</serializer> -->
				</configuration>
			</plugin>

Here’s my maven conf for the packages:

<dependency>
			<groupId>org.reflections</groupId>
			<artifactId>reflections</artifactId>
			<version>0.9.9-RC1</version>
		</dependency>
		<dependency>
			<groupId>org.reflections</groupId>
			<artifactId>reflections-spring</artifactId>
			<version>0.9.9-RC1</version>
		</dependency>

How to initialize WebApplicationContext faster?

This is where spring spends most of the booting time.

As a science fiction technique, we should be able to serialize and save it in memcache. Unfortunately this is a mess.

I have tried the approach of ContextLoaderListerner and serialize the ApplicationContext to memcache after it’s properly initialized.

Unfortunately this solution didn’t worked because the beanFactory isn’t serialized together, so I had to refresh the AbstractRefreshableApplicationContext, which cause the complete boot to happen.

This is possible at testing time, but unfortunately it’s just an static cache: http://static.springsource.org/spring/docs/3.2.x/spring-framework-reference/html/testing.html#testcontext-ctx-management-caching

Use proguard to shrink classes and reduce classpath scanning

This topic is complex enough for generating another blog thread. I will post as soon as I have some time. We used the maven plugin net.sf.proguard to do this while in the packaging lifecycle.

It’s a rather complex configuration, but it shrunk our package from 50mb to 23mb.

Our maven config is available here: https://gist.github.com/mufumbo/6406033

Our proguard configuration is available here: https://gist.github.com/mufumbo/6406055

Java to Objective-C code conversion

It works! Like science fiction. We have used J2Objc: A Java to iOS Objective-C translation tool and runtime.

On our recipe app we have a core part that converts imperial to the metric standard. This means that 1kg of flour is very different that 1kg of iron, so the code is extensive and has to deal with various string formatting issues.

For such component I was very reluctant to rewrite the entire thing in C or objective-c, as our android and server-side were already tested and working well with the Java version.

Our workflow while doing such task was:

1. Keep trying to translate the Java code into objective-c. In our case we had issues with java.text.DecimalFormat, so we had to substitute that class with our own implementation. We also had issues with TXT files that we were reading from the classpath, so we just embedded them into the Java code, already formatted.
2. After we got the whole project to export we used J2Objc to port the code in an extremely ugly format.
3. We created a nicer objective-c interface that reads the ugly code and presents in a code that looks beautiful.
4. Then we created a script that does the conversion for us and generates a static objective-c library as result. Awesome!

One codebase, well tested. Have fun and let me know about your adventure!

 

 

7 tips for successful consumer app bootstrapping according to Fat Spiderman

Fat spider man successful entrepreneur

I have always admired this personage in Plaza Mayor (Madrid). Imagine how many millions of people recognize his face after years of hard work?

Do you want your app to be used and seen by tens of millions of people? Follow his strategies!

This will only work if you are developing and focusing on only one high quality app. No average quality apps will succeed with his chops.

Hazelcast + Spring + @Cacheable

Today I’ve integrated Hazelcast with the @Cacheable spring annotation. I’ve chosen Hazelcast over EhCache + Terracotta, because it has a simpler configuration and there’s no need to run another deamon, which facilitates dev-environment setup.

Here is my hazelcast spring configuration, that is included in my main spring configuration:

<?xmlversion="1.0"encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
    xmlns:context="http://www.springframework.org/schema/context"
    xmlns:hz="http://www.hazelcast.com/schema/spring"
        xmlns:cache="http://www.springframework.org/schema/cache"
    xsi:schemaLocation="http://www.springframework.org/schema/beanshttp://www.springframework.org/schema/beans/spring-beans-3.1.xsd
 http://www.hazelcast.com/schema/springhttp://www.hazelcast.com/schema/spring/hazelcast-spring-2.1.xsd
 http://www.springframework.org/schema/contexthttp://www.springframework.org/schema/context/spring-context-3.1.xsd
 http://www.springframework.org/schema/cachehttp://www.springframework.org/schema/cache/spring-cache-3.1.xsd
 "> 
    <cache:annotation-driven cache-manager="cacheManager" mode="proxy" proxy-target-class="true" />

    <bean
        class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"
        p:systemPropertiesModeName="SYSTEM_PROPERTIES_MODE_OVERRIDE">
        <property name="locations">
            <list>
                <value>classpath:/hazelcast-default.properties</value>
            </list>
        </property>
    </bean>

    <hz:hazelcast id="instance">
        <hz:config>
            <hz:group name="mygroup" password="mypassword" />
            <hz:network port="5700" port-auto-increment="false">
                <hz:join>
                    <hz:multicast enabled="true" />
                    <hz:tcp-ip enabled="true">
                        <hz:interface>127.0.0.1:5700</hz:interface>
                    </hz:tcp-ip>
                </hz:join>
                <hz:interfaces enabled="true">
                    <hz:interface>127.0.0.1</hz:interface>
                </hz:interfaces>
            </hz:network>

            <hz:map name="default">
                <hz:map-store enabled="true" write-delay-seconds="0"
                    class-name="com.mufumbo.server.cache.hazelcast.EmptyCacheMapLoader" />
            </hz:map>

            <hz:map name="null-map" />

            <hz:map name="app" backup-count="3" async-backup-count="1"
                time-to-live-seconds="10" max-size="100" eviction-percentage="50"
                cache-value="true" eviction-policy="LRU" merge-policy="hz.LATEST_UPDATE" />
        </hz:config>
    </hz:hazelcast>

    <hz:config id="liteConfig">
        <hz:lite-member>true</hz:lite-member>
    </hz:config>

    <!-- set hazelcast spring cache manager -->  
    <bean id="cacheManager" class="com.hazelcast.spring.cache.HazelcastCacheManager">
        <constructor-arg ref="instance" />
    </bean>
</beans>

Please, notice the mode=”proxy” proxy-target-class=”true”. Without that configuration the beans with a super constructor and @Cacheable haven’t loaded. Notice that this isn’t a Hazelcast issue, it’s a Spring AOP issue, even if you use the SimpleCacheManager instead of Hazelcast one.

I was wondering about using mode=”aspectj”, but it was taking too much time, so maybe another day.

ATTENTION: CGLib proxies requires that the class needs to provide a default constructor, i.e. without any arguments. Otherwise you’ll get an IllegalArgumentException: “Superclass has no null constructors but no arguments were given.” This makes constructor injection impossible.

ATTENTION 2: Be careful if you have a BeanNameAutoProxyCreator matching the class that you flag as @Cacheable. In that case it means that there’s already an Cglib proxy behind, which can’t happen. It’s a hassle because I was using a BeanNameAutoProxyCreator to match all my *Service classes in order to create the JDO transactions on the methods create*, update* and save*. If you had the same problem, replace all your BeanNameAutoProxyCreator configuration with an AOP configuration like:

<tx:annotation-driven transaction-manager="transactionManager" mode="proxy" proxy-target-class="true" />

    <aop:config>
        <!-- http://blog.espenberntsen.net/tag/pointcut/ -->
        <!-- For all the classes annotated with @Service -->
        <aop:pointcut id="serviceMethodsCut" expression="within(@org.springframework.stereotype.Service *)" /> 
        <aop:advisor advice-ref="txAdvice" pointcut-ref="serviceMethodsCut" />
    </aop:config>

    <aop:config>
        <!-- For all the methods annotated with @Transactional -->
        <aop:pointcut id="transactionalCut" expression="execution(@org.springframework.transaction.annotation.Transactional * *(..))" />
        <aop:advisor advice-ref="txAdvice" pointcut-ref="transactionalCut" />
    </aop:config>

    <tx:advice id="txAdvice" transaction-manager="transactionManager">
        <tx:attributes>
            <tx:method name="update*" propagation="REQUIRES_NEW" rollback-for="java.lang.Exception"/>
            <tx:method name="insert*" propagation="REQUIRES_NEW" rollback-for="java.lang.Exception"/>
            <tx:method name="create*" propagation="REQUIRES_NEW" rollback-for="java.lang.Exception"/>
            <tx:method name="delete*" propagation="REQUIRES_NEW" rollback-for="java.lang.Exception"/>
            <tx:method name="save*" propagation="REQUIRES_NEW" rollback-for="java.lang.Exception"/>
            <tx:method name="store*" propagation="REQUIRES_NEW" rollback-for="java.lang.Exception"/>
            <tx:method name="get*" propagation="REQUIRED" read-only="true" />
            <tx:method name="*" propagation="SUPPORTS" read-only="true" />
        </tx:attributes>
    </tx:advice>

 

Also, remember to update your pom.xml with the AspectJ configuration:

<dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-aspects</artifactId>
            <version>${org.springframework-version}</version>
        </dependency>

        <dependency>
            <groupId>aspectj</groupId>
            <artifactId>aspectjrt</artifactId>
            <version>1.5.4</version>
        </dependency>

        <dependency>
           <groupId>org.aspectj</groupId>
           <artifactId>aspectjweaver</artifactId>
           <version>1.6.12</version>
        </dependency>

            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>aspectj-maven-plugin</artifactId>
                <version>1.4</version>
                <configuration>
                    <Xlint>warning</Xlint>
                    <complianceLevel>1.7</complianceLevel>
                    <source>1.7</source>
                    <target>1.7</target>
                    <encoding>UTF-8</encoding>
                    <aspectLibraries>
                        <aspectLibrary>
                            <groupId>org.springframework</groupId>
                            <artifactId>spring-aspects</artifactId>
                        </aspectLibrary>
                    </aspectLibraries>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>test-compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

Spring-mvc + Velocity + DCEVM

Java web development can be frustrating at times. Things that slow me down:

• Excessive pre-configuration to be able to deliver results
• Many people use JSP’s in a confusing way and end up with mixed patterns in the views.
• Server restarts and deployment kill my productivity.

In order to improve dev speed I have been using these three technologies together:

• Spring-MVC: This one makes it easier to bind controllers and views together, simply using annotations. This makes it very easy to create well defined controllers before the execution of the view.
• Velocity: One of the most simple and powerful template engine available. With this I can define clear and simple templates that can access and interact with Java objects at the runtime.
• DCEVM: Dynamic Code Evolution VM. A modification of the Java HotSpot(TM) VM that allows unlimited class redefinition at runtime. In our case it will enable to deploy changes in java classes without restarting the servlet container.

Since there are many tutorials on how to use these technologies singularly, this post will only cover how to bind these three technologies together. I would also suggest the usage of Maven to glue the dependencies together.

Lately many people are excited about the Play Framework which adds speed to Java development. Personally, I don’t like being too tight to a framework, but it seems very good.

I would recommend veloeclipse eclipse plugin for coloring the templates.

Glueing Velocity With Spring

There are many tutorials covering this part, like this one from velocity and this one from spring, but no one of them talks about WebappResourceLoader and how to use relative paths in the velocity templates and in the controllers.

Here’s the spring configuration that I use:

    <bean id="viewResolver"
        class="org.springframework.web.servlet.view.velocity.VelocityViewResolver">
        <property name="cache" value="true" />
        <property name="prefix" value="" />
        <property name="suffix" value=".vm" />
        <property name="toolboxConfigLocation" value="/WEB-INF/velocity/tools.xml" />
        <property name="exposeRequestAttributes" value="true"/>
        <property name="exposeSessionAttributes" value="true"/>
        <property name="exposeSpringMacroHelpers" value="true"/>

        <property name="attributesMap">
            <map>
                <entry key="dateTool"><bean class="org.apache.velocity.tools.generic.DateTool" /></entry>
                <entry key="escapeTool"><bean class="org.apache.velocity.tools.generic.EscapeTool" /></entry>
            </map>
        </property>
    </bean>

    <bean id="velocityConfig"
        class="org.springframework.web.servlet.view.velocity.VelocityConfigurer">
        <property name="configLocation" value="/WEB-INF/velocity/velocity.properties" />
        <property name="resourceLoaderPath">
            <value>/</value>
        </property>
        <property name="velocityProperties">
           <props>
                 <prop key="contentType">text/html;charset=UTF-8</prop>
           </props>
          </property>
    </bean>

It’s very important to use  org.apache.velocity.tools.view.WebappResourceLoader in order to facilitate development.

Using ClasspathResourceLoader makes development painful because depending on your configuration it won’t reload the templates when you’re changing, or sometimes it will refresh the entire webapp context in order to refresh a single template. This process can take you minutes after each template change.

Here’s my configuration for velocity.properties:

runtime.log.invalid.reference = true
runtime.log.logsystem.class=org.apache.velocity.runtime.log.CommonsLogLogChute

input.encoding=UTF-8
output.encoding=UTF-8

directive.include.output.errormsg.start = 

directive.parse.max.depth = 10

velocimacro.library.autoreload = true
velocimacro.library = /VM_global_library.vm
velocimacro.permissions.allow.inline = true
velocimacro.permissions.allow.inline.to.replace.global = false
velocimacro.permissions.allow.inline.local.scope = false

velocimacro.context.localscope = false

runtime.interpolate.string.literals = true

resource.manager.class = org.apache.velocity.runtime.resource.ResourceManagerImpl
resource.manager.cache.class = org.apache.velocity.runtime.resource.ResourceCacheImpl

resource.loader = webapp, class

class.resource.loader.description = Velocity Classpath Resource Loader
class.resource.loader.class = org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader 

webapp.resource.loader.class = org.apache.velocity.tools.view.WebappResourceLoader
webapp.resource.loader.path = /WEB-INF/views/
webapp.resource.loader.cache = true
webapp.resource.loader.modificationCheckInterval = 2

Now you can start writing a simple java controller:

@Controller
public class MyControllerClass {
    @RequestMapping(value = "/my-url-path/{myPathVariable}", method = RequestMethod.GET)
    public String processRequest(@PathVariable String myPathVariable, @RequestParam(required = false) Long editId, Model model) {
        model.addAttribute("world", "world");
        return "/pages/forum/admin/forum-create";
    }
}

Now you can create simple velocity view with a templates like:

#parse("/parts/header.vm")

#parse("/parts/left-menu-p.vm")

Hello ${world}.

#parse("/parts/footer.vm")

Notice that no XML configuration was necessary, just the creation of the view and the controller. Also, thanks to DCEVM, there’s no need to restart the webapp after creating a new controller.

After configuring spring-mvc + velocity, the most important part is to configure DCEVM in order to not need to restart our tomcat container after every change in the classpath.

Configuring DCEVM

First download the binary from http://ssw.jku.at/dcevm/binaries/

If you’re in a windows or linux environment, just open the jar and choose the JDK that you want to have the modified VM running. Notice that you can enable and disable DCEVM support for that JDK. On Linux only the 32-bit JDK is supported.

If you’re running MacOSX you’ll have to download the 32-bit version of the Soylatte VM. This is available here: http://landonf.bikemonkey.org/static/soylatte/bsd-dist/javasrc_1_6_jrl_darwin/soylatte16-i386-1.0.3.tar.bz2

Unzip the soylatte VM under /Library/Java/JavaVirtualMachines/soylatte16-i386-1.0.3/

Now just run the dcevm-0.2.jar and choose /Library/Java/JavaVirtualMachines/soylatte16-i386-1.0.3/

After setting up the new VM we need to setup the eclipse project.

1. Select the project properties and use the new JRE.
2. Download tomcat and create a new server in eclipse, choose tomcat.
3. Make it sure that your tomcat run in your newly created JRE.
4. Open the new server configuration and make it sure to check “Automatically publish when resources change”
5. Go to the server “modules” tab and edit your project web modules. Disable “Auto Reload”. This is extremely important since it will save you hours and days of restarting time.
6. Just run your tomcat and every time you make a change on your project it will be pushed to the server. No need to restart!

For a more compreensive tutorial about how to configure tomcat + eclipse, please visit: How to Set Up Hot Code Replacement with Tomcat and Eclipse

If you like the DCEVM idea, also take a look into jRebel, which is even more powerful.

Also, if you use the Datanucleus JDO database, don’t forget to install the JDO eclipse plugin, in this way your classes are compiled on the fly after changes, so no need to restarts for enhancement.

Configuring Run-Jetty-Run Eclipse Plugin

Another simpler way is to install the Run Jetty Run plugin and run it over the new soylatte jvm. When creating your debug profile, remember to click in the JRE tab and choose the Soylatte VM. Running a jetty container with this plugin is 20 times faster than making a “mvn jetty:run”.

Install the plugin from their update site: http://run-jetty-run.googlecode.com/svn/trunk/updatesite

After the plugin is installed, remember to enable build automatically and disable source scanner in the jetty plugin:

This will make it possible to save your java files and velocity templates without restarting the server. One cool configuration that I’ve made on mine is that I added the spring and velocity configuration directories to the “Custom Scan Folder and Files”, so every time I change any files it will redeploy the webapp. Notice that redeploying the webapp with this plugin is 20 times faster than making a “mvn jetty:run” from scratch.

Craigslist for iOS

This app is my wife‘s creation. She did this as a mini project to learn iOS and Objective-C. Please, download on the appstore for iPad: http://itunes.apple.com/us/app/craigslist-for-ipad/id457238829 or for iPhone: http://itunes.apple.com/us/app/craigslist-elite/id493885783

Learn more about this app in this page.

Google Analytics lags on Android. How to make it more responsive!

Google Analytics can be your best friend in order to track your mobile user behavior. Unfortunately the current Android implementation has performance limitations and the most problematic is that it uses SQLite to store your events.

Everyone who wants to write a responsive app knows that you can’t do SQLite operations in the UI Thread. Having to wrap the Google Analytics calls into a separated thread can be painful, so I wrote a very simple helper to handle it inside threads. I have many tracking events inside “button click” and it was taking about 200ms to execute, it’s too much on the UI Thread. It’s also too much if you have “onCreate” because it will take long time to open your new activity.

This helper is also very wrong because it maintains a static reference to the context. I do this in order to have better numbers on visit and “time on site”. You can just remove the static reference if you don’t like that.

Notice that my implementation has this: “Thread.sleep(3000);”
It means that I don’t want repetitive Google Analytics SQLite to be competing with my app inserts or gets.

This LAG happens because SQLite uses the internal memory which can be very slow depending on many factors, including concurrent SQLite operations or just internal memory without many space.

I hope it helps someone. Here’s the complete code:

package com.mufumbo.android.helper;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import android.content.Context;
import android.util.Log;

import com.google.android.apps.analytics.GoogleAnalyticsTracker;

public class GAHelper {
    String activity;
    static GoogleAnalyticsTracker tracker;
    static int instanceCount = 0;
    long start;

    // Limit the number of events due to outofmemory exceptions of analytics sdk
    final static int MAX_EVENTS_BEFORE_DISPATCH = 200;
    static int eventCount = 0;

    static final ExecutorService tpe = Executors.newSingleThreadExecutor();

    public GAHelper(final Context c, final String activity) {
        this.activity = activity;
        instanceCount++;
        if (tracker == null) {
            tpe.submit(new Runnable() {
                @Override
                public void run() {
                    tracker = GoogleAnalyticsTracker.getInstance();
                    tracker.start(Constants.GOOGLE_ANALYTICS_ID, Constants.GOOGLE_ANALYTICS_DELAY, c.getApplicationContext());
                }
            });
        }
    }

    public void onResume() {
        this.trackPageView("/"+this.activity);
    }

    public synchronized void destroy () {
        instanceCount--;
        if (instanceCount <= 0) {
            tpe.submit(new Runnable() {
                @Override
                public void run() {
                    Log.i(Constants.TAG, "destroying GA");
                    if (tracker != null)
                        tracker.stop();
                    instanceCount = 0;
                }
            });
        }
    }

    protected void tick() throws InterruptedException {
        Thread.sleep(3000);
        this.start = System.currentTimeMillis();
    }

    public void log (final String l) {
        if (Dbg.IS_DEBUG) {
            Dbg.debug("['"+(System.currentTimeMillis()-start)+"']["+eventCount+"] Logging on '"+this.activity+"': "+l);
            if (l.contains(" ")) {
                Log.e(Constants.TAG, "DO NOT TRACK WITH SPACES: "+l, new Exception());
            }
        }

    }

    public void trackClick(final String button) {
        checkDispatch();
        tpe.submit(new Runnable() {
            @Override
            public void run() {
                try {
                    tick();
                    tracker.trackEvent(
                            "clicks",  // Category
                            activity+"-button",  // Action
                            button, // Label
                            1);
                    log("trackClick:"+button);
                } catch (final Exception e) {
                    Log.e(Constants.TAG, "Error tracking", e);
                }
            }
        });
    }

    public void trackEvent (final String category, final String action, final String label, final int count) {
        checkDispatch();
        tpe.submit(new Runnable() {
            @Override
            public void run() {
                try {
                    tick();
                    tracker.trackEvent(
                            category,  // Category
                            action,  // Action
                            activity+"-"+label, // Label
                            1);
                    log("trackEvent:"+category + "#"+action+"#"+label+"#"+count);
                } catch (final Exception e) {
                    Log.e(Constants.TAG, "Error tracking", e);
                }
            }
        });
    }

    public void trackPopupView (final String popup) {
        checkDispatch();
        tpe.submit(new Runnable() {
            @Override
            public void run() {
                try {
                    tick();
                    final String page = "/"+activity+"/"+popup;
                    tracker.trackPageView(page);
                    log("trackPageView:"+page);
                } catch (final Exception e) {
                    Log.e(Constants.TAG, "Error tracking", e);
                }
            }
        });
    }

    public void trackPageView (final String page) {
        checkDispatch();
        tpe.submit(new Runnable() {
            @Override
            public void run() {
                try {
                    tick();
                    tracker.trackPageView(page);
                    log("trackPageView:"+page);
                } catch (final Exception e) {
                    Log.e(Constants.TAG, "Error tracking", e);
                }
            }
        });
    }

    public void checkDispatch() {
        eventCount++;
        if (eventCount >= MAX_EVENTS_BEFORE_DISPATCH)
            dispatch();
    }

    public void dispatch(){
        eventCount = 0;
        tpe.submit(new Runnable() {
            @Override
            public void run() {
                try {
                    tick();
                    tracker.dispatch();
                    log("dispatched");
                } catch (final Exception e) {
                    Log.e(Constants.TAG, "Error dispatching", e);
                }
            }
        });
    }
}

HTC sense ui breaks the way the WebView settings behave

I am “amused” to discover that HTC sense has changed even the way android breaks to access of a WebView from outside the UI thread.

Since I don’t own a phone with HTC sense, it took me months to understand this and I only fixed the issue thanks to a kind user who sent me logs.

For example, calling this outside from a UI thread works on the normal android SDK’s:
webSettingsObj.setBuiltInZoomControls(true);

Instead, on HTC Sense phones it breaks here:

java.lang.RuntimeException: Can't create handler inside thread that has not called Looper.prepare()
 at android.os.Handler.(Handler.java:121)
 at com.htc.multitouch.MultiTouchDetector$MultiTouchHandler.(MultiTouchDetector.java:180)
 at com.htc.multitouch.MultiTouchDetector.(MultiTouchDetector.java:242)
 at android.webkit.WebView$HTCMultiTouch.fnCreateMultiTouchListener(WebView.java:11781)
 at android.webkit.WebView.enableMultiTouch(WebView.java:10893)
 at android.webkit.WebView.updateMultiTouchSupport(WebView.java:905)
 at android.webkit.WebSettings.setBuiltInZoomControls(WebSettings.java:474)

Now, I know know that anything that changes the UI shouldn’t be executed outside of the UI Thread, but still..

upload using multipart post using httpclient in android

A very common caveat, when doing android applications, is fighting to keep the APK size small.

Many applications need the ability to upload binary data to their server and when you arrive there you see that the android SDK doesn’t have the http-client libraries to send multipart posts.

The easiest way is to include the JAR for httpmime and apache_mime4j, but it takes way too much space; 300kb.

To overcome this you can implement your own HttpEntity. In this way the code is:

public class SimpleMultipartEntity implements HttpEntity {

    private final static char[] MULTIPART_CHARS = "-_1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
        .toCharArray();

    private String boundary = null;

    ByteArrayOutputStream out = new ByteArrayOutputStream();
    boolean isSetLast = false;
    boolean isSetFirst = false;

    public SimpleMultipartEntity() {
        final StringBuffer buf = new StringBuffer();
        final Random rand = new Random();
        for (int i = 0; i < 30; i++) {
            buf.append(MULTIPART_CHARS[rand.nextInt(MULTIPART_CHARS.length)]);
        }
        this.boundary = buf.toString();

    }

    public void writeFirstBoundaryIfNeeds(){
        if(!isSetFirst){
            try {
                out.write(("--" + boundary + "\r\n").getBytes());
            } catch (final IOException e) {
                Log.e(Constants.TAG, e.getMessage(), e);
            }
        }
        isSetFirst = true;
    }

    public void writeLastBoundaryIfNeeds() {
        if(isSetLast){
            return ;
        }
        try {
            out.write(("\r\n--" + boundary + "--\r\n").getBytes());
        } catch (final IOException e) {
            Log.e(Constants.TAG, e.getMessage(), e);
        }
        isSetLast = true;
    }

    public void addPart(final String key, final String value) {
        writeFirstBoundaryIfNeeds();
        try {
            out.write(("Content-Disposition: form-data; name=\"" +key+"\"\r\n").getBytes());
            out.write("Content-Type: text/plain; charset=UTF-8\r\n".getBytes());
            out.write("Content-Transfer-Encoding: 8bit\r\n\r\n".getBytes());
            out.write(value.getBytes());
            out.write(("\r\n--" + boundary + "\r\n").getBytes());
        } catch (final IOException e) {
            Log.e(Constants.TAG, e.getMessage(), e);
        }
    }

    public void addPart(final String key, final String fileName, final InputStream fin){
        addPart(key, fileName, fin, "application/octet-stream");
    }

    public void addPart(final String key, final String fileName, final InputStream fin, String type){
        writeFirstBoundaryIfNeeds();
        try {
            type = "Content-Type: "+type+"\r\n";
            out.write(("Content-Disposition: form-data; name=\""+ key+"\"; filename=\"" + fileName + "\"\r\n").getBytes());
            out.write(type.getBytes());
            out.write("Content-Transfer-Encoding: binary\r\n\r\n".getBytes());

            final byte[] tmp = new byte[4096];
            int l = 0;
            while ((l = fin.read(tmp)) != -1) {
                out.write(tmp, 0, l);
            }
            out.flush();
        } catch (final IOException e) {
            Log.e(Constants.TAG, e.getMessage(), e);
        } finally {
            try {
                fin.close();
            } catch (final IOException e) {
                Log.e(Constants.TAG, e.getMessage(), e);
            }
        }
    }

    public void addPart(final String key, final File value) {
        try {
            addPart(key, value.getName(), new FileInputStream(value));
        } catch (final FileNotFoundException e) {
            Log.e(Constants.TAG, e.getMessage(), e);
        }
    }

    @Override
    public long getContentLength() {
        writeLastBoundaryIfNeeds();
        return out.toByteArray().length;
    }

    @Override
    public Header getContentType() {
        return new BasicHeader("Content-Type", "multipart/form-data; boundary=" + boundary);
    }

    @Override
    public boolean isChunked() {
        return false;
    }

    @Override
    public boolean isRepeatable() {
        return false;
    }

    @Override
    public boolean isStreaming() {
        return false;
    }

    @Override
    public void writeTo(final OutputStream outstream) throws IOException {
        outstream.write(out.toByteArray());
    }

    @Override
    public Header getContentEncoding() {
        return null;
    }

    @Override
    public void consumeContent() throws IOException,
    UnsupportedOperationException {
        if (isStreaming()) {
            throw new UnsupportedOperationException(
            "Streaming entity does not implement #consumeContent()");
        }
    }

    @Override
    public InputStream getContent() throws IOException,
    UnsupportedOperationException {
        return new ByteArrayInputStream(out.toByteArray());
    }

}

server-side calls are better for Ad serving. Downside of AFMA/AdMob.

Ad networks that offer server-side calls gives us a big advantage. We can simply get a image + link and print on a android WebView.

Due to limitations of the android framework it’s the only way to have non-intrusive ads inside a WebView. Specially when the WebView must be layout_height=”fill_parent”, for a correct scrolling and zoom control.

JavaScript ads are not an option, since the only work around is to inject JavaScript into the WebView which prevents me from working with JavaScript ads that are based on a inline include.

Advantages of the server-side call instead of SDK:
– Reduce APK size: I can implement my own WebView to print ads. AdMob SDK takes 140kb, Google AFMA takes 40kb, so 30% of my app is taken from ads sdk’s.
– AFMA SDK is slow: rendering the adsense javascript is slow. I feel my app freezing and coming back after rendering adsense. Admob is much faster.
– The developer have total control to optimize it.

FYI: After the death of Quattro, the best network that is offering server-side calls is Millenial Media.