Rafael Sanches

September 1, 2013

Appengine – improving cold startup with spring enabled apps

Filed under: Uncategorized — mufumbo @ 5:59 pm

First of all, in 2013 I refuse to work like 1983. Most of the issues with cold startup seem to be related with appengine being not able to guarantee fast disk operations at all times. There are many times, when their servers have an oversubscribed i/o, that booting time become a bottleneck if you’re doing classpath scanning and things like that.

Below are the bottlenecks for appengine cold startup with spring:

  1. At the very beginning, before spring is even called, appengine do a nice classpath scanning searching for taglib stuff. Even if you don’t use taglib, jsp or any other 1997 stuff.
  2. context:component-scan is slow. Again, because of classpath scanning.
  3. WebApplicationContext takes forever to initialize.
  4. reduce classpath scanning in general.

Easy fixes:

Set default-lazy-init=”true” as global beans level.

You can read https://developers.google.com/appengine/articles/spring_optimization, but just ignore where it says about component-scan, as that one is easily solvable by caching at maven time. (explained below)

Solving Taglib Classpath Scanning (TLD search of file)

Yes, this happens even if you don’t use taglib at all. Here are the symptoms:

INFO] Jul 11, 2013 11:47:51 PM com.google.apphosting.utils.jetty.JettyLogger debug
[INFO] FINE: TLD search of file:/Users/mufumbo/workspace-sts/allthecooks-gae/target/allthecooks-gae-1/WEB-INF/lib/lucene-core-3.6.2.jar
[INFO] Jul 11, 2013 11:47:51 PM com.google.apphosting.utils.jetty.JettyLogger debug
[INFO] FINE: TLD search of file:/Users/mufumbo/workspace-sts/allthecooks-gae/target/allthecooks-gae-1/WEB-INF/lib/appengine-api-1.0-sdk-1.8.1.jar
[INFO] Jul 11, 2013 11:47:51 PM com.google.apphosting.utils.jetty.JettyLogger debug
[INFO] FINE: TLD search of file:/Users/mufumbo/.m2/repository/com/google/appengine/appengine-java-sdk/1.8.1/appengine-java-sdk/appengine-java-sdk-1.8.1/lib/impl/agent/appengine-agentruntime.jar

…….. 500 more of these lines later…

This issue is not solvable. This is my stackoverflow thread about this issue. By the way, God forbid myself from using taglib, jsp, struts or anything related to that ugly technology. Also, if you still use those kind of technologies, take the time to read how to make yourself free from 1997 programming with: spring-mvc + velocity + dcvm.

Solving context:component-scan slowness

There’s a nifty library called reflections that enables you to pre-cache the classpath scan for your annotations with a maven plugin.

Here’s my spring configuration, that replaces :

<reflections:component-scan base-package="com.yumyumlabs" collect="true" save="false" parallel="false">
		<reflections:exclude-filter type="regex" expression="org.springframework.(?!stereotype).*"/>
    </reflections:component-scan>

Attention to the parameter “collect“, it basically says that it will enable you to use the serialized information generated by maven at packaging time. Here’s my maven conf for the plugin:

<plugin>
				<groupId>org.reflections</groupId>
				<artifactId>reflections-maven</artifactId>
				<version>0.9.9-RC1</version>
				<executions>
					<execution>
						<id>reflections-generator</id>
						<goals>
							<goal>reflections</goal>
						</goals>
						<phase>process-classes</phase>
					</execution>
				</executions>
				<configuration>
					<!-- Set com.yumyumlabs because it's the basePath for spring scanning! -->
					<destinations>${project.build.outputDirectory}/META-INF/reflections/com.yumyumlabs-reflections.xml</destinations>
					<includeExclude>+com.yumyumlabs.*</includeExclude>
					<parallel>true</parallel>
					<!-- <serializer>JavaCodeSerializer</serializer> -->
				</configuration>
			</plugin>

Here’s my maven conf for the packages:

<dependency>
			<groupId>org.reflections</groupId>
			<artifactId>reflections</artifactId>
			<version>0.9.9-RC1</version>
		</dependency>
		<dependency>
			<groupId>org.reflections</groupId>
			<artifactId>reflections-spring</artifactId>
			<version>0.9.9-RC1</version>
		</dependency>

How to initialize WebApplicationContext faster?

This is where spring spends most of the booting time.

As a science fiction technique, we should be able to serialize and save it in memcache. Unfortunately this is a mess.

I have tried the approach of ContextLoaderListerner and serialize the ApplicationContext to memcache after it’s properly initialized.

Unfortunately this solution didn’t worked because the beanFactory isn’t serialized together, so I had to refresh the AbstractRefreshableApplicationContext, which cause the complete boot to happen.

This is possible at testing time, but unfortunately it’s just an static cache: http://static.springsource.org/spring/docs/3.2.x/spring-framework-reference/html/testing.html#testcontext-ctx-management-caching

Use proguard to shrink classes and reduce classpath scanning

This topic is complex enough for generating another blog thread. I will post as soon as I have some time. We used the maven plugin net.sf.proguard to do this while in the packaging lifecycle.

It’s a rather complex configuration, but it shrunk our package from 50mb to 23mb.

Our maven config is available here: https://gist.github.com/mufumbo/6406033

Our proguard configuration is available here: https://gist.github.com/mufumbo/6406055

July 26, 2013

Java to Objective-C code conversion

Filed under: Uncategorized — mufumbo @ 7:20 pm

It works! Like science fiction. We have used J2Objc: A Java to iOS Objective-C translation tool and runtime.

On our recipe app we have a core part that converts imperial to the metric standard. This means that 1kg of flour is very different that 1kg of iron, so the code is extensive and has to deal with various string formatting issues.

For such component I was very reluctant to rewrite the entire thing in C or objective-c, as our android and server-side were already tested and working well with the Java version.

Our workflow while doing such task was:

  1. Keep trying to translate the Java code into objective-c. In our case we had issues with java.text.DecimalFormat, so we had to substitute that class with our own implementation. We also had issues with TXT files that we were reading from the classpath, so we just embedded them into the Java code, already formatted.
  2. After we got the whole project to export we used J2Objc to port the code in an extremely ugly format.
  3. We created a nicer objective-c interface that reads the ugly code and presents in a code that looks beautiful.
  4. Then we created a script that does the conversion for us and generates a static objective-c library as result. Awesome!

One codebase, well tested. Have fun and let me know about your adventure!

 

 

April 26, 2013

7 tips for successful consumer app bootstrapping according to Fat Spiderman

Filed under: Uncategorized — mufumbo @ 7:27 pm
fat spiderman

Fat spider man successful entrepreneur

I have always admired this personage in Plaza Mayor (Madrid). Imagine how many millions of people recognize his face after years of hard work?

Do you want your app to be used and seen by tens of millions of people? Follow his strategies!

This will only work if you are developing and focusing on only one high quality app. No average quality apps will succeed with his chops.

August 24, 2011

Craigslist for iOS

Filed under: Uncategorized — mufumbo @ 11:01 pm

This app is my wife‘s creation. She did this as a mini project to learn iOS and Objective-C. Please, download on the appstore for iPadhttp://itunes.apple.com/us/app/craigslist-for-ipad/id457238829 or for iPhonehttp://itunes.apple.com/us/app/craigslist-elite/id493885783

Learn more about this app in this page.

May 7, 2009

Install ffmpeg for alembik transcoding server with 51.40.4 on ubuntu

Filed under: transcoding, Uncategorized — Tags: , , , , , , , , , , , — mufumbo @ 12:31 pm

I find the alembik transcode server a cool piece of software, but I found lots of caveats when started to use it.

The most difficult part is that alembik supports only two versions of libavcodec for ffmpeg, so you must compile it from the sources, since they are quite old.
That happens because ffmpeg changes the parameters from version to version and the alembik team had choosen both 51.40.4 and 51.48.0 to be supported. I think that in the future it will be possible to have an external “.properties” file, so I hope that it will not be necessary to compile ffmpeg by hand, every time.

I’ve used this references:
http://sourceforge.net/mailarchive/message.php?msg_name=48A176CC.8080300%40kimiasol.com (here i’ve found the rpm of a ffmpeg with libavcodec version 51.40.4 to download)
http://juliensimon.blogspot.com/2008/12/howto-compiling-ffmpeg-x264-mp3-xvid.html (this post teach how to install libamr from mediubuntu repository)

install both amr, you can install also from mediubuntu repo.

$ wget http://ftp.penguin.cz/pub/users/utx/amr/amrnb-6.1.0.4.tar.bz2
$ wget http://ftp.penguin.cz/pub/users/utx/amr/amrwb-7.0.0.1.tar.bz2
$ sudo apt-get install g++ nasm zlib1g-dev libx264-dev libfaad-dev faad liba52-0.7.4-dev libgsm1-dev libmp3lame-dev \
libtheora-dev libvorbis-dev libxvidcore4-dev libfaac-dev libimlib2-dev libfreetype6-dev

Use an aplication to convert the rpm to tar and decompress it (with alien or something else):

$ wget http://dag.wieers.com/rpm/packages/ffmpeg/ffmpeg-0.4.9-0.9.20070530.rf.src.rpm

$ ./configure --enable-libmp3lame --enable-libogg --enable-libvorbis --enable-libamr-nb --enable-libfaad --enable-libfaac \
--enable-libgsm --enable-xvid --enable-x264 --enable-liba52 --enable-liba52bin --enable-pp --enable-shared --enable-pthreads \
--enable-libtheora --enable-gpl --disable-strip --enable-libfaadbin --prefix=/usr/local --enable-swscaler --enable-libamr-wb

Install it as a package or with “make && make install” and settup alembik with “ffmpeg.version=51.40.4”

Blog at WordPress.com.

%d bloggers like this: