I ran into an interesting bug/fact today while messing around with MySQL 5.5. It seems that in the DDL, you can’t say “Type=InnoDB|MyISAM|Foo” anymore. You have to say “Engine=InnoDB”.

This will break your propel:build-all , or propel:build-all-load .. or if you manually try to execute the sql from data/sql/*. You’ll get the following error:

You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘Type=InnoDB’ at line 11

This issue is resolved in Propel 1.5.0, but I’m not sure when Symfony will include that in the Symfony 1.4 series.

To fix this issue, all you need to do is edit one file:

symfony/lib/plugins/sfPropelPlugin/lib/vendor/propel-generator/classes/propel/engine/builder/sql/mysql/MysqlDDLBuilder.php

Line 156, change it to say: $script .= “Engine=$mysqlTableType”;

Hope this helps anyone else using Propel, Symfony and MySQL 5.5

My apologies to Ron Burgundy for the title of the post, but I’m just a fan of the wisest man to ever live.

It’s not for you.

You, being the average PHP developer, systems administrator, hacker.  A lot is being made of how CPU load went down by 50% etc, that’s a great number and on the surface very hard to argue with, but to really understand it, you must understand the problem a lot better. Facebook didn’t decide to write HipHop and use it in production because they had all this money laying around and felt like paying 3 engineers to work on something. They looked at other existing alternatives, implemented them, and when they maxed them out, decided to switch.

HipHop Does Not:

If anything, HipHop should be your last resort.

But, it’ll help me reduce the number of web heads!

I need one MyISAM table in my schema, I’m using Symfony 1.2 and Propel 1.3. As flexible as Symfony and it’s YAML files are, Propel currently doesn’t allow you to change the storage engine on the fly. You use the storage engine as specified in propel.ini. As discussed in the Propel FAQ .

So, the next best thing to do now is to have it so that when Symfony runs propel:insert-sql, I can piggyback my own SQL to it. Luckily, this is possible, and it’s explained here

I now have my own SQL file running at the end, where I do an ALTER TABLE. For the lazy web.

1. Create your Alter table:

ALTER TABLE `database`.`footable` ENGINE=`MyISAM`;

2. Save the file in $projectdir/data/sql/piggyback.sql

3. Edit $projectdir/data/sql/sqldb.map and append the following line:

piggyback.sql=dbconnectioname (usually propel)

Save the file and run

./symfony propel:insert-sql

Goal: Build a PHP 5.3 package, that I can install and upgrade on new ubuntu slices as needed, without having to compile on each box.

This is an amalgamation of different blog posts that did certain things really well, but not everything I wanted. The post I refer to specifically Installing PHP 5.3 on Ubuntu by Brandon Savage .

Prep your system

Setup your development server to be able to compile things. By default, most installations will not come with compilers installed.

apt-get install checkinstall

Say yes, and let it follow the dependencies as needed.

Get the development headers for some of the extensions you’ll be compiling in.

apt-get install postgresql-8.3 postgresql-client-8.3 postgresql-client-common postgresql-common postgresql-server-dev-8.3
aptitude install mysql-client mysql-client-5.0	mysql-common mysql-server mysql-server-5.0 mysql-server-core-5.0

apt-get install libtidy-dev curl libcurl4-openssl-dev libcurl3 libcurl3-gnutls zlib1g zlib1g-dev libxslt1-dev libzip-dev libzip1 libxml2 libsnmp-base libsnmp15 libxml2-dev libsnmp-dev libjpeg62	libjpeg62-dev libpng12-0 libpng12-dev zlib1g zlib1g-dev libfreetype6 libfreetype6-dev libbz2-dev libxpm4-dev libmcrypt-dev libmcrypt4

Download PHP and Compile it

mkdir ~/srcs
cd ~/srcs

wget http://us3.php.net/get/php-5.3.0.tar.gz/from/this/mirror
tar xvfz php-5-3-0.tar.gz
cd php-5.3.0

Now, you can either do the ./configure –help and pick your options, or you can use my configure options for PHP 5.3

make
make test

checkinstall

checkinstall will ask you some basic questions about the package, answer and let it build the package for you. You will run into an interesting issue at first, apxs2 will complain about a LoadModule not being in the httpd.conf file. This is an artifact of how Ubuntu/Debian likes to handle it’s configuration files. The simplest thing is to add a line at the bottom of your httpd.conf that loads a benign module. Mod Pony seems cute .

Configure your PHP like any normal apache server. Reload apache, put the handy phpinfo(); in your documentroot, and then have fun.
Be sure to remove the phpinfo(); from your production servers.

If you want MySQL 5.1, just do apt-get install mysql-server-5.1 , be sure to change the dev header packages to the same version as well.

Notes on using Symfony with multiple databases.

I’m building a new application in symfony, and I need to use some data from an existing application written by another developer.

I can’t just extend the existing application for reasons we don’t need to get into, but I do need to interface with the data, since the existing
app doesn’t really have a nice way to handle a SOA, or share data easily, my next best option is to talk to the original app’s db.

Things you need to think about.

1. No Hacks This is not a hack, symfony/propel support this out of the box, the documentation mentions it briefly but it’s definitely possible.
2. Permissions Should you share the same username/password? or different? In my case, I wanted to make sure symfony couldn’t write to the existing app, so I created a user with SELECT privileges only. (This will affect your ability to run symfony propel:insert-sql (it’ll give you an error))
3. You WILL need MULTIPLE Schema.yml files The docs specifically state this, Symfony will take into account every file ending with schema.yml or schema.xml in the config/ folder
4. Filesystem I want to keep as much of a separation between the two databases, not only is the user different, so is the location of the Models. This can be done easily with changing the package attribute in the schema.yml file.

Conventions / Background

• Existing application is called, “earlyprototype”, and the database name is also “earlyprototype”.
• New application is called, “newfunapp”, and the database is also called “newfunapp”

The How to

Build your YAML files

Since you need a schema.yml or xml file, you can either generate one using a tool that will take your data defintion and create the YAML for you, or you can cheat and build a temp project, configure it to work with earlyprototype , and then do a symfony propel:build-schema . This will give you a schema.yml file in your config directory.

You will take this file and copy it to the newfunapp now, but you’ll give it a different name. For simplicity, you’ll call it earlyprototype.schema.yml.
So you’ll end up with something like.. ~/websites/newfunapp/config/earlyprototype.schema.yml .

For the purpose of this tutorial, take your schema.yml file for the newfunapp, and rename it to newfunapp.schema.yml

Edit your Schema YAML files

You now need to edit your YAML files. They need to look something like the following:

earlyprototype:
  _attributes:
    package: lib.model.earlyprototype
    defaultIdMethod: native

newfunapp:
  _attributes:
    package: lib.model.newfunapp
    defaultIdMethod: native

What you’ve changed here is the name from “propel”, to earlyprototype and newfunapp, so now symfony won’t be confused. And, you’ve told symfony to put the models in their own directories inside the lib/model/ folder of your app.

Edit your database.yml file

Now, you need to tell symfony about the DSN for the different databases. Just like you can specify your original db in the environments, you can do the same for the different dbs. Keep the changes to the schema.yml we made above in mind.

dev:
  newfunapp:
    param:
      classname: DebugPDO
      username: newfunappexampleuser
      password: testingexample
      encoding: utf8
  earlyprototype:
    class: sfPropelDatabase
    param:
      classname: DebugPDO
      dsn: 'mysql:host=localhost;dbname=earlyprototype'
      username: earlyprototypereadonlyuser
      password: testingexample
      encoding: utf8
  propel:
	 class: sfPropelDatabase
	 param:
	    dsn: 'mysql:host=localhost;dbname=pearanalytics3'
	    username: thiscantpossiblywork
	    password: thisisnotevenarealpassword

Possible Bug: For some reason, I noticed I need to have the ‘propel’ connection here, just the name

Build your SQL and Models

Build SQL

 symfony propel:build-sql 

Build Models

symfony propel:build-model

When you build your models, you’ll see that the lib/model/ directory has two subdirectories, that’s where
your models are now.

Connecting to the the different databases

Now that you have multiple connections, you can’t just call the basic methods, you need to specify the connection when you make your queries.

Execute a query with a different connection

AllreportsPeer::doSelect($c, Propel::getConnection('earlyprototype')); 

This relates to the possible bug thing above. If you don’t have a propel connection defined in the databases.yml, the code above will not work.

For the past few months, I’ve been helping my friend develop and market Philtro .
We’ve gone through various iterations of the elevator pitch for it, and the one that seems to be kinda working, is: “It’s like a spam filter for your Twitter account.”

At SXSW, I got the opportunity to talk to Guy Kawasaki about this tool, and he said “There is no spam on twitter, if you don’t like it, don’t follow them”.

While that’s an easy way to handle spam, I also realized that the word Spam means different things to different people.

On Twitter, nothing is UCE. It’s easy to block the profiles with the attractive women, selling Blackberries, iPhones, and who want to chat with me in private with a webcam because 140charactersistoosmalltodiscusstheirdesires.

Philtro is not trying to eliminate viagra ads from your tweetstream. The aim is to help you make the most out of twitter, friendfeed, facebook without alienating people.

Example:

I follow Dan, because he’s a great guy to follow when it comes to design patterns, PHP and Macs. He’s got great insight into those topics, he’s also an avid fan of Twilight, Traveling pants, and underwater basket weaving. I am not.

I want to see what he has to say about the topics I share with him, but I don’t care about his latest Twilight fanclub meetings.

This is where Philtro comes in, it figures out the topics I care about based on the training I give it. “Training” means marking 50 unique
tweets as “Thumbs up ” or “Thumbs down”.

So, Philtro gets rid of uninteresting tweets to you. It’s not a spam filter in the traditional sense of the word, it’s not recommending Twitter users to you (yet…). I can’t think of a word for it, but the best analogy would be a chain of emails at work, to which you’re CC’d on, but the topic is something you have no say on. You can’t remove yourself from other people’s REPLY ALL button, so you’re just stuck deleting emails all day.

If you want to get into the private beta faster, use my super duper special link .

It’s not Unsolicited, and the Tweeter is not “Junk”, what is it?

Marco Tabini has a great post discussing the cost of the cloud, and the current state of affairs. He calls for a simpler cloud platform, not just in terms of cost, but ease of use and products and services that adapt to changes in the market. Though the $100/month Mosso offering is mentioned (this site is hosted on Mosso), I would like to point out the recent acquisitions by Rackspace/Mosso that make their cloud offerings even more compelling than AWS for me.

• Slicehost : Rackspace announced the acquisition in late 2008. This allows me to spin up virtual instances or Slicehost’s version of AMI’s called Slices (this was updated due to Ian’s comment below. You cannot migrate an AMI from EC2 to Slicehost). fairly quickly, and cheaply, but more importantly I know what definitively what my costs will be if I know how much bandwidth I’ll be using up.
• JungleDisk: I think is a good way to get into the consumer space, I’ve told a lot of my friends to use JungleDisk to back up their files, with the obvious caveat to encrypt sensitive information and not to back up your SSN # to the cloud
• Limelight CDN: I think this is just the static file portion, not the video stuff yet, but it’s pretty neat.

Now acquisitions alone don’t make something worthwhile, it’s what you do with them. I’m not a 100% clear on what the plans are with JungleDisk, but I have seen what they’re doing with Slicehost and Limelight and it’s pretty slick.

If you’re a Mosso customer, you can now enable Cloudfiles and Cloud Servers.

Cloudfiles has in my opinion a much better set of documentation of it’s API, and a much lower barrier to entry. I was literally able to upload content to the system in less than 5 minutes of signing up. You can already use Cyberduck and access the cloud files system via a simple to use FTP client. There’s also a firefox extension. This incidentally highlights the power of open source. I believe the Mosso guys did the development themselves for Cyberduck and contributed the code back, I could be wrong. This is a great equalizer for a player late in the game. S3 is now available as an option on almost every major closed source file transfer application, but getting a company to invest time in your system is hard to do. There has to be a demand, which may not come because there’s no easy way to access your system. So they solved the ease of use problem right off the bat.

Apart from ease of access/adoption, what I really like, is that the incoming bandwidth to cloudfiles is free, if you’re using Mosso. So, if my web front end accepts a file upload, I can take that and upload it to cloudfiles without incurring bandwidth costs. So, now I have a load balanced web server and a fairly low cost CDN available to me under one control panel.

Then, add Cloudservers. Which is basically slicehost, without the bandwidth priced in. And the same rules apply here, any bandwidth to and from Mosso and Cloudservers is complimentary. So now, I can spin up my slices of MySQL, and use my flavor of Linux and my custom MySQL patches, and create my own cluster, or master/slave environment without the need to share resources with the other MySQL users in the cloud. This really gives me the flexibility that other shared hosting providers lack.

Lastly, something you can’t put a price on. Customer Support. Rackspace is truly fanatical about support, they really do give a crap, which is refreshing. What’s awesome is that the same quality of support is available for the Cloud offerings as is for their Managed Hosting customers. I can’t afford managed hosting, and I always thought that sure if I’m paying $500/month for an ok system the support better be good, but for $100/month , virtually unlimited websites and clients, and I get the same level of professional/non script reading support? Why should I waste time trying anything else? I have no idea who or what to call or email to when it comes to AWS.

Disclaimer: I think my last paragraph sounds like a marketing gimmick, so.. I’ve edited it a few times, but seriously, in this economy, and even in better economies, customer service is key, and these guys should run a customer service university for other tech companies.

Thanks to a comment by John Frank of Amazon in the comments below. There is indeed a way to get 24/7 support. AWS has had 24/7 phone and email support for quite some time: http://aws.amazon.com/premiumsupport/#overview

I’d like to know who else you think is a major player for the consumer/bootstrap businesses out there as well as enterprises. I’ll be following this post up with a set of tools I use, to make my life in the cloud easier. I’d like to know what you use as well, applications, code libraries, etc.

If you’re considering using Mosso, I suggest you read Snipe’s Notes they are extremely well written and informative, and a coupon for two months free .

Thanks to Expandrive . You can now use Subversion (SVN) on websites hosted at Mosso . The idea of mounting a directory you’d normally ftp/sftp to, and then using SVN on it, at first seemed oddly implausible to me. But, I tried it recently, and got exactly the results I wanted. I even had the repository hosted at Unfuddle . I think this post is fairly obvious, but if you have any questions feel free to ask in the comments.

My last post was for all users, technical and non. This post is for the techies who know me, and are switching. Some are coming from Windows, some from Linux, some are designers who need to do some local development. So, I’ll try to keep this post as simple as possible, and perhaps more like a tutorial, where you can “copy and paste” commands and files.
If you’re going to be doing any kind of web development, chances are you’ll be deploying to a linux/unix server. I don’t know anything about Windows or IIS, so ignore this post if you’re looking for IIS help. I’m also assuming basic level of knowledge with how to find and edit files either via Finder, or via Terminal. For the non unix folks ~ for the rest of the article, means “Your home directory” . Your home directory is usually /Users/yourusername

Security Caveat

If you’re doing this on a laptop, please keep your company, or your clients security and confidentiality in mind. Disable auto login, use strong passwords, enable computer locking via Keychain.
I use these settings as a base. You can and should do a little more, but that’s not the focus of this post.

Things you’ll need

1. XAMPP I prefer it over MAMP, because it seems to get upgraded faster, so the chances of a more recent PHP version are higher. Of course, for the uber nerds, you can get Macports, or Fink. If you’re comfortable, you can even install the Dev Tools CD, get GCC, and compile things yourself, something I used to do, but I no longer need the cutting edge.
2. When I feel like getting cutting edge, I do install Vmware Fusion , turn on a Linux image, and have fun.
3. Expandrive This is more for production, and remote servers, but I find this tool ridiculously handy. It’s better than the free variant of Macfuse, so it’s worth the money. Here’s how it’s better: the way to manage the drives is simpler, and reconnects to remote drives are more stable than the free sshfs variant.
4. You can get MAMP and MAMP Pro MAMP Pro is worth the upgrade, if you decide to get MAMP.
5. MySQL If you need MySQL 5.1, or for some reason 4 or 3, you need to go get it yourself, but XAMPP offers MySQL 5.0 by default
6. Textmate An excellent editor if you don’t like vim/emacs.
7. MacVim This is gVim for mac, with all the mac keyboard bindings, and the power of vim. (this is subjective: and the ease of vim). I’ve replaced the default vim, with this for my personal use
8. Subversion 1.5 By default, OS X comes with SVN 1.4, some of my repositories have been upgraded to 1.5 so I needed to upgrade
9. Git I’ve been playing on some projects which aren’t live, but merely proof of concepts, and I decided to learn git, with those projects. I’m really digging it, though I’m still struggling with Braid and getting it to do things like SVN Externals.
10. Versions App This is a nice GUI for SVN, if you’re into GUIs

Setting up your Shell

Most of these things will go into ~/.bash_profile

Set your editors

Open up ~/.bash_profile in your favorite editor, and add the following lines

export SVN_EDITOR=’vim’
export EDITOR=’vim’

Of course, you can replace vim, with emacs, pico, nano, or if you hate your life, ed.

Change your paths

Since we’ve downloaded XAMPP, macvim, Subversion and Git, we need to make sure the newer version of these apps is available to us.

PATH=/usr/local/git/bin:/opt/local/bin:/usr/local/mysql/bin:$HOME/bin:$PATH
PATH=$PATH:/Applications/xampp/xamppfiles/bin:/opt/subversion/bin

I prefer to move the older binaries out of the way.
Open a new shell and type which php , or which svn
if you see things like /usr/bin/php or /usr/bin/svn, it means you’re still using the older versions. So do the following: (# bash represents your bash prompt, not a comment)

# cd /usr/bin
# sudo mkdir old
# sudo mv php* old
# sudo mv svn* old

You need to do some other things with Macvim, email me if their install instructions aren’t clear to you.

Setup your virtual hosts, and hosts file

Since you’re doing web development, you need to setup apache, and a means to access the virtual hosts on your laptop. Since you’re using XAMPP, you need to make sure that the built in Apache is not turned on. Go to System Preferences -> Sharing and make sure Web Sharing is not checked

The easy, but expensive way

You can download VirtualHost X and manage your virtual hosts with that, but the app is fairly basic, you still need to know your custom directives, if you want to do anything advanced. What’s nice is that it edits your /etc/hosts file for you, so it’s all done in one step. It’s not as advanced as MAMP Pro. But, it’s fairly straightforward and flexible as a GUI.

The somewhat harder, but cheaper way

Since we’re using XAMPP, you need to do the following:

# sudo vim /Applications/xampp/etc/httpd.conf : and uncomment the line asking for httpd-vhosts.conf
# sudo vim /Applications/xampp/etc/extra/httpd-vhosts.conf

Here you can create your own virtual hosts. I use a naming convention of laptop.domain.com, which always points to 127.0.0.1. You could edit your Name servers and add that record, but then it doesn’t work if you don’t have an internet connection. So, I prefer editing my /etc/hosts file.

Example Virtual Host Configuration

NameVirtualHost *:80

#Local Symfony installation for website

< VirtualHost *>
ServerName “laptop.example.com”
DocumentRoot “/Users/vluther/websites/example/web”
<Directory /Users/vluther/websites/example/web/>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
allow from all
</Directory>
Alias /sf /Users/vluther/websites/example/lib/vendor/symfony/data/web/sf

ErrorLog /var/log/apache2/example_error.log

CustomLog /var/log/apache2/example_access.log combined

</VirtualHost>

Notice I keep the logs in /var/log/apache2/. This is so I don’t need to mess with the log rotation scripts of OS X.

Example /etc/hosts file

127.0.0.1 laptop.example.com example
127.0.0.1 laptop.example2.com example2

In conclusion

I’m not going into details of how you create your local working copies with git, svn, or finder. If you really need help with that, email me, or ping me on twitter

If you’re using Django, or RoR, you can skip most of the apache configuration options, and just use the built in web servers that come with the frameworks. For production/stage testing I recommend installing linux on a vmware image, and then messing with FastCGI/mod_python for Django or whatever you need to do with ruby.

By no means is this supposed to be the only way you do things, there’s more than one way to skin this cat. Please share your insights and methods in the comments below.

Jeremy has a great post about why DHH’s article on Sharding is….dumb, i mean flawed.

If you’re a startup, and think you’re about to experience tremendous growth, listen to Jeremy, not DHH. I’ll let Jeremy and the other tech heavy hitters handle the technical flaws with his article.

I’m not sure about his definition of “reasonable” is, but I can’t find a single system with 128GB of RAM within my definition of “reasonable”. Secondly, the whole notion that bigger/better/faster is the solution is absurd. That’s like saying we need a bigger land fill because the one we have currently is full. Sure, that’ll work, but how do you install said landfill into place? do you dig under the current one? do you use “bolt on” land fills?

DHH completely misses the purpose of sharding in my opinion. It’s not just about the load and distribution of reads/writes for mysql, but the organizational benefits are invaluable. I’ve been using some form of sharding for the past 5 years, and it’s helped me considerably. The drawbacks he’s mentioned, can also be a benefit. The number of times I’ve been able to upgrade a certain set of customers without affecting the others, has been invaluable for us, and for the customers. Your entire system doesn’t have to go down just because MySQL doesn’t have online changes yet. We’ve been able to upgrade our system without any noticeable downtime to our customers, because of sharding.

I’m sure he’s right on certain points and situations, but I’d suggest the end user not consider his word as gospel, nor mine, nor Jeremy’s.. your situation is unique, but don’t discard a methodology just because newer/faster hardware is around the corner.