Monday, February 22, 2010

How I built FeederTweeter in less than a day (appengine/oauth/tweepy/feedparser/bitly)

In this post, I'll look at some basics of what it took to stitch together a simple appengine web app which checks an atom feed and posts new entries to a twitter account. This is mostly an exercise, though I do intend to use it personally. I realize there are existing sites (and blogging services) that provide this type of functionality out of the box.

I'm not planning on making the service itself publicly available, but I am open sourcing it. You can easily setup and deploy your own instance on appengine for your own purposes. It's not ready for multiple users or a public site, but I do think it is ready for you to use it personally on your own appengine account.

Let's get started.

Part 1: Parsing the atom feed

Universal Feed Parser is an excellent python library for parsing atom/rss feeds and ever since google added transperant urllib support to appengine (in v1.1.9), it's been even simpler to use this 3rd party lib. Here's an example of using feedparser:
 import feedparser
atomxml = feedparser.parse('http://netsmith.blogspot.com/feeds/posts/default')
entries = atomxml['entries']
You can then pull out titles, links, etc with expressions like these:
 print entries[0]['title'] # title of the first entry
print entries[1].link # link from the second entry
print [entry.title for entry in entries] # titles for all entries
Universal feed parser provides plenty of ways to get at the structured results of the Atom/rss feed. See Universal Feed Parser for some excellent concise examples.

To check for new entries, we could write a pubsubhub integration or poll the rss/atom feed every so often and check for new entries. Since my blog is still on blogger.com (crazy, I know!), we'll use a polling strategy (booo!) by creating a cron.yaml file that looks like:
cron:
- description: Check for new posts and tweet them
schedule: every 5 minutes
url: /tasks/poller
and then creating a recurring task that retrieves the feed and checks to see if there are any new posts like this:
...
url = 'http://netsmith.blogspot.com/feeds/posts/default'
d = feedparser.parse(url)
idsThatHaveAlreadyBeenTweeted = getAlreadyTweetedEntries(url) # retrieve from datastore

for entry in d['entries']:
if not entry.id in idsThatHaveAlreadyBeenTweeted:
tweet(entry)
updateTweetedEntries(url, entry) # updates the datastore
...
As far as fetching the feed and checking for new entries, that's pretty much it!

Part 2: Posting to twitter

Posting to twitter is a relatively straightforward process, and I was able to hack something together very quickly by building on the shoulders of some excellent libraries and examples (specifically tweepy and tweepy-examples/app-engine). Coincidentally, right as I reached the point where I got it all figured out, I saw Nick Johnson (google) post this article in which he outlines how to authenticate a user with twitter using appengine-oauth on appengine.

Rather than re-hash what he and others have explained fairly well (OAuth), I want to just recommend you either check-out the tweepy app engine example and start tweaking/experimenting from there or follow through Nick's article for a little bit more explanation. Either way, you need to be writing some code around oauth to really get your hands around it and either place is an excellent start.

I do want to point out an interesting wrinkle though when it comes to oauth on appengine.

Since you can specify a callback url for oauth, you'll want to use code like the following to set the appropriate oauth callback url depending on whether you're testing your app locally or running it in the cloud on appengine:

import os
if os.environ.get('SERVER_SOFTWARE', '').startswith('Devel'): # running on local server
TWITTER_CALLBACK = 'http://127.0.0.1:8080/oauth/callback'
else:
TWITTER_CALLBACK = 'http://yourapp.appspot.com/oauth/callback'
If you run into any 'unauthorized' messages while testing twitter oauth,
try:
  1. resetting you twitter api key
  2. ensuring that the twitter method you're using is one your authorized for -- i.e. - make sure you're set to read/write access at the twitter api key level if you're attempting to post a tweet.
Once you've successfully authorized through OAuth, post the tweet with tweepy:

# after a successful oauth authentication w/twitter
import tweepy

tweet = 'Hello twitter world'
auth = tweepy.OAuthHandler(TWITTER_CONSUMER_KEY, TWITTER_CONSUMER_SECRET)
auth.set_access_token(thisUser.access_token_key, thisUser.access_token_secret)

api = tweepy.API(auth_handler=auth, secure=True, retry_count=3)
api.update_status(tweet) # post the tweet

When it comes to integrating with the twitter api on appengine, it's really easy and straightforward with tweepy, once you've figured out OAuth (which should only take a couple of hours at most).

Part 3: Shortening url's

Once you can parse atom feeds, authenticate users, and post to twitter, the only piece that's really missing is shortening those long (thanks again blogger!) blog urls. For this we'll use bit.ly. Here's a simple method for shortening a url with bit.ly:

from django.utils import simplejson
from google.appengine.api import urlfetch

class BitLy():
def __init__(self, login, apikey):
self.login = login
self.apikey = apikey

def shorten(self,param):
url = "http://" + param
request = "http://api.bit.ly/shorten?version=2.0.1&longUrl="
request += url
request += "&login=" + self.login + "&apiKey=" +self.apikey

result = urlfetch.fetch(request)
json = simplejson.loads(result.content)
return json

(btw, I had this clipped from somewhere but can't find the source, so if you know where it came from, please let me know and I'll add credit)

To use it write something like this:
bitly = BitLy(BITLY_LOGIN, BITLY_API_KEY)
shortUrl = bitly.shorten('www.yahoo.com')['results']['http://www.yahoo.com']['shortUrl']
If you don't want to use bit.ly, You can always quickly build and host your own url shortener on appengine (like this).

Part 4: Putting it all together

So far we've covered how to grab the feed, post to twitter, and shorten a url prior to posting. That+oauth is the bulk of what is necessary to understand how to build this simple web app.

To put it all together into a working appengine app is relatively simple once you have built an appengine project or two. If you haven't (or if you're impatient like me), it can help to have a good, simple, webapp appengine project template (for reference, mine is here) to get up an running quickly.

Wrapping up, I want to mention that most of this project was developed in a test driven manner using continuous testing to keep myself honest and verify and test behavior of external api's. I'm using Ale in conjunction with my project starter template for this.

Here's the source for FeederTweeter in a working appengine web app. It includes a simple (and horrible looking!) web ui that is set to be only accessible by the admin of the appengine account. There may be some slight differences from what is shown here -- usually for the sake of brevity in this blog post.

Friday, February 19, 2010

5 things you should know about developing for Google appengine

I've been building a few apps on google app engine over the last year (after spending 10 years doing java apps), and here are some things I noticed not many people are discussing when they mention appengine:
  1. It's pay for what you use, but it will probably be completely free for you. The free usage quotas are _generous_ and even when you reach them, the sophisticated billing google has set up allows you to 'budget' for particular resources. Even with a 'budget', _you only pay for what you use_ (rather than the typical monthly fixed fees in play with every other hosting provider).
  2. The google appengine team is working at a blazingly fast pace. In the last year, I've seen a flurry of releases with some of the most requested features and improvements (cursors to work past the 1000 results limit, auto-datastore retries, secure connections by default, better instrumentation/optimization tools, etc).
  3. Use Python unless you can't. The learning curve for the language is small and your web apps will be slimmer and more lightweight (as opposed to going the java route on app engine). There is a reason google hired Guido van Rossum and there is a reason he is working with the app engine teams. Python feels closer to the metal and anytime you're working with something that's constantly evolving, it is beneficial and efficient to be able to get to the things you need to without being roadblocked by too many layers of abstraction.
  4. Your apps deploy to appname.appspot.com by default. This is interesting for a few reasons: 1. It's free, 2. There are a limited number of names available, and 3. It means that to look like every other web app you'll most likely end up purchasing and linking a separate domain to your app. You can easily set up a custom domain+cname (i.e. - www.myapp.com) to use your app, but not yet a naked domain (myapp.com). Most apps are working around this by just setting up forwards/redirects with their domain hosting provider.
  5. I think this is going somewhere you may not have thought. Looking at the resources Google is putting behind app engine, and the way it lowers the cost of entry and the cost of infrastructure management for custom applications, I can tell you one thing: A behemoth like Google, that has already infiltrated enterprises with a 'drop-in' search appliance (after honing the inner workings with consumers), is not a far jump away from creating a 'drop-in' app hosting appliance/grid (after honing the inner workings with its consumers -- app developers).
btw, this post is a really elaborate excuse to test an app-engine based webapp I hacked together for myself to tweet out my new blog entries (example) automatically.

Wednesday, February 03, 2010

Angled linear gradients in sass/fancy-buttons

I'm so happy with how simple it is to make a nice looking button for a web-page with the combination of fancy-buttons and sass/compass that I just had to share:

Recently a design concept came back that included buttons (done in photoshop of course) in this style:



I had used fancy-buttons in the prototype of this project (www.coolchars.com). I was very happy that my html boiled down to simple 'button' tags and some fairly straight forward sass (which generates all the goobly-gook css). I was so happy with sass+fancy-buttons that I wanted to resist the urge to fall back to image based buttons for this angled gradient... so I did a little digging. It turns out you can do an angled gradient.

The following sass:
!lightgrey = #d8d8d8
!darkgrey = #adadad

button
+linear-gradient("left top", "right bottom", !lightgrey, !darkgrey, color_stop(65%, !lightgrey))
//...other styling irrelevant to the gradient example goes here...
can help produce a button that looks like:




The two keys to this are the directional cues ("left top", "right bottom") and the color_stop function. While we only used one gradient here, color_stop will let you do multiple color gradients stopping at different percentage points across a button if needed.

The only downside is that css generated for the angled linear gradient is pretty much webkit only right now. Firefox 3.6 has new angled gradient support and the author of fancy-buttons is currently working on adding FF support. Other browsers fallback to background color/image styling based on what you set up for your buttons in your sass files.

Detecting iPhone/iPod/iPad clients on Google App engine

Recently, while working on a fun little side project (www.coolchars.com), we needed to detect and render a very different page for iPhone users (no Flash and slightly different page structure to accommodate iPhone copy&paste)


When the iPhone (or iPod/iPad) requests a url, it includes a user-agent string like the following:


Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3


To detect and parse that from a webapp-based application on appengine, just do the following:


import os
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.ext.webapp import template

class MainPage(webapp.RequestHandler):
def get(self):
user_agent = self.request.headers.get('User-Agent', '').lower()
iphone = 'iphone' in user_agent or 'ipod' in user_agent

if iphone:
self.redirect('/index_iphone.html')
else:
self.redirect('/index.html')

application = webapp.WSGIApplication([('/', MainPage)], debug=True)

def main():
run_wsgi_app(application)

if __name__ == "__main__":
main()


Of course you can do something more sophisticated like parsing and rendering templates differently, but this is just an example.

A quick way to test it out is launch Safari, go to Develop->User Agent->Other, and pop-in the user-agent string at the top of this blog entry. Be sure to test it out on the simulator or the actual device since screen size will affect rendering as well.