Have you heard of IFTTT? It’s available at http://ifttt.com. Pronounced “ift” (like “lift” without the l), IFTTT is a free Web tool that uses channels to easily automate Web tasks. You can get a basic overview at https://ifttt.com/wtf but the premise is really simple — you choose a trigger (like a new item on an RSS feed, someone tagging you on Facebook, someone following you on Twitter, etc.) and in response to that trigger you can choose an action (automatically following a new Twitter follower page, sending Facebook-tagged photos of you to Dropbox, storing your Tweets in an Evernote account, etc.)
At first glance it looks simple and somewhat limited, because there are only so many triggers and actions. But as I spent a lot of time playing with it (I’m using it to automate a bunch of stuff at work) I realized that it could help me solve one of those annoyances that’s been bugging me for a long time, and that is keeping up with The Flickr Commons.
The Flickr Commons is a group of about five dozen institutions and repositories from all over the world that have come together to make some of their collections’ visual content available online without copyright. Group members include the New York Public Library, NASA, the National Archives of Norway, and the National Library of Scotland. So you can imagine there’s tons of great material there.
Unfortunately I couldn’t find a way to look at the latest Commons photographs in toto. I could look at individual institutions and follow them through an RSS feed; I could search Commons content; I could not find a way to look at the latest Commons stuff. I did not want to have to monitor 60-odd feeds. I wanted all the latest Commons content in one place.
IFTTT to the rescue!
IFTTT and RSS Feeds
IFTTT lets you pull content from RSS feeds as one of its triggers, which is probably what I do the most with it, as there are countless RSS feeds out there. Each institution participating in Flickr Commons has an RSS feed of the latest photographs added to its content.
I grabbed an RSS feed from one of the Flickr Commons members and started messing with it. Since an image thumbnail shows up in the feed, I tried grabbing the image and sending it any number of places, like Picasa and Dropbox. I wanted to make the photographs available publicly and I wanted to have an easy way to go to the original image if I saw something I liked and wanted to look at more closely (remember, the RSS feed has only a small image and not the full-sized photograph.) Picasa didn’t allow me to append enough information and Dropbox didn’t allow me to delineate the images enough.
So finally I ended up using Flickr itself — specifically, my own photostream.
Setting Up IFTTT
The IFTTT trigger/response sets are called recipes. So my recipe trigger was new content in one of the Flickr Commons institutional feeds. (I had to set up about 60 recipes, which was the most tedious part of this whole business.) If you want to play along at home and have an IFTTT account, I shared my recipe at https://ifttt.com/recipes/52593.
The action was to take the content from the institution’s feed and put it in my own Flickr photostream. But that wouldn’t be enough because there’s only so much good I’d get from a random image – I’d also want to know where it came from and where I could go to see larger versions of the image. So in addition to just moving the image over, the recipe also puts the source of the image and a link back to the original image in the description. There’s also an option to create new tags for each image as well — remember that because I’m going to come back to it later.
The Harvest on My Photostream
So I set up umpty-zillion recipes based on RSS feeds from Flickr Commons institutions let them run, and within a day I started having images automatically post to my Flickr photostream at http://www.flickr.com/photos/taracal/.
The URL in the description is not clickable from the galley page, but it is clickable on the individual picture’s page.
So what do I have now? Now I have a constantly-growing group of photos from the Flickr commons as my very own photostream, but in addition I have an RSS feed of all the latest content posted to Flickr Commons (via my account’s RSS feed on Flickr.) And with IFTTT, I can take that feed and do something else with it. In this case, I set up IFTTT to send me an alert via the iOS notification Pushover whenever the RSS feed updated. This came in handy when a picture of Queen Elizabeth came through on my iPhone and I was able to immediately text it to my anglophile friend Dee.
I had no hesitation in setting up these RSS feeds of visual content to aggregate on my own photostream because the Flickr Commons is just that — a Commons — and violating copyright was not a concern. Besides, I made sure that each description sourced the original image and linked back to it, trying to ensure that nobody thinks I’m the creator/keeper of these images.
If the aggregation of thumbnails, with clear attribution and links back to original content, could be considered fair use, I would really like to go further with this. There are so many institutions using Flickr. If you do just a simple people search for State Library you’ll find all kinds of goodies.
With IFTTT you could take the RSS feeds of the institutions in which you’re most interested and start a flow of thumbnails to your own Flickr stream, but more than that, you could give all pictures from that group of institutions the same tag and start creating your very own repository.
For example, I could go through Flickr’s people search and find North Carolina organizations — the NC State Archives, the Museum of Natural Sciences, the North Carolina State Library for the Blind, etc. I could set each of these up with an IFTTT recipe to send new content to my photostream, and tag each item as it’s added with not only the photo’s description but also with a unique tag of my own — maybe NCGROUPRB (something that probably isn’t replicated elsewhere on Flickr.) Then I just let it run. What I’m doing here is creating my very own Flickr subset from lots of different sources, in this case photographs from North Carolina organizations and institutions. (You could do this with any other topic you can imagine that can be found in the people search — state fairs, national museums, or even cooking schools!) When searching this collection, I could use incredibly general search queries (school, food, etc.) along with my unique tag and have success in finding images relevant to my context because I had narrowed down the searched pool of images in advance via the IFTTT image aggregation.
This setup isn’t perfect — IFTTT limits how much you can extract from a given RSS feed — but I’m having a lot of fun with my newly aggregated feed of Commons content and looking at a lot more pictures. If you find this useful and end up doing your own Flickr mini-content-curation project, let me know in the comments!
If I never update this blog again, blame Sunlight Labs. I read their latest blog post and now I can’t… stop… playing… with… Scout.
Scout, at https://scout.sunlightfoundation.com/, is an alerts service which gives you updates on federal and state legislation, as well as speeches in Congress and Federal regulations. Federal legislation I’ve found all sorts of tools for, but when I was poking around for a place to get state legislative updates last year, I had a heck of a time — it was pretty much hit and miss and seemed to depend a lot on what state you’re in.
Scout starts out looking like a search engine so I did a simple search for “solar power”. The results page lets your break down your search results into several sections: bills in Congress, speeches, in Congress, state bills, and federal regulations. Choosing one of these allows you to do a little more delineation; for example, choosing to look at bills in Congress lets you choose what stage they’re at (passed, vetoed, etc.) and choosing state bills allows you to specify particular states.
Information in the search results is minimal; looking at solar energy bills in Montana provides brief information on the three bills that were returned, but additional information and the full text of the bill is no more than a couple clicks away. Similarly, searching for the number of times the word goofy has been used in congressional speeches (apparently former senator Byron Dorgan likes that word a lot) provides a brief context of the speech in the search results but the original speech is only a click away, with additional clicks to the source and the original GPO transcription with all its Green Acres references intact.
So the searching is good but the alerting is great. To use alerts you’ll need to have an account (it’s free) and if you want to get SMS alerts you’ll have to verify your phone number (Scout sends you a text and you enter the verification code from the text.) To set up alerts just do your searches. For every search result you’ll see a blue “Create Alert” button above the search results. Click that to save an alert.
All your alerts will be gathered in one spot, and you can edit them there to specify whether you want your alerts by text, e-mail, or not at all.
I immediately set up several alerts for state legislation; hopefully this’ll be easier for me to keep up with what’s going on in my state than what I’m using now, which is a couple of push notifications and lots of manual review. Thanks, Scout!
In what observers (well, me anyway) are calling a long-overdue move, Twitter has announced (http://blog.twitter.com/2012/07/simpler-search.html) several enhancements to its search engine. One of them has me a little leery, one of them has me absolutely thrilled, and the other ones I’ll have to try. Here’s a rundown on what’s new.
Search Autocomplete: According to Twitter, autocomplete is supposed to make search suggestions for you as you type. Which sounds great (and I love to get an idea of what people are searching for) except I couldn’t get it to work. I tried two different browsers (Chromium and Firefox) and made sure I was logged in, and nada. Even typing things like Justin got me no auto-complete suggestions. Maybe it’s not available at the moment?
Spelling Corrections: This is the one I’m leery about; Google already “corrects” your spelling even when you’re properly spelling what you want! I did a search on Twitter for appple; Twitter gave me lots of results for apple but also included at least one result for appple; as long as what I actually searched for is still in there I’m fine with it. (Interesting trick: looking at the “Top” results gave me tweets for both appple and apple. Looking at “all” results gave me tweets only for appple. All = Verbatim?)
Results with Real Names and User Names: Here’s the way Twitter describes it: ” When you search for a name like ‘Jeremy Lin,’ you’ll see results mentioning that person’s real name and their Twitter account username.” That’s nice. But I also like that I can do a search for “Museum” and get Museum Twitter results at the top of the search page, something I don’t remember from before. It also worked for the phrase “state library,” which is going to come in handy because of…
Restricting results to people you follow: You can now get results just from people you follow. Which means I can do my searches for things like database and archive and collection and not get a pile of spam. And oh look! This search supports special syntax, so I can search people I follow for filter:links and just see what people I follow are pointing to without the chatter. Aahhhhhh. Now if could only get that sent to me once a day as a list….
Of course, with that in mind I need to find more great folks to follow. Got a suggestion? Leave a comment or send me an e-mail!
This weekend the first annual Hopscotch Music Festival is taking place in Raleigh. Over a hundred bands, parties, and general downtown rocking out for four days.
I work for one of the sponsors so I wanted to keep track of the festival, pictures of the events, news coverage, and so on. To do so I set up several information traps that I’ll use just through this weekend and then expire. It was an interesting exercise so I wanted to share with you what I did.
There’s already been plenty of news coverage so I knew there would more during the festival. Hopscotch is an unusual enough word that I was able to use the query hopscotch location:nc at Google News and get almost all relevant results. (Remember, the location: syntax restricts results to media within a specific state. I might miss a few items, but on the other hand I’ll get really targeted results.) I picked up the RSS feed and put it in my reader.
Finding relevant blog posts was a bit tougher. I tried Technorati but a search for hopscotch got only six results total and most of them were not relevant. Searching Google Blogs for hopscotch found a lot more content and a little spam; still, the results were clean enough that the result feed went in my reader.
I couldn’t find a search interface for Bloglines, and Blogdigger had no results at all. Icerocket had plenty of results but there was a serious relevance problem. Narrowing down my search to “Hopscotch Music (I didn’t want to add another word as I wasn’t sure if people would refer to it as “Fest” or “Festival”) brought me a good set of results, and I added that RSS feed to my reader.
Hopscotch has tags devoted to it and of course you can do some geographic searching with Twitter. But despite the fact that I can do some narrowing of my Twitter search, I did not want to trap for just the hashtag #hopscotch. If I did that, I would be flooded with a lot of less-useful content, like tweets for people who arrive at venues, or leave, or so forth. So I decided to trap for multimedia content.
Searching for #hopscotch yfrog and #hopscotch plixi and #hopscotch twitpic will clue me in to pictures taken at the festival and quickly put online. Those queries went into my RSS feed reader after I saw they were already producing good results even though the festival started just this afternoon. (I’m writing this Thursday night.) I’m testing another query, hopscotch http -yfrog -twitpic, to pick up all the tweets with links that aren’t pictures, but at the moment those are mostly Foursquare checkins.
(These traps are only going to be active for three days, so I probably won’t abandon any of the traps before the end of the festival. But if I were building these traps to keep them for a long period of time, I’d pay careful attention to what my RSS feeds were producing and quickly dump any that were providing spammy or useless results. I only have so much time to review what I’m picking up.)
Looking at the pictures on Twitter reminded me that Flickr might be getting images of Hopscotch as well. A test revealed that hopscotch was currently working okay as a search term, with lots of band pictures and only one irrelevant result. So that went in the feeder too. (For information on how to make keyword-based RSS feeds for Flickr, check out my article.)
That was Quick
Setting up this set of traps only took about twenty minutes. I skipped a lot — didn’t get into discussion forums, for example, didn’t try to trap Facebook, and didn’t expand my news story search beyond what Google offers. But I feel this’ll give me a good overview of what’s going on a feedback from a wide variety of attendees. I’ll try to do a followup article next week about what I found and what I’d do differently next time.
I got a question from @doctorwallin on Twitter. She asks:
“Thx for all your info. Do you know if there’s a way to get keyword news alerts via twitter, similar to google alerts by email?”
PEK Interactive notes another option in this article, while Mashable notes a slightly expanded way to monitor various types of replies and mentions on Twitter.
You can do a Twitter search at http://search.twitter.com/advanced (that’s the advanced page that gives you all the search options.) Once you’ve run a search, you’ll notice on the results page that you can get an RSS feed of your search results.
In January 2009 I wrote an article on effective information trapping with Twitter. Maybe it’ll be useful.
Thanks for the question! I hope this helps.
I have no idea if I’m going to do this regularly but it was easier to answer the question in an article than a 140-character tweet.
I read a recent blog post at Search Engine Roundtable noting that Google Alerts had had its algorithm tweaked and because of that fewer alerts had been going out. (There was also a pointer explaining what to do to loosen things a bit so more alerts go out.) I had noticed that I wasn’t getting as many alerts as I had been, but I was comfortable that I wasn’t missing too many stories.
As I thought about it though I realized that if you are using Google Alerts — and only Google Alerts — for keeping track of new stories and new resources, this change to the algorithm might have thrown for your a loop. So to make sure that doesn’t happen again, and potentially to give you some new ideas, here are six tips for making the most of alerts for information trapping purposes.
1. Allow for Overlap — When I wrote Information Trapping a few years ago, there was plenty on Google Alerts and Web search, but nothing on Facebook or Twitter. One thing I did mention, though, holds true for both of them: build in some overlap. If you’re using a set of tools that provide alerts for the same kind of resource — like, say, Google Alerts, Tracerlock, and Yahoo Alerts, you might be tempted to create unique, non-overlapping queries for each one.
Don’t do it. Instead, duplicate your queries with the idea that you’re going to have a 10-20% overlap in the results you get back, and that you will be doing some duplicate reading. That way if you do lose a resource, or it gets tweaked, you’re not going to miss much. Evaluating your volume of overlap might tip you to when a story or resource is getting an extra-large amount of play.
2. The Right Tool for the Right Job — Google Alerts provides alerts by e-mail, but you don’t have to get all your alerts by e-mail. In fact, you may feel a little overwhelmed if you do. Instead, you might want to take the most critical alerts you’re following — the ones where you want to know immediately — and get them by e-mail or even text message. Items of secondary importance, e-mail or RSS feed. And items of lesser importance, perhaps just RSS feed. I monitor at least a hundred queries via Google Alerts — but I also have several hundred RSS feeds in my feed reader. The tools are complementary. (I have never understood the “e-mail alerts vs. RSS” controversy. Isn’t “both” a legitimate answer?)
3. Constantly Reevaluate — It’s easy to imagine that Google has always been the biggest search engine, that RSS feeds have always been around, etc. But that’s wrong. AltaVista was the dominant search engine for a long time, and RSS didn’t become popular until well after 2000 (I’m thinking maybe 2003 or 2004.) Stay aware of the resources you’re using. You may find that over time the alerts you’re getting are becoming less useful, or that the resource is going off in a particular direction, or even that it goes defunct. There’s always going to be a certain amount of churn in Web sites and alert services; be prepared for it and don’t be afraid to switch, or at least to try new resources. (Otherwise you might find yourself still using Feedster and DayPop!)
4. Expand Your Horizons — I am a confirmed text-crawler, but now more than ever the Web is about multimedia. So don’t confine yourself to Google Alerts, Twitter, or other text-based alert systems when you want to keep up! You can get alerts or keyword-based RSS feeds from YouTube, Flickr, Slideshare, and other non-text Web sites. I have an RSS feed for Flickr photos tagged flowchart and it gets me some crazy stuff, but not in overwhelming amounts. If I tried to monitor Google Alerts for the word flowchart I’d be buried in results.
5. Remember there are People on the Other Side of the Screen — You’re not going to get alerted to everything in your sphere of interest. You’re just not; you can’t keep up. Don’t feel bad; this has been true since about 1994. But you might get a little more information if people know what you’re interested in. I have plenty of people who send me terrific sites that I had not heard of, and I try to return the favor when I know what someone’s interested in. So if you find a great resource that you think someone you know would like, pass it on. And maybe you’ll get that back in good karma when someone sends you a link to a great new search engine, database, or whatever you’re interested in.
6. Don’t Be Afraid to Get General — If you have a very specific interest, say forensic accounting, it may be that you can articulate all the topics you want to follow via several well-crafted queries. And if you can that’s terrific and you’re doing a lot better than me. But it may be that your interests are pretty far ranging and you can’t put them all in a query. In that case, don’t try. Instead, monitor resources that are focused on your topic but which aren’t narrowed as far as specific keywords. Twitter lists (which you can find by the thousands at Listorious) are one example. Another would be Facebook “Like” pages — did you know they have RSS feeds? So do Facebook groups.
7. Enough is Enough — It would be easy to follow all these tips and create for yourself a huge firehose of information useful, relevant, and interesting to you. There’s only one problem: IT’S STILL A FIREHOSE. Having all that information flowing to you does you no good if you can’t ingest and use it. So don’t feel compelled to create alerts for every last site out there. Instead, focus on generating keyword-based RSS feeds and e-mail alerts that are as specific as possible, and if you feel yourself unable to keep up with alerts, cut back. It’s better to have 100 alerts and be able to fully read and use them, than to have 1000 that you barely look at because you’re constantly overwhelmed.
1) First, I ListiMonkey’d. ListiMonkey is a service that will e-mail you the contents of any Twitter list you specify. You can set up how often you want to get a list of tweets, and you can specify how often you want to receive them. You’ll receive up to 100 tweets per e-mail. You can do some preliminary filtering through ListiMonkey, though I found there was a limit to how many terms I could filter. Every e-mail I got (maybe 300-400 a day) went into a text file.
2) Next, I TextPipe’d. I took my one day’s worth of tweets from Twitter lists (a big text file) and fed it to a software program called TextPipe, which describes itself as an “industrial strength text transformation, conversion, cleansing and extraction workbench.” Using TextPipe I stripped out all the HTML, removed all duplicate lines (every tweet is on its own line), removed all lines that had cruft I didn’t want (filtering out two or three dozen keywords) and then output it to a nice, clean, much smaller text file.
3) Then, I TEA’d. Using the TEA Text Editor, I scanned through the list of remaining Tweets, deleting the tweet-lines I didn’t want to review further. After I was done with that I used TEA’s HTML tools to convert the list of leftover, “to be looked at further” tweets into an HTML document.
4) At this Point, I Converted. TEA can turn the list into an HTML file, but the problem remains that the links are unclickable. So my last step was to go to David Weinberger’s Convert URL’s to Hyperlinks utility and turn my basic HTML file into a basic HTML file with clickable URLs.
5) Finally, I Firefox’d. I opened this HTML file in Firefox and quickly opened and scanned through the tweets I had put aside for further review.
Going through these steps is going to let me review a lot of content from a lot of lists and save me a tremendous amount of time.
A few additional thoughts:
a) I can probably do this in Perl. I know, but I can experiment with and implement filters in TextPipe way faster than I can do it in Perl.
b) It’s not perfect. TextPipe doesn’t truly remove all the duplicates, as the same tweet can be posted three times with three different bit.ly URLs. To eliminate those I’ll have to do some spadework with regular expressions.
c) TextPipe is expensive. TextPipe Standard is $199. For the amount of time this will save me in trying to keep up with all the tweetstreams that capture my interest, it’ll pay for itself.
d) This solution will tempt me to subscribe to even more Twitter lists. THIS is the problem I’m going to have to watch out for….