How to track anything and everything in your lifestream

Db08 Comments

Usually, most lifestream services out there only allow you to track specific places and usually these places are only the most popular like facebook, flickr, del.icio.us etc. Furthermore they only give you limited control on the look.

This is mainly the reason why I went for my own hosted lifestream. It allows me to track anything I can get a feed for as well as modify it as much as I want to look the way I want it.

However there is always the problem of how to track things that do not provide a feed of some sort. Sure, you might be able to use co.mments to track you blog comments but what if you want to log the comments you left on a forum, in a bug tracker or in a private location? How about when you need to track something completely different than that, like an email of yours to a mailing list or a new torrent you’ve uploaded?

The answer is: Simple. Bookmarking.

All you need to do is utilize an online bookmarking service like del.icio.us or getboo to tag your actions and then use the tag feeds to import them into your lifestream. Not only then can you get a feed for the various actions of yours, but if you utilize extra tags, you can categorize it further afterwards, provided your lifestream services can play with categories.

For example, I am currently bookmarking all the comments I leave on the internet and tag them as “mycomments“. If you check these bookmarks you will also see that they are further tagged as “blogcomments”, “comment_start” etc. Once a comment is tagged, Getboo provides it as a tag feed which I import into my google reader lifestream (for archiving purposes). Finally, my lifestream takes that and depending on the original tag feed the item comes from, it formats it appropriately in the list. Thus you will see all the comments use the same icon & colour while the FSD actions another. However they are both in the same service (getboo) and in the same feed (Google reader lifestream).

Not only that, but if you look deeper you will see that even items from the same source have further differences. Under the FSD class you can see items marked as “Voted Up”, “Buried” or “Commented on”. They are all however coming from the same original getboo tag feed and this is where the tags/categories of a bookmarking service come especially handy. Using the amazing capabilities of SimplePie and some newfound php knowledge I’ve cooked up a method to query the categories of each lifestream item and further format them based on what I find.

At this point, (with only free software tools mind you: getboo, wordpress, simplepie and simplelife) I have a lifestream that is much more inclusive than, I believe, any other option out there, and the capabilities I have are truly endless.

Currently, I am planning to getting logs for mailing lists, bug fixings, wiki edits and whatever else I do on the intertubes. Once everything is ready, I just need to wrap everything up and give it to you as an enhanced WordPress plugin 😉

PS: I suggest getboo throughout this article because del.icio.us is already used by many other lifestreams out there already. If you start bookmarking your actions there, you’ll get them double unless you unregister del.icio.us totally it from your service. That will mean however that your normal bookmarks won’t be tracked anymore. By using an alternative, you avoid that fate and get the best of both worlds.

Oh, and did I mention that if you really have to, you can also host your own getboo? 😉

So what are you waiting for. Start tagging 😉

Candles and Smoke

Db0
Disclaimer: This is mostly a test to play with my wpg2 enabled lightbox plugin as well as display photos rather than a normal blogpost 😉
2875

So last Sunday, as always me an Vio visited Sam’s place as has become the normal for the group in the last few weeks. I don’t usually joing as I just can’t understand shit of what they are saying in German. It’s not just that they talk fast, but they tend to just eat half of the sentences while replacing the other half with slang I am not familiar with. Meh!

2863

This is actually the reason why I do not join such excursions that often. I just know that I’ll not be able to take part in the discussion, will miss all the jokes and generally, other than being in the presence of friends, miss the rest.

2884

Thus it needed a specific incentive to get me to join, and this was Munchkin. I was actually hoping to play a game or two this time since it’s been a while since last I did. So I took my handy bag (which is doing a disintegration act due to the sheer weight of the cards) and my new shiny camera, and headed there.

2896

Unfortunately it didn’t happen after all. It pretty much evolved as I assumed, with everyone talking except me 🙁 Fortunately I had hugs and a nice gadget to keep me occupied. I wasted my time playing with the camera settings and taking pictures of whatever. I still haven’t managed to discover how to take nice long-exposure photos (Too many options of photo stuff confuse me), but I’m getting there.

2893

In the end, I ended up falling asleep in Viola’s knees for, from what they tell me, 1.5 hour. I woke up late enough that nobody really was interested in playing Munchkin anymore.

Oh well. At least I got to show you all my new photos 😀

Using a Google Reader as a lifestream archive

Db05 Comments

After I set up my lifestream to track a host of custom actions (like comments on other blogs, or votes on FSD) I discovered that many of the feeds I was using were having a item limit (from 10 to 20). That meant that if I made, say, 11 or more comments in various other blogs, the older ones would dissapear from the list, even if the date was still in the limit I’ve set. Only the newer 10 would always be shown and it would seem as If I made no comments below that date.

It was not a bigg issue but I really wanted it to work right so as to get a more accurate overall idea of where I am spending my time (graphs and so).

I remembered then, that Google Reader seems to always have the option to go back into older posts in the feed, up to the time that you made the subscription through it. This meant that it archives everything in that feed and stores it in its own server (Damn that Google storage is endless). Also, Google Reader has the option to put various subscriptions under a specific folder/tag, allowing you to read all the subscriptions together.

It also allows you to make that tag public. A Public tag allows anyone to see the feeds that you share through it but also gives you a feed to subscribe in turn. This feed, unlike the original ones, can provide feeds data as far back as the time you made the Google Reader subscription (and even further if someone else had subscribed to it before you, I believe). All I needed to to is append ?n=# at the end of my google reader feed where # is the number of items I want to see each time the feed is polled. (I selected 100)

So there I had it. An easy way to make an arvhive of my tracked activities without setting up my own database.

The only problem now was to make the lifestream plugin recognise from which feed in turn each item comes and then assign it the correct class. This was a more challenging question as my previous method to compare the original feed with the current item’s could not work. I discovered that I could filter by the items target link but since I wanted to use the same service to track different activites, it would always filter them the same way. In case you’re wondering, I use Getboo to bookmark pages that I commented on as well as FSD articles I’ve voted up, down or commented (Since, unfortunately, FSD does not provide such feeds by itself until now). The number of things I track through Getboo will only increase as I find more actions that don’t provide feeds.

Thus I needed some way to the original feed through my Google Reader lifestream feed.

Well, how very fortunate of me that Google Reader publishes an Atom feed and not only that but it includes the source of the original feed for your convenience. 😀

A little trial & error with the (truly excellent) SimplePie APIs, and I’d managed to find and write the code that I needed to use.


if ($feedurl == $greader_lifestream_feed) {
if ($source = $item->get_source()) {$source_url = $source->get_permalink();}
if (stripos($source_url, 'twitter') !== false) {
$class = 'twitter';
$tf_counter = $tf_counter+1;
$current_title = 'Status Update (Twitter)';
}
if (stripos($source_url, 'mycomments') !== false) {
$class = 'comments';
$extra_text = 'Commented on: ';
$getboo_comment_feed_counter = $getboo_comment_feed_counter+1;
$current_title = 'Comments Elsewhere';
}
if (stripos($source_url, 'fsd') !== false) {
$class = 'fsd';
$extra_text = 'Voted up: ';
$fsd_counter = $fsd_counter+1;
$current_title = 'Votes up on Free Software Daily Article';
}
}

And yes. I’m proud of my little code snippet. 😀

So now, if I have any feed with a sufficiently low item limit, all I need to do is subscribe to it through Google Reader and add the appropriate if statement in my plugin code and presto! Instant archiving. There is only a (very) slight problem in the sense that the lifestream update is delayed by the time it takes for google reader to refresh the original feed and then update the public tag.

All I need to do now is figure out a way to separate items in the google reader feed through the category tags that getboo provides. That would be sweet as it would cut down on line number but unfortunately the get_category() simplepie function does not seem to work as intended for me 🙁

How to hunt for WordPress performance hogs

Db03 Comments

So after my previous post on how I discovered my major causes of wordpress pain, I kept looking on what is causing my other delays. Eventually, while looking at my source code, I discovered that my wordpress page was always reporting how many SQL queries it took and how long they took to complete. Looking at the footer, this was done by this command:

<?php echo get_num_queries(); ?> queries. <?php timer_stop(1); ?> seconds

Now, I understood that this was very helpful but I couldn’t understand why it was returning a time like 10 seconds when the page was loading for 4-5. A quick google search for this code however did lead me to a page that was explaining how to Keep WordPress overhead down. Through there, I realized that those seconds were some sort of CPU time taken to draw those queries.

Through this then, I now had a semi-accurate way to monitor the impact of each plugin on my site. I only needed to figure out which parts of the site were hogging down the queries and increasing the load time.

I created a quick table in a Spreadsheet and started logging down the differences.

  • First, I made the performance text visible, so that I don’t have to look at the source code each time (You can still see it by scrolling to the bottom of the page)
  • I disabled the WP-Cache so that the queries are done every time
  • I then started disabling each plugin in turn and reloading a page. Then I compared that with the previous loading time (where the plugin was enabled.) I noted down the difference in queries and the average/approximate CPU time difference.

Unfortunately since the CPU time tended to vary from load to load, I couldn’t get an accurate number of the difference. The table I made in the end looked something like this

Plugin name Queries CPU Time Taken
Bad Behaviour 1 5
Hide Text 0 Negligible
Greeklish 0 Negligible
One click install 0 Negligible
SimplePie WordPress Plugin 4 Negligible
Security Scan 1 1

After I went through all the plugins, then I started doing the same thing with my various widgets and snippets of code in my theme that might be causing this.

Once this was done, I had one likely suspect plugin and a few widgets that were probably contributing to this slowdown. You see, my theme, a very heavily hacked HemingwayEx, was using some custom widgets in order to draw from WordPress the latest comments, recent posts and whatnot. These widgets queried the database each and every time a page would load which was quite frankly impractical.

Fortunately a short time before, I had discovered the very useful bundle of post plugins. They were also there to hunt for recent comments & posts and related posts and they also included a built-in caching mechanism! As sweet as it gets.

I quickly modified my widgets to run this code instead and the results were wonderful. By disabling also the single other plugin I found that was slowing things down, Bad Behaviour, I’ve now managed to drop the loading time considerably (or so it seems for the time being). Not only that, but I’ve managed to put a few new toys to use that do a better job than HemingwayEx’s built in functionalities.

Currently my site seems to load lightning fast compared to before but I’m not absolutely convinced this will not change once my shared hosting environment gets bogged down. Hopefully, even if that happens I will still have a much much better performance than before.

I am still stuck at around 65-80 queries per page load but I can’t figure out where they come from. I am hoping that I’ll eventually be able to trace it but for now it seems enough.

Btw, I just loved the functionality of the Post Plugins to measure their own execution time and report it back to you. This is something that all plugins should be doing in order to give the web designer an idea on what is going on with his site. I’m going to work to see if I can implement that in the Lifestream plugin I’m working on and then see if I can place it on other plugin as well.

Btw (2), Can anyone explain why more people don’t take advantage of the Plugin cache plugin? I am fairly certain that a number of heavy plugins (like Popularity contest) could make good use of it.

As always, if you have any more ideas on where to look and tweak to improve my WP performance even more, I’ll be glad to hear them (AKA Whining because no one is commenting on my blog :P).

PS: It’s a pity that I had to disable Bad Behaviour as it worked quite well until now. I also don’t like to reply too much on just a single anti-spam plugin. Unfortunately, the performance hit was undeniable and I’d rather my site doesn’t take an extra 3-4 seconds to load every time.

Squashing Wodrpess Performance Bugs

Db02 Comments

I think I’ve finally managed to trace down and smite all the things that were making my site so Gawd awfully slow. I had finally been annoyed enough by 30 sec loading times that I just had to do something about it.

Initially I thought it was with my host, and I did contact support about it via Email. They did check out the MySql slow logs and noticed some strange queries from BadBehaviour, but when this little plugin blocks around 1000 bots per week, it’s not worthwhile to disable it :).

In Dreamhosts’s opinion I should try and disable my plugins and then switch to another theme to check. Well, even though it was painful, I did it but it didn’t really make much of a difference. My site still seemed to be slow. However a test site I created had a quite normal speed. Perplexing

Initially I thought that it might just be the difference in DB size and indeed, my other WP sites, which have much less content that this, seemed to be loading faster (albeit slightly).

Fortunately a few google searches led me to some good content. Initally I discovered Firebug from a post explaining how it can help you diagnose WordPress performance issues. That was quite helpful to tell the truth. I managed to immediately spot that the WordPress Automatic Upgrade (WPAU) and the Popularity Contest were severely slowing down my Dashboard. I also found a few items in my theme that were just taking too long to load (like my FSF badge for some reason). Disabling them dropped the load time but I still had a problem.

You see, for some reason there is a 5-9 second loading time from the time I click on a link, to the time the page refreshes and starts to load. Firebug, unfortunately, will not give you any information on what is causing this, other than to tell you that the link is loading. It will not even take into account this initial loading time when giving you the total load time of the page (thus a page taking 13 secs will be reported as taking 4.5). Since this loading time dissapeared

This meant that I had no idea what was causing it, so back to the search I went. This time I managed to find a more interesting post on Improving WordPress performance which not only gave some great info, it linked to some other sources. Unfortunately, most of the tips given were to be used on a private server and were not really applicable to a shared hosting service like mine. Of course, this person was talking about receiving 500.000 hits a day, whereas I’m barely receiving 3000…a month. Still I applied the wp_config.php edit although it didn’t seem to make much of a difference.

My final hint came from wolfie who advised that I remove all widgets and links that point to other addresses as these may increase the loading time. I did that and I also took down the Popularity contest as well, just to see the results.

It seems it worked. The initial loading time has dropped to around 4-5 seconds. Although I don’t have as much bling in my site anymore, I am hoping that the increase in speed will make up for it. Unfortunately 4-5 secs is still quite bad and while I still don’t have any idea where to look in order to fix that, it’s at least better than 6-9 secs + 4 secs of extra stuff. With the WP-Cached page, the load seems to be instantanteous OTOH so at least that helps. Still for a ~100 hits per day, it certainly should not happen.

As an aftereffect, I decided to finally update this site to WP 2.5 as the functionality of Popularity contest was the only thing preventing it. At least I now have a snazzier Dashboard 🙂

After improving my performance I decided to also start a WordPress Performance page on the dreamhost service wiki. I would urge all of you, Dreamhost customers or not, to take a look and improve this page so that others that have similar issues can find a focal point to troubleshoot. I also hope to see tips specific for Dreamhost shared hosting customers eventually.

If there is a good WordPress improvement page or tips that I should check btw, feel free to let me know. Also, if you have any idea what causes these ~4 secs of initial loading delay, I would be really glad to hear it.

Charted life

Db0

I’m not the one to pat himself on the back for the smallest thing, but I must say that I feed quite proud about my latest little hack in my lifestream.

I was initially inspired by yahooza’s lifestream (via the lifestream blog) after I saw that little chart on the right side. He used the google charts API to achieve which seems quite easy to use and very impressive for online use. Thus I digged in my Simplelife plugin and started hacking around.

You can see the result below. For my limited php knowledge, this was quite a achievement 🙂

My personal pie chart

Of course the above is a static image, but within my lifestream page, this is updated dynamically every time the page loads. I like 😀

Of course, this will soon be available to you all since the Simplelife v1.4 is coming out soon and the author would like to implement some of my ideas in it.

Now all I need to do is to find a way to get the dynamic image in a format that I can put in other places without having to run the script along with it 😀

Where're my damn icons?

Db0

I don’t get it. My lifestream icons are not loading for some reason even though you will see in the page source that they are loaded as part of the class. For some reason, initially the last.fm and facebook icons stopped loading after I hacked the SimpleLife plugin to make it play with all kinds of feeds. Then, I discovered why the colours of the custom feeds were not being used (a missing colon) and fixed it, and now <a>all</a> my icons are gone…

Grrr!

Nothing I do seems to fix this and I just cannot see why it is happening. There does not seem to be an obvious error in the code and the rest of the css class (text, background colours) are loading just fine. In the source of the page, you can see the css clearly loading correclty


a.lastfm {
background: 0c0c0c url(http://dbzer0.com/wp-content/plugins/simplelife/lastfm.png) no-repeat 10px 50% !important;
border-top: 1px solid 0c0c0c !important;
border-bottom: 1px solid 0c0c0c !important;
color: #BFBFBF !important;
}

And then the item call

<li class=”item”><a href=”http://www.last.fm/music/Machinae+Supremacy/_/Stand” class=”lastfm” title=”db0’s Recently Played Tracks”><span class=”timesf”>09:48</span> Machinae Supremacy – Stand</a></li>

And the icon exists…
Anyone have any idea what is going on?

At least now I can have feeds from any source (feedburner, del.icio.us, google reader) without losing formatting. I just need to wait for the next plugin version to add options for more streams 🙂

UPDATE: Nevermind. I found the culprit. Apparently I’m blind…

Mighty plugin hacking

Db02 Comments

Well…for my newbie standards at least 😛

I’ve been playing around with the SimpleLife WordPress plugin, trying to create a simple lifestream I can embed in my about page and perhaps in my sidebar as well. It turned out that this was a job which needed a bit more attempt to make it work.

Below are the changes I did in order to wrangle it.

  • Had to edit the plugin in order to get it to diplay correctly in the plugins page. For some reason it floods the screen with it’s contents. Apparently editing and saving should solve it but in my case I needed to delete some newlines as well.
  • Changed the classes .date and .time to .datesf and .timesf and modified where they were called. Leaving them as they were, they were screwing with other css classes on my site under the same name
  • Changed the if statement for last.fm to look for the string ‘last.fm‘ instead of ‘last‘ since that string may be a part of any other url.
  • Put three new if statements, similar to the ones that check for last.fm and facebook, that check if the link goes to this blog, the ACP or the Wesnoth Journals. Then set the class according to that. The way the plugin is setup, it does not assign the class depending on the feed url bur rather according to the current link url. This means that you cannot use feedburner as your feeds address as it will never trigger an if statement[1]. I actully find the way this is handled a bit weird as it would serve much better to check the feed url and assign a class, rather than check the current link. I’ll have to check if that’s actually feasible…
  • Commented away the if statements for the first three feeds. I’m using the custom ones instead now.
  • Manually set the colour of the last two feeds within the plugin. For some reason even though the php call takes the variable I’ve set in the plugin configuration page (I can see it in the source code of the page) the colour is not being used and the text stays black. Weirdly enough, the background colour is changed well enough and the “blog feed” works just fine.
  • I didn’t want all my delicious feeds to be posted because I seem to be doing an awful lot of them. I prefer to log only the ones that signify my comments for which I use the mycomments tag. Fortunately I’ve discovered that you can use a del.icio.us tag by using the form: username/tagname
  • Created a new page template and inserted the php code there.

All-in-all, the plugin seems very promising and it already seems to work fine for me. However it does seem quite error-prone and lots of people most likely won’t be able to use it yet. Fortunately this will be fixed in the short future.

Now I just have to see if I can add more stuff for it to track.