Techie blurbs

Mass download flickr albums

2011-08-26T04:14:00.000-07:00

Is quite simple using the flickr API:

import java.io.IOException;



import javax.xml.parsers.ParserConfigurationException;



import com.aetrion.flickr.Flickr;

import com.aetrion.flickr.REST;

import com.aetrion.flickr.photos.Photo;

import com.aetrion.flickr.photos.PhotoList;

import com.aetrion.flickr.photosets.PhotosetsInterface;



public class Backup {

    private Flickr flickr = null;



    public Backup() throws IOException, ParserConfigurationException {

        this.flickr = new Flickr("Put flickr api key here", "Put flickr api secret here", new REST());

    }



    public void doBackup(String photoSetId) throws Exception {

        PhotosetsInterface pi = flickr.getPhotosetsInterface();

        PhotoList pl = pi.getPhotos(photoSetId, 0, 0);

        for (Object po: pl) {

            Photo p = (Photo)po;

            System.out.println(p.getOriginalUrl());

        }

    }



    public static void main(String[] args) throws Exception {

        Backup bf = new Backup();

        bf.doBackup(args[0]);

    }

}

The above code will print the URLs of the original images in the album whose identifier you have to pass as command-line argument. And you'll need the flickrapi library for this. And you have to request a flickr API key+secret to have access to flickr's API (google on how to do this).

If you made this into a nice software, share it as a comment!

The dark side of Android

2011-01-27T06:00:00.000-08:00

I have had my gmail account since 2006 (about-ish). Back in the days when it was in invite-only beta-phase. Used it ever since.

For christmas 2010, I've got a nice shiny Android phone (htc desire). 3 days later, as I check my email via the web GUI, the 'last accessed from' text that most don't even notice ever had a dark red background, with 'WARNING' word. Clicking on it, this is what I could see:

Coincidence? Could be, but too little chance to practically consider. More likely that some shady app from the market (yes, I've only installed from the market, and never ever gave my password to anybody).

What is even more surprising is the IP! 255/8 is reserved unallocated, so I wonder how google could be so stupid to even accept routing advertisments for any prefix below 255/8!

I'm almost certain this has something to do with registering my android phone with Google, using my good old reliable google account. I'm also considering that google uses 255/8 internally (maybe only for registering android phones?), and they screwed something up with the access alerts. I don't know and I can't know as google doesn't support non-premium users.

So here it goes. I can't draw any conclusions, neither can anyone else, but maybe it will help someone.

Wordpress widget: Count all

2010-06-20T06:24:00.000-07:00

About

I've been playing around with wordpress 3 to replace blogger.com, which was getting worse and worse over the years. Wordpress turned out to be quite nice, I was positively surprised to see what a great thing it has evolved to.

It has extensive widget support as well. But when I was looking for a baby age counter, I found 2, neither one of which worked. I think it was caused by their age; I've found a reference to wordpress 2.3, and the code was also looking quite horribly written. I'm not a php developer myself, but having extensive experience with other platforms, I can cope with it, and just by looking at it I could tell it was Bad Code (tm).

So I decided to write one. PHP didn't seem too hard. I've started off by googling for "wordpress widget tutorial", and found this great tutorial by Jesse Altman. It was basically a skeleton HelloWorld widget, well-written, well-explained. Without much further ado (except the PHP reference doc), I was able to create a baby counter widget.

Install

It's a single .php file; just drop it into the wp-content/plugins directory of your wordpress (or use the admin interface to install it directly).

Use

After activating, you can use the "Count all" widget under the 'Appearance' menu.

The values are self-explaining, but for reference:

Title is the widget's title. It can contain HTML; its value is simply embedded into the resulting HTML page.
Content is the widget content field. It uses the DateInterval::format function to format the field contents. You can use any field specifiers from that function. Its contents are also embedded into the resulting HTML page, so over the %-tags, you are free to use any HTML you want.
Date specifies the date we should count distance from. Since this plugin simply takes the difference between 'now' and the specified date, it can be used for counting down (for dates in the future), or counting up (for dates in the past). It's all up to you.

That's it. You can have multiple instances of this widget, of course.

Requirements

This plugin needs Wordpress 2.8+ (tested to work on 2.8.4 and 3.0), and PHP 5.3 (for the DateInterval class).

Download

Here

How to replace FileVault with EncFS

2010-02-16T14:00:00.000-08:00

UPDATE: I've been upgraded to Snow Leopard (10.6), and a lot of issues outlined below are solved. Recovering disk space takes a couple of seconds only, IO speed increased considerably, and backup also seems to take a lot less time. I would recommend you NOT to follow the guide below for Snow Leopard and up if these were the only items bugging you.

Background

MacOSX comes with a built-in encrypted filesystem: FileVault. Its purpose is to protect private data stored on your computer. It does its job quite nicely, but far from perfect. Me, coming from the linux world, was quite put off by its limitations, so I started looking for an alternative that would make me happy.

FileVault is an encrypted filesystem. This means that is doesn't encrypt files, it encrypts partitions. How it works is when you hit the button 'Turn on FileVault' in the security preferences, it creates a dynamically growing file in your home that contains the encrypted contents of your original home. When you log in, this is mounted, using your login password. The whole operation is seamless. In the background, it is relying on the diskimages framework. If interested, check man hdiutil for details and more options.

In earlier releases of FileVault, the whole encrypted home was a single huge file on the filesystem. Whenever growth was necessary, it increased the encrypted file size. However, this caused problems for Time Machine, Apple's built-in backup tool. Since the file is encrypted, it was not possible to make incremental backups, so Apple decided to implement a feature called 'sparsebundle', that splits this huge file in 8MB chunks, so that Time Machine can make incremental backups of the encrypted home.

You can see how problems stack up. It's not only FileVault, but how it works with other applications, most importantly Time Machine.

This HOWTO was written on a Mac OS X 10.5. Other versions may or may not work - you'll have to figure it out yourself. The idea and the commands are the exact same once you get encfs installed properly.

FileVault: pros and cons

- Long waiting times at logout
- No shrinking while logged in
- Doesn't work well with Time Vault
- Proprietary

+ Well worked out and tested
+ Encrypts you entire home for better privacy

Why

Get your space back

For example, if you have a 300GB disk, and you copy 50GB into your home temporarily, MacOS will expand the sparsebundle to accommodate the extra 50GBs. However, after you've removed it, it will not shrink it, and it will be recovered only after you log out. Until then, 50GBs will be gone from your 300GB disk, and only be available in your encrypted home dir.

EncFS is a file-based encryption, and it relies on the existing file system for storage, so this issue simply does not exist; the encrypted and decrypted directory share the same partition, and thus, disk space.

Get rid of the long waiting times at logout

FileVault is very slow. In the before example, it takes FileVault about 20-30 minutes to recover the disk space it has allocated to the encrypted filesystem earlier. This of course is done when you are logging out, waiting for the laptop to reboot, shut down, etc..., so basically, from your own time.

EncFS stores its encrypted files in the same partition, and thus, it has no such problems whatsoever.

Back your data up while logged in

With FileVault, there are no backups made while you're logged in. This is again caused by the bad design of filesystem-in-a-filesystem. Data is backed up when you logout - but then it takes quite a while.

Take for example the case when you copy 100GB in your homedir, and then erase 50GB. On logout, FileVault will take 20-30 minutes to recover disk space, and then Time Machine will take another 20-30 minutes to back the extra data up.

Using EncFS, backups are done on-the-fly, while you are logged in. This means you can stay logged in as long as you like, there is no need to log out just to make a backup of your homedir, like you had to do with FileVault.

Be flexible

EncFS is fully open source. You know what's going on in the background. You can fine-tune the encryption to your needs. You can even have more than just your home encrypted. You can use different encryptions on different parts of your system (a 128bit for your pictures and a 256 bit for your emails, for example). You don't have to encrypt your home, only the parts that you need. It's all up to you.

You can also have better interoperability with other tools. Since it is fully file-based, you have the ability to process the encrypted directory to your liking, in your own way.

Why not

FileVault is a lot more tested than EncFS. There are known problems with EncFS, as it only support basic POSIX operations (no locking, extended attributes, etc...). This works well for simple file storage or multiplatform applications, like MacPorts, Firefox, Thunderbird, etc..., but encrypting your whole homedir is known not to work. As a consequence, you are losing some privacy, leaving parts of your home unencrypted. If you are sensitive about this, or you are unsure whether you won't leave important data uncrypted, don't go ahead.

Because EncFS encrypts on file level, the number and size of files will be exposed in the encrypted dir. This is not a big deal, just go check out the contents of the encrypted directory for yourself. You'll probably keep wondering what the heck is in there, even though you know what you are storing there.

You also need to spend some time to evaluate what and how you want to protect. Setting up EncFS will take some time, it's not just clicking a button and going for lunch while it finishes.

If you are not a geek, don't do this - it's not meant for you. I would say the baseline is setting up and using macports - if you can do that, this will not be a problem, either.

How - the easy way

You can use macfusion to set up encfs via GUI. I won't go into details, as I observe this method to be inferior to command line.

It is a good tool for non-geeks to encrypt their data, though, if you don't mind some click-click now and then.

How - the geek way

Once you have macports installed, set up encfs:


$ sudo port install encfs

The next step is to organize your mounts. How I did it was I've created /e and /e.crypt and chown'd it for my user:


$ sudo mkdir /e /e.crypt
$ sudo chown llama /e /e.crypt

Of course, you can create multiple directories for multiple users if needed.

The next step is to set up the encrypted directory. This can be done by:


$ encfs /e.crypt /e

On the first run, encfs will guide you through the setup options. If unsure, just pick the default options (by simply hitting enter) for every question, they are well defined.

Now you have your encrypted directory in /e.crypt and your decrypted directory in /e. The next step is to decide what you want to encrypt. It's fully up to you. I went with the configuration directory of ssh, Thunderbird and Firefox.

Don't forget that encfs doesn't support fancy filesystem operations, so don't just throw your whole homedir in there - it won't work.

The next step is to copy your stuff there and create a softlink on the original location:


$ mv .ssh /e
$ ln -s /e/.ssh
$ mv Library/Thunderbird /e
$ ln -s /e/Thunderbird Library/
$ ...

So, now you have the final setup. The next step is to make mounting/unmounting automatic on logon/logoff. For this, we are going to use the so-called login hook. This is a special Apple feature; in short, it lets you define an executable that is executed after login, but before changing UID to the user - so, practically at the point when all login-related tasks have been done by root, and we are about to change UID to the user. We will use this to mount our encrypted directory.

Now, we want an automatic mount of the encrypted volume on login. This is handy if you have Firefox and Thunderbird started up on login.

How I've solved this was putting the password in the user's keyring. That's a safe storage for passwords. So open the Keychain Access application (easiest is from the little lock in the menubar), pick the Login keychain, category Passwords, and add a new key in there. Set the keychain item name and account name to 'EncFS', and the password to the password that you given to encfs during the first-time initialization.

Now let's create the script that is to be executed as login hook. Create file /usr/local/sbin/loginhook, and copy the following into it:


#!/bin/bash
# $1 = logged in username
# try to mount /e using encfs entry in keychain
su - $1 -c "security find-generic-password -ga EncFS" 2>&1 >/dev/null | cut -d'"' -f2 | sudo -u $1 /opt/local/bin/encfs -S /e.crypt /e

What it does is grabs the password you've added to the login keychain using the command-line tool 'security', and pipes it into encfs executable, using the '-S' flag to tell it to fetch the password from standard input.

Next we create the logout hook. It is the exact same as loginhook, but executed at logout. So create the file /usr/local/sbin/logouthook, and copy this into it:


#!/bin/bash
# $1 = logged out username
# try to unmount /e
umount /e

Yes, it simply unmounts the encrypted directory on logout.

Don't forget to set the executable bit on these scripts:


sudo chmod +x /usr/local/sbin/log*hook

Now, instruct the login window application where the login and logout hooks are:


sudo defaults write com.apple.loginwindow LoginHook /usr/local/sbin/loginhook
sudo defaults write com.apple.loginwindow LogoutHook /usr/local/sbin/logouthook

That's it, you're done.

If you would need to use encfs for more than one users, it is a pretty straightforward change to the above - remember, the first argument of the login and logout hooks is the username. Also, remember to store the 'EncFS' key in the login keychain, because that is specific to each user, so each of them can have different password.

There are a lot more things that you can do with this, just use your imagination. For me, this does the job, and is perfectly extensible would the need for anything else arise.

Conclusion

The biggest mistake Apple did with FileVault is storing the encrypted home directory on a virtual file system. All of FileVault's drawbacks originate from this. The implementation is brilliant, free of bugs, fast and well thought over. But why they decided to have all the trouble with a filesystem in a filesystem remains a mystery.

For me, until Apple manages to come up an encryption that is not based on a broken idea, EncFS really gets the job done. Leave a comment if this short howto helped you too.

How to convert WBZ files to JPG (free!)

2009-02-17T05:42:00.000-08:00

Webshots has beautiful pics, but their format is a horrid one, titled 'WBZ'. The interesting part is that it actually contains an encrypted JPEG file inside, along with some extra info.

To extract the JPEG file from the WBZ, you can use the following python script. It takes multiple arguments and saves the JPEG files according to their webshots name. Good news is, it doesn't even require any extra package - python core does it.

It is not my code - it was part of a GUI software. I merely ripped the GUI and other useless parts of that code. I don't remember what it was - sorry.

Anyway, here it is. Enjoy!

http://pastebin.com/f77e4e93c

How to run firefox3 on ubuntu edgy, feisty or dapper

2008-09-02T02:17:00.000-07:00

OK, I've tried hard to find a way to run firefox3 on these ancient systems. Trying to search the net didn't help much, sadly.

This walkthrough will probably work on all old(er) linuxes, where you don't/can't upgrade just to get firefox3. The main issue is firefox3's dependency on a newer version of gtk, so we will compile one for ourselves and then set LD_LIBRARY_PATH to use our compiled libraries.

So, here it goes:

1. Download firefox3 from getfirefox.com.
2. Download source of atk at ubuntu.com.
3. Download source of cairo AND pixman at cairo's website.
4. Download source of gtk+, glib and pango at GTK's website.

Save all sources in your home dir.
Now, on to compiling (you can use any other path as destination, I'll use ~/firefox):


mkdir -p ~/firefox/src
export LD_LIBRARY_PATH=~/firefox/lib
export LDFLAGS=-L${HOME}/firefox/lib
export CPPFLAGS=-I${HOME}/firefox/include
export PKG_CONFIG_PATH=${HOME}/firefox/lib/pkgconfig/
export CFLAGS=-I${HOME}/firefox/include
cd ~/firefox
tar xvjf ~/firefox-3.0.1.tar.bz2 (or whatever version you've downloaded) (this will also create ~/firefox/firefox)
cd ~/firefox/src

Untar all source tarballs you've downloaded above - they will all create their own directories.
Now you've got atk, cairo, glib, gtk+, pango, pixman in ~/firefox/src.

Compile them each by executing:


cd (package dir)
./configure --prefix=$HOME/firefox
make
make install

The order of package compilation matters, the right order is: glib, pixman, cairo, pango, atk, gtk+.

If you're done, you can execute firefox like this:


LD_LIBRARY_PATH=~/firefox/lib ~/firefox/firefox/firefox

Of course you can organise the directories better if you wish - but for a quick, working firefox3 in your homedir the above is just fine.

Have fun!