Category: Web Technology

New additions to my back up plan

I’ve been using Dropbox, Carbon Copy Cloner, iCloud, iTunes Match, Time Machine, Google Drive and Crashplan as part of my back up plan for a while. Recently I also added OneDrive and Copy, because they both give me a decent amount of cloud storage, and in the case of Copy it’s a decent interface to store large PDF files (such as downloads of Mac Format and Linux Format) that are too large for the Kindle app to deal with.

Today I also started experimenting with BitTorrent Sync. It’s not cloud storage, but it will mirror directories between multiple computers, and is only limited by disk space and the speed of my local network. I’ve not got much further than installing the client yet, but will write more about how it works if it proves to be interesting/useful.

What’s your backup plan?

This week at work we have been working on a video to promote backing up data. The tagline is “what’s your backup plan?” – which has made me think about how I back up my data, and how well what I actually do measures up to what we recommend.

The basic message is that for a file to be backed up, it needs to exist in an identical version in more than one location (and ideally three locations, one of which is physically separate from the actual machine the data is created on). I do try and adhere to this, although I think I’m still a step away from being as safe as I’d like.

I have two basic backup strategies. One is to ensure that any file I edit exists in some sort of cloud storage system (usually Dropbox, iCloud or Evernote). The other is to ensure that any computer I create data on is backed up regularly using at least two different methods/products. The combination of these two systems, plus the fact I use quite a few computers, ensures I always have several copies of everything, and can access historical copies of my data and bootable clones of my whole computer in almost all scenarios.

Most of this is now automated, in that all my machines back up locally through Time Machine on an hourly basis, and once a day to a bootable clone created using Carbon Copy Cloner. This works fine providing my house doesn’t burn down. I also back up my main home and work computers once a week to a disk that I keep with my at all times, but this has to be done manually, which isn’t ideal.

My iOS devices back up to iCloud, but also back up to my computer every time they are plugged in (with the backups then being themselves backed up as part of my other backups). I don’t have any unique data on them (at least not for long), but I still think it’s worth being able to restore them quickly and to have a second (and sometimes third) copy of all my apps.

So that’s my backup plan. It’s not perfect, but it covers most of the bases.

Easter Project : Setting up a new NAS

My project for this Easter was to set up some sort of storage solution for the vast array of music, films, TV shows and photos I have, and also to organise and catalogue them more effectively.

I asked for a few suggestions on Twitter, and there were a couple I liked the look of. The one that nearly won was the HP micro server, but I eventually settled for a Synology NAS device with a 4TB WD Green hard drive, which I’d read good things about, and which would also stream all my media to my iPad with minimal configuration. I still plan on getting some sort of Linux server at some point, but I think that’s a different project for another day.

Setting up the device was easy. Anyone reading this could do it, and there are plenty of guides on the internet. What was harder was coming up with a sensible way of organising my data, and making sure that I wasn’t just copying the horrifically complicated file structure from my old NAS without rationalising if it was still the best way of doing things.

Sorting the photos was fairly easy, once I’d tracked down where they were all stored. I created a directory, and then made a folder for each event, in a “year – event” format. That way everything will list in vague chronological order and I’ll remember what year things like weddings and holidays happened.

Films and TV shows was trickier, because I wanted to be able to access these more often, and in a way that allowed me to watch them easily. In the end I went with Synology’s default file structure, and made separate directories for films and TV shows. I then put each TV show in a separate folder, and made several folders for the films (by director if there were a lot by the same director, and then catch-all folders for English language films and those with subtitles). Copying the data was an overnight job, but it all went smoothly.

I then explored ways of accessing films and TV shows on other devices. Firstly I downloaded Synology’s iPad app, which is really pretty, and tries to find metadata and cover art for everything. When it works, it works really well, but it failed on a few of my more obscure films (especially the foreign ones), and whilst it played them fine, it gave them incorrect titles, which I think might annoy me in the long term. I also had to specifically point it at folders to index – it wasn’t quite intelligent enough to know where things were on its own.

I then fired up VLC, which I already use for offline playing of films when I’m travelling. It saw the NAS straight away, allowed me to browse the whole device, and pretty much just worked in the way that VLC has always worked for me. I’m not sure why I didn’t try this first, but it’s nice to see that software I’ve been using for 10 years still does the job I need it to do.

I already have all my music backed up in two places, so it wasn’t a huge priority to move it to this device straight away. But seeing as I had the space, I figured making a copy of my iTunes library wouldn’t hurt, so I at least have a snapshot of some of my music on this device. At some point I need to go through my old NAS and make sure there isn’t anything there that I don’t have in iTunes Match, but that is a job for another day.

As well as the music, I made a one-time backup of historical email and files. Both of these usually live in the cloud, but again I thought a local copy wouldn’t hurt.

Talking of backups. I also decided that now I have this new device I can probably afford the space to do Time Machine backups of my two laptops. Both currently back up to an external USB drive, which it reliant on remembering to plug the drive in, and so I am currently setting Time Machine up on both laptops with a view to being able to use the USB drives for something else soon.

I’m really pleased with the new NAS, and it has a lot of features I’ve not explored yet (like being able to run WordPress and Mediawiki and all the things I wanted a Linux server for). It’s taken a day and a half to set up, but the vast majority of that was copying data. The actual setup took minutes, and there was nothing that required being technical, using the command line, or understanding too much about networking.

And I still have half of the Easter break left to do other things, which is an added bonus.

Productivity

I’m generally regarded as someone who is quite organised and productive, which still baffles me from time to time because I don’t think I’m that organised at all. Most of what I’m writing about here seems fairly instinctive to me, but in the hope that it might help someone ready this then I’ll try and outline how I organise my day, and how I maintain at least a demeanour of getting things done.

My first rule is to get up when my alarm goes off, and to automate as much of my early morning routine as possible. I do the same things each morning, so it shouldn’t require much thought at all, and generally it doesn’t. I can be up and out of the house in about 20 minutes, as long as there is nothing to disrupt my routine (one of my cats bringing me a gift is the usual suspect for that). I then have a 25 minute walk at the start of my commute, and I tend to use that to listen to music and think about the challenges of the day ahead. This is followed by a 25 minute train journey, during which I read either the Metro or whatever book I’m currently reading on my iPad. By the time I get to work I’m wide awake, and focused on the day ahead. I then have a cup of coffee and start work.

A few years ago I did an exercise where I recorded everything I did for a week, and tried to match time slots to specific sorts of task. I’ve repeated this regularly for a few years now, and I have a fairly good idea of how to plan my day to get the best out of the time and energy I have. Solitary tasks such as writing, answering email, and tasks that require technical focus get done first thing in the morning while I’m wide awake and the office is quiet. I then put aside two slots for meetings – a morning slot for collaborative work, ideas generation, and meetings where I need to contribute a lot, and then an afternoon slot for meetings where I need to be present, but am not one of the main contributors. The rest of my day I work though my todo list, and my email inbox (both of which which I like to keep as close to zero as possible).

I also automate as much of my working week as possible. I have set weekly meetings with my manager, my co-worker, my team, and my direct reports. I also have set monthly meetings with a variety of other people and teams. All of my meetings are recorded in Google Calendar, and the agendas appear in Evernote 15 minutes prior to the meeting starting, thanks to the magic of IFTTT. I then make notes in Evernote on my iPad, and move any action points to my todo list as soon as the meeting finishes. Minutes are then archived to a workbook, so that my Evernote inbox contains only my todo list and things I am actually working on at that moment.

I used Google Calendar to organise everything I do (and everything I plan to do), and go back afterwards to ensure that how I spent my time is accurately recorded. This allows me to track how much time I spend on tasks, and how my work and personal schedule change over time. I also colour code everything, and have separate calendars for work and my life outside of work, which I strongly recommended as a compartmentalising exercise if nothing else.

I’m not a great fan of clutter, although anyone who has seen the inside of my study might debate that fact. I like to keep a clear desktop (physical and virtual), and I’ve been largely paper-free since November 2013, which has helped both with the reduction of clutter and with general productivity.

My Setup – 2014

I’ve not done a post about my setup for a couple of years, and as a few things have changed I thought it was worth an update.

Hardware

Right now I seem to be a big believer in a two computer setup, but it’s not always the same two computers, because I split my time between two offices at the University plus my own study at home.

On the left hand side of each desk I work at, I have a large monitor. This could be plugged into a computer, waiting for a laptop, or plugged into my iPad and used as a TV.

On the right hand side I have a laptop, or a space for a laptop. My general philosophy regarding laptops is that it’s hard to beat a Mac with maxed out memory and an SSD drive, and as such I hardly use anything else now. I think it’s quite possible to get decent speed and performance out of a fairly old machine, as long as that machine is configured correctly for what it is being used for, and as long as I have access to one powerful computer for occasionally processing digital media then there is nothing else I do that requires me to have bleeding edge hardware.

I’m also currently using iPads a lot more than I thought I would. I have one for accessing work email and making notes at meetings, and another (smaller) one for carrying around with me at all times and acting as a portable media and internet machine. Since I started using iPads, I’ve found that my laptops hardly ever leave my desk, and I do toy with the idea of a setup that consists of one powerful desktop machine and an iPad.

Software

All but one of my regularly used desktops/laptops are now Macs running OS X, although I do maintain several VMs running Linux (Ubuntu and Debian) and Windows. I sometimes need to test applications on every single OS/browser combination, and I’m actually not sure how I managed to do this sort of work before I started using VirtualBox.

I use three browsers on an everyday basis. Safari on my iOS devices, Firefox for work, and Chrome for personal web browsing. Firefox works better with some of the web applications the University use, and I like to compartmentalise data from the various areas of my life anyway. I also have one machine that runs the development versions of all three browsers, which I largely use for testing purposes.

I also use a wide variety of other software, most of which I’ve mentioned in pervious posts. The big changes are that I use Evernote for a lot of things now (which deserves a separate post), and I’m also increasingly managing my work email through Good for Enterprise on my iPad, which makes Inbox Zero achievable rather than just being a pipe dream.

Backups

As far as backup goes, any machine that stays in one place (or mainly stays in one place like my heaviest laptop) backs up nightly (via Carbon Copy Cloner) to an external hard drive. I also have a portable hard drive that I back up to weekly with a bootable copy of the two machines where I regularly create data (as opposed to consume it). When I’m not backing up to it, this drive is kept in a different physical location to the machines it is backing up. Additionally, all my music is in iTunes Match, my photos are in iCloud, my work laptop backs up to another machine via Crashplan, and everything text based I’m currently working on will exist in either Evernote, Dropbox or Google Drive, depending on what it is and who else needs to access it.

I test my backups monthly (sometimes more than monthly), including booting all the full disk clones to make sure they actually boot. I think this is important.

My dream setup

I think I am probably fairly close. I would like some machines I use to be newer, lighter, or faster, but on the whole I think I am satisfied right now apart from wanting to put an SSD drive into my Mac Mini, which I plan on doing very soon. Of course, that doesn’t stop me looking wistfully at the 13” Macbook Pro with Retina Display and the new iPad Air, but I very much plan on waiting a few months until I buy anything else.

Previous versions of this post

July 2010 – http://teknostatik.co.uk/2010/07/18/my-first-stab-at-self-interview/
March 2012 – http://teknostatik.co.uk/2012/03/18/what-im-using-to-get-the-job-done-in-2012/

A few thoughts about Evernote

I thought I should start using Evernote, at least to see what the feature set was actually like. I must say I am quite impressed, especially as there are quite a few other useful products that plug into it. So far I have taken and annoyed photographs, collected a few recipes, and even handwritten some notes in my horrible and very much unpractised scrawl. All of these things sync to all my devices, and can be viewed easily and via an interface that is aesthetically pleasing.

I should probably investigate these things further.

The start of something new

Today I started using my new iPad mini to do as much of my writing as possible. I wanted to see if it could replace my laptop for short non-work trips, and was interested to see if writing on this device would seem natural, or if the user interface would get in the way of my creativity.

So far I have found it a pleasant experience. I can’t type quite as quickly yet, but I am sure that will come in time, and it already feels quite natural to type on the built in keyboard. I would certainly consider using this as my only device on short trips, and whilst it is hardly a great effort to carry around my MacBook Air, it would also be nice to travel supremely light on occasion, and the iPad mini fits the bill perfectly.

Other things I’ve been impressed with so far are the streaming capacities when I am connected to my home network. I can stream music and movies to and from the iPad, and can also use it to get music from Amazon cloud player to my Apple TV. Web browsing is also a delight, and I think this device could easily replace my laptop for the sort of casual browsing that I generally engage in after a busy day at work.

There are still things I need a laptop for, but I can see myself using this iPad a lot over the next few months – especially as I am travelling so much.

Backing up Gmail with Gmvault

This weekend I have been experimenting with Gmvault (http://gmvault.org/) in order to back up my various Gmail and Google Apps accounts to my computer. I’m using Mac OS X, but almost all of this will work with Linux too.

Firstly I downloaded the software, extracted it, and ran the following command once to download all of the mail in each account:

./gmvault synch example1@gmail.com

I then wrote a script to automate the process:

#!/bin/bash
#change to the correct directory
cd /Applications/gmvault/bin/
#run a quick sync on all my gmail and google apps accounts
./gmvault sync -m -t quick example1@teknostatik.org
./gmvault sync -m -t quick example1@gmail.com
./gmvault sync -m -t quick example2@gmail.com
./gmvault sync -m -t quick example3@gmail.com

Once that was working, I automated it with cron to run a few times a day.

Restoring the email to another Gmail account is a slow process, and you should probably only do it a few times a month (and always overnight). Again I’ve scripted this bit, but have commented everything out unless I actually need it. Having done the initial upload, and because I now have two local copies of everything, I’ll probably only run this one monthly.

#back up all downloaded email to a dedicated gmail account in the cloud
#./gmvault restore backup_account@gmail.com
#or just for the last month
#./gmvault restore -t quick backup_account@gmail.com

Further reading:

Green Impact

(originally posted as http://greentechteam.org/site/green-impact-at-the-it-service-desk)

My department have done so much to promote sustainability and green issues, but in this article I am going to concentrate on the work done by my team – the IT Service Desk.

In 2012 IT Services were awarded a Green Impact Gold Award for the first time. As part of that initiative we looked at the way we work and made some changes. We cut down on paper, explored virtualisation technologies, and set up a green board in the office to make everyone aware of environmental issues and how each person could contribute.

In 2013 we went a little further, and started a more proactive approach to bringing down the carbon footprint of the team, and raising awareness of the role each of us has to play in building a sustainable future.

Initiatives we undertook in 2013 include:

Putting together a whole workstation by re-using and scrounging furniture and IT equipment. We have used a lot of the old kit from Aston Web C-block that was going to be binned, and have actually saved a fair bit of money by doing things this way. People often forget that reusing things is generally better than recycling them, and this initiative nicely demonstrates that. Also, we did it all in about two hours, and carried all the furniture ourselves rather than using a van.

Along similar lines, we have also just refurbished another office using furniture from C-block, and cascaded our old furniture to other parts of the library.

We have a new Green Board, in the corridor outside our office. We have all the usual things, plus a Green Ideas Tree. Students can write suggestions on it, and we can pass these on to the relevant people periodically. We also have a poster detailing iPhone and Android apps to do with sustainability and environmental concerns (including QR codes so people can download them).

For years we have re-used old PCs as servers, test machines, and as a way to have access to as many different OS/browser combinations as possible for testing purposes. This year we measured the power consumption of these older machines and found they were using significantly more electricity that the other computers in the office. As a result of this we now use virtual machines for anything that doesn’t involve running something on specific hardware, which has cut down massively on power consumption, as well as making the office feel a lot less cluttered.

And finally, we’re also trying to raise awareness with our staff. Just little things like asking them to justify having a second monitor, making sure PCs and printers are switched off when not being used, and trying to avoid using fans, heaters, and anything else that consumes a lot of power. We also use an online Knowledge Base as our primary way of disseminating information to students, which cuts down on the amount of paper we use.

Hopefully what we have done so far has made a difference, but we already have plans for the next twelve months to build on this good work and hopefully aim for a Gold Plus award this time next year.