Thursday, February 27, 2014

BBEdit/TextWrangler Codeless Language Module for IDL


UPDATE April 2015: I've now updated the CLM for IDL vesion 8.4, for use with BBEdit 11.x. You can download it here.

For nearly a decade I've been using IDL, a programming language that's good for visualising lots of data. Recent versions ship with an IDE based on Eclipse, which is either horrible or great depending on whether you're an advocate of the Eclipse environment. I'm not.
Call me old fashioned, but I like to code in a simple text editor, and debug from a terminal. I work on Mac OS X, and my choice of text editor is BBEdit or TextWrangler, both from BareBones. TextWrangler is free and should be on every Mac, and BBEdit costs money but has some great additional features. I've tried to find a text editor as useful on Windows, but there just isn't one.
A few years ago I wrote a Codeless Language Module (CLM) for BBEdit (and TextWrangler), so that it can apply syntax highlighting to IDL source code. The CLM is a .plist file, which is a flavour of XML that contains keys described in BareBones' documentation that allow lots of cool stuff to happen. This includes:
  • syntax highlighting (comments, quotes, etc)
  • keyword highlighting (and predefined names)
  • autocompletion of function names
  • function scanning (makes function names apear in the function pop-up menu for easy navigation)
  • automatic recognition of language based on filename extension
  • code folding
Here's what it looks like:


The IDL Configuration.plist file is provided here for you to download. There are some comments at the top of the file about how it was generated - mainly for my own memory so I can regenerate it when a new version if IDL comes out. It refers to some custom functions I wrote to extract information from the IDL documentation and help files, and generally make life easier. If you'd like to get hold of these functions let me know.

Using CTAGS with IDL and BBEdit/TextWrangler

As an added bonus, we can also use Exuberant Ctags to scan IDL code files and generate 'tags' files. These are used by BBEdit for autocompletion. The use of ctags is well documented in the BBEdit manual, so I won't go into detail here, except to say that they're great, and I have written a routine to generate a 'tags' file for IDL codefiles, which you can download here. You also need to have the appropriate stuff in ~/.ctags for ctags to scan your *.pro files correctly. These are the lines that do the magic:
--langdef=IDL
--langmap=IDL:.pro
--regex-IDL=/(^[ \t]*(pro|function)[ \t$]+)([A-Z_a-z0-9:]+).*$/\3/d,definition/i
If you have any problems or suggestions, please let me know.

Connecting to an SMB share...

How to connect to an SMB share...

with spaces in its name...

from Mac OS X using the Terminal or a bash script

This is something I needed to do in order to back up some Windows systems with rsnapshot, a python script that uses rsync. Getting it to back up Windows systems over the network requires that the Windows computers share the directory or directories that need to be backed up, and the Mac OS X box mounts the shares for the backup to run. There may be other ways, but this works well for us at the moment. The only issue has been working out how to mount Windows shares with spaces in their names from a bash script, so the backup could be scheduled to run automatic and unattended.

It took a while to figure out. Googling didn't really help much - there were plenty of suggestions but nothing actually worked. These ranged from quoting the space with backslashes, replacing the quote with its octal representation (\040), using double-quotes. In the end, many people suggest that renaming the share to get rid of the space is the way to go.

In the past this is what I'd done: altered the name of the Windows share to remove the spaces, but in the back of my mind I knew it should be possible. Finally I got around to working it out properly...

The following are known to work in Mac OS X 10.9 with shares from Windows 7, XP and Vista. There are two steps to mount the share.

Create the mount point

This is pretty straightforward. If this is done in the bash shell the spaces can be quoted with a backslash, or the entire name can be double-quoted. The mount point can be created anywhere in the file system, but if it's in /Volumes the mounted volume will be available in the Finder.

mkdir /Volumes/Share\ Point

or

mkdir /Volumes/"Share Point"

Mount the share

This is done at the bash prompt or in a script using the mount command. You must specify the username and password, the address of the server, the name of the share and the mount point.

The path to the mount point is fairly simple. In typical bash fashion it can be specified with double quotes or the space could be preceded with a backslash - either will work. The real trick is getting the path to the SMB share exactly right. My first (failed) attempts were to use double quotes as follows:

> mount -t smbfs "//username:password@ip_number/Share Point" "/Volumes/Share Point"
mount_smbfs: URL parsing failed, please correct the URL and try again: Invalid argument

which clearly did not work. Next try was escaping the space with a backslash:

> mount -t smbfs //username:password@ip_number/Share\ Point "/Volumes/Share Point"
mount_smbfs: URL parsing failed, please correct the URL and try again: Invalid argument

which also failed. Similarly, using the octal representation of the space:

> mount -t smbfs //username:password@ip_number/Share\040Point "/Volumes/Share Point"
mount_smbfs: mount error: /Volumes/Bruker Backup: Unknown error: -1073741412

this gave an exciting new error message, but still failed :(

Finally, I was successful when I replaced the space with %20 its URL-encoded representation:

mount -t smbfs //username:password@ip_number/Share%20Point /Volumes/Share\ Point/

Phew! The share will now be available in /Volumes, and also visible in the Finder. When you're finished with it and it's no longer needed, it can be unmounted in the Terminal with

umount /Volumes/Share\ Point

or by Ejecting it in the Finder.

Getting it to work in a bash script

I use a function within a bash script to do the mounting of smb shares. I pass in the username and password, the server name and the share name and the function does the heavy lifting. If the share name to be mounted has spaces in its name, the spaces must be replaced with %20 in the function call. Within the function the share name is used with the %20 in the call to mount, and the %20 is replaced with a space and the share name is double-quoted when used to create the mount point, specify the mount point in the mount call, and to unmount the share when it's no longer needed. A proof of concept script would look something like:

#!/usr/bin/env bash

IP="192.168.0.1"
USERNAME="username"
PASSWD="password"
SHARE_NAME="Share%20Point"

# bash can replace the %20 with a space...
MOUNT_POINT="/Volumes/${SHARE_NAME//\%20/\ }"

mkdir "${MOUNT_POINT}"

mount -t smbfs "//${USERNAME}:${PASSWD}@${IP}/${SHARE_NAME}" "${MOUNT_POINT}"

# do something useful with the share at this point

umount "${MOUNT_POINT}"

Saturday, May 25, 2013

encoding video for topfield pvr

Sometimes we use our Topfield PVR to play back video from 'other' sources. What I mean by this is video that it didn't record itself, which might be video from the web (eg. youtube), or recordings made on a linux box with a DVB card. The PVR is a TF 7100 HD PVRt Plus.

The manual for the PVR is a little vague on the subject. It says: "You can enjoy video files in DivX, vob, mkv, or mp4 format on the digital receiver." Not really any mention of codecs or container formats, and nothing on audio encoding. The unit has a USB port on the front (and another on the back) where an external hard drive can be attached, but according to the manual: "To copy files to the external hard drive, it must be formatted through the Topfield in the JFS file system. After this, the external hard drive can only be used with the Topfield PVR." Bad news for anyone not running linux (even heard of JFS??), but after some experimenting, here's what I've found actually works...

Video Encoding

Video formatted for iOS seems to work. I have a bash script that does the encoding using ffmpeg, but anything that can produce mp4 formatted video for iOS should be able to produce video that the Topfield can display. I experimented with DivX avi files as suggested by the manual, but as far as I know aspect ratio is not encoded into avi files, so the video was display in the incorrect aspect ratio. It would be possible to correct this during playback in the PVR, but I found that mp4 files displayed with the correct aspect ratio with no adjustment necessary.

Disk Filesystem

I use a USB stick, formatted in EXT2 format. The Topfield box reads this OK, using the USB port on the front. To write the video files to the disk you either need access to a linux box, or a linux virtual machine. I use a VM running Debian, on Mac OS X. It's a little slow, but it works. This should work just the same with an external USB hard drive, although I've not tried it.

Permissions

Last point, which seems to be crucial, is that the Topfield can only see folders and files that have root for the owner/group. It may not be necessary to have *both* owner and group set to root, but this is what I've found works. For example, here are the permissions on a demo video file supplied on the PVR:

---------- 1 root    root    65337872 Feb  6  2009 Alterna_Compilation.divx
Very limited, but the Topfield seems to like it. Hope it works for you.

For those that are still reading...

Here's ffmpeg's info on the video file supplied with the Topfield PVR. Make of is what you will:


~ > sudo ffmpeg -i /Users/foo/Alterna_Compilation.divx 
[mpeg4 @ 0x7fc78a029600] Invalid and inefficient vfw-avi packed B frames detected
Input #0, avi, from '/Users/foo/Alterna_Compilation.divx':
  Duration: 00:02:05.46, start: 0.000000, bitrate: 4166 kb/s
    Stream #0:0: Video: mpeg4 (DX50 / 0x30355844), yuv420p, 720x400 [SAR 1:1 DAR 9:5], 23.98 fps, 23.98 tbr, 23.98 tbn, 30k tbc
    Metadata:
      title           : Video 
    Stream #0:1: Audio: mp3 (U[0][0][0] / 0x0055), 48000 Hz, stereo, s16p, 160 kb/s
    Metadata:
      title           : Audio 
At least one output file must be specified

Tuesday, January 10, 2012

a ruby wrapper for 'svn switch'

A while ago our local network admins decided static IP numbers were a Bad Thing, causing chaos for me when trying to use a machine as a server for subversion (among other things).

I found myself trying to check changes in subversion repositories back in to the server, only to find that the server was no longer at the IP address from which the source had been checked out. The answer (apart from attacking the network support people) is the svn switch command. This is a multi-purpose tool with lots of options, so I wrote a wrapper to simplify the process of changing the repository's root URL and nothing else. It was also an excuse to learn a bit of ruby, which is way cool.

#!/usr/bin/env ruby

    new_server_address= ARGV[0]
    fail "Specify the new server name or IP number" unless new_server_address

    svn_info = %x{svn info}
    fail "'.' is not a working copy" if svn_info == ""

    url_regex = Regexp.new('^URL: (.*$)')
    repository_url = url_regex.match(svn_info)[1].to_s
    fail "Unable to find the URL" unless repository_url

    split_repository_url = repository_url.split('/')
    split_repository_url[2] = new_server_address
    new_repository_url = split_repository_url.join('/')
    puts "Switching from \n#{repository_url} \nto \n#{new_repository_url}"
    puts "Continue? [Y]"
    confirm = STDIN.gets.chomp
    if confirm.upcase == "Y" then
        system("svn switch --relocate #{repository_url} #{new_repository_url}")
    end

Subversion setup

Start by editing /etc/apache2/httpd.conf so that the httpd-subversion.conf file gets included.

Include /private/etc/apache2/extra/httpd-subversion.conf

There's no httpd-subversion.conf file by default, so we need to make one. On my system, it needs to contain the following:

LoadModule dav_svn_module libexec/apache2/mod_dav_svn.so
LoadModule authz_svn_module libexec/apache2/mod_authz_svn.so

<Location /svnrepos>
    DAV svn
    SVNParentPath /usr/local/svnrepos
    # the following svnindex.xsl (and svnindex.css) are located in the
    # server DocumentRoot dir, which is defined in httpd.conf with the following line:
    # DocumentRoot "/Library/WebServer/Documents"
    SVNIndexXSLT "/svnindex.xsl"
    # require SSL connection for password protection
    SSLRequireSSL

    # user authentication
    AuthType Basic
    AuthName "Subversion Repository"
    AuthUserFile /etc/apache2/subversion.auth

    # only authenticated users may access the repository
    Require valid-user
</Location>

The .xsl and .css files were copied from my old server. They're just to make browsing the subversion repo more nice.

The subversion.auth file is generated using htpasswd as follows, and stored in /etc/apache

sudo htpasswd -cm /etc/apache/subversion.auth my_user_name

After the command is issued, a password must be entered (twice).

Test the install by trying to access a subversion repository (presumably in /usr/local/svnrepos, according to the above config). You should be required to use a https connection, and be prompted for your username and password.

Generate SSL keys on Mac OS X Lion

I've done this successfully using a script I wrote a while back. One of the gotchas seems to be that the "Common Name" should be the server name (or static IP number if you don't have one). This must match the ServerName directive in httpd.conf.

#! /bin/bash
echo; echo -e "Generating Certificate Authority (CA) [1]:"
openssl genrsa -des3 -out ca.key 4096

echo; echo -e "Generating Certificate Authority (CA) [2]:"
openssl req -new -x509 -days 365 -key ca.key -out ca.crt

echo; echo -e "Generating Server Key:"
openssl genrsa -des3 -out server.key 4096

echo; echo -e "Generating certificate signing request (CSR):"
echo -e "       ** name entered for Common Name (CN) should match your server name **"
openssl req -new -key server.key -out server.csr

echo; echo -e "Signing CSR with CA:"
openssl x509 -req -days 365 -in server.csr -CA ca.crt \
        -CAkey ca.key -set_serial 01 -out server.crt

echo; echo -e "Making insecure version of server.key for apache startup"
openssl rsa -in server.key -out server.key.insecure

echo; echo -e "Renaming server secure/insecure keys"
mv server.key server.key.secure
mv server.key.insecure server.key

sudo cp server.key /etc/apache2/server.key
sudo cp server.crt /etc/apache2/server.crt

Still seems to work...

In order to get apache to use the ssl certificates it's necessary to change httpd.conf

First uncomment the LoadModule line to get ssl_module to load:

LoadModule ssl_module libexec/apache2/mod_ssl.so

Uncomment the line in httpd.conf to get the httpd-ssl.conf file loaded:

Include /private/etc/apache2/extra/httpd-ssl.conf

Now make some changes to httpd-ssl.conf. This file is in the extra/ directory.

Set the ServerName correctly (and the ServerAdmin email address, if you like)

ServerName name.of.your.server:443
ServerAdmin email.address@nowhere.com

Set the SSLCertificateFile and SSLCertificateKeyFile to the correct file paths:

SSLCertificateFile "/private/etc/apache2/server.crt"
SSLCertificateKeyFile "/private/etc/apache2/server.key"

And that should be it. Try loading https://localhost in a web browser. Check the logs if there's a problem.

Moving redmine to apache

This post follows the last one about installing or migrating redmine and mysql. Picking up from the point where redmine was working through WEBrick...

A couple of extra ruby gems are required to get apache to serve redmine to the world. Passenger does most of the hard work, and when it installs it provides it's own config instructions. Neato!

sudo gem install passenger
sudo passenger-install-apache2-module

The last command is really cool, it does lots of work compiling the apache module. I guess it's something from the rails community. The instructions tell us to add some lines to the apache configuration file. I'm putting them in /etc/apache2/other/passenger.conf. After I'm done, the passenger.conf file looks like this:

# settings from passenger-install-apache2-module
LoadModule passenger_module/Library/Ruby/Gems/1.8/gems/passenger-3.0.11/ext/apache2/mod_passenger.so
PassengerRoot /Library/Ruby/Gems/1.8/gems/passenger-3.0.11
PassengerRuby /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby

# additional settings
PassengerUserSwitching on
PassengerUser _www
PassengerGroup _www

Set user:group on /etc/apache2/conf/extra/passenger.conf

sudo chown _www:_www /etc/apache2/conf/extra/passenger.conf

And get apache to load the passenger.conf file by adding the following line to httpd.conf

Include /private/etc/apache2/other/*.conf

Redmine runs on apache using a virtual host. Include the httpd-vhosts.conf file by uncommenting the relevant line in httpd.conf:

# Include /private/etc/apache2/extra/httpd-vhosts.conf

Then edit the httpd-vhosts.conf file to get things working. Comment out the example configs (anything referring to *dummy-host*), and add the following config sections:

<VirtualHost *:80>
        DocumentRoot "/Library/WebServer/Documents"
        ServerName localhost
</VirtualHost>

<VirtualHost *:80>
        ServerName your.ip.number.here
        ServerAlias redmine.local
        ServerAlias your_server_name.local
        <Directory /Library/WebServer/redmine>
                Order allow,deny
                Deny from all
        </Directory>
        <Directory /Library/WebServer/redmine/public>
                AllowOverride All
                Options Indexes FollowSymLinks -MultiViews
                Order allow,deny
                Allow from all
        </Directory>
        RailsBaseURI /redmine/public
        RailsEnv production
</VirtualHost>

Restart apache in the terminal with sudo apachectl graceful, or in System Preferences->Sharing (toggle Web Sharing off and on).

Go to http://your.ip.number/redmine/public in a web browser to check if all is working. The first time you load redmine, the server needs to start ruby so it takes a moment, but the response should be quicker after that.

If things go wrong it might be helpful to open the Console to look at relevant logs.

Monday, December 19, 2011

Compiling mplayer on Mac OS X 10.7

Compiling mplayer on Mac OS X 10.7 (Lion)

Have been doing this for ages, but with Lion there's a new error:

~/mplayer > ./configure
No FFmpeg checkout, press enter to download one with git or CTRL+C to abort

Cloning into ffmpeg...
remote: Counting objects: 7969, done.
remote: Compressing objects: 100% (5420/5420), done.
Receiving objects: 100% (7969/7969), 9.84 MiB | 1.40 MiB/s, done.
remote: Total 7969 (delta 5169), reused 4198 (delta 2409)
Resolving deltas: 100% (5169/5169), done.
Checking for cc version ... 4.2.1 
Checking for working compiler ... yes
Detected operating system: Darwin
Detected host architecture: x86_64
Checking for cross compilation ... no 
Checking for host cc ... cc 
Checking for CPU vendor ... GenuineIntel (6:23:10) 
Checking for CPU type ...  Intel(R) Core(TM)2 Duo CPU     E8335  @ 2.93GHz 
Checking for kernel support of mmx ... yes 
Checking for kernel support of mmxext ... yes 
Checking for kernel support of sse ... yes 
Checking for kernel support of sse2 ... yes 
Checking for kernel support of ssse3 ... yes 
Checking for kernel support of cmov ... yes 
Checking for mtrr support ... yes 
Checking for GCC & CPU optimization abilities ... core2 
Checking for byte order ... little-endian 
Checking for extern symbol prefix ... _ 
Checking for assembler support of -pipe option ... yes 
Checking for GCC support of -mstackrealign ... no 
Checking for PIC ... yes 

Error: Your binutils version is too old to compile for 64-bit

Check "config.log" if you do not understand why it failed.

Googling for the answer didn't help, but I finally came across the solution: use clang.

~/mplayer > ./configure '--cc=clang'

and off it goes...

Wednesday, September 28, 2011

iViewFox compatibility...

I have been having issues with iViewFox ceasing to work every time there's a FireFox update. Not sure why - it seems to be a problem on some computers and not others. Turns out there's a version compatibility string in one of the Add On's files, which can be edited...

The file is install.rdf, and it's located at ~/Library/Application Support/Firefox/Profiles/.default/extensions/iViewFox@forboden.com/install.rdf. It's an XML file that can be edited in your favourite text editor. In the file there's a line that says:

<em:maxVersion>4.0.*</em:maxVersion>

which I changed to:

<em:maxVersion>7.0.*</em:maxVersion>


Restarted FireFox, and the Add On seems to be functioning correctly.

Friday, August 26, 2011

Migrating redmine and subversion to Mac OS X 10.7

I've been meaning to write up an article on setting up redmine and subversion on Mac OS X 10.6 - mostly to aid my own memory in case I need to do it again, but also for anyone else who might want to do it. When I first set these things up they had to be built from source (along with their dependencies), and then configured from the ground up. Thankfully now lots of the required components are part of Mac OS X, so the process is much, much simpler.

Last week the Mac Pro I've been running as a server failed (suspected dud power supply), and while it's out for repair I've had to shift the redmine and subversion installations onto an alternate server to keep things moving. Mac OS X 10.7 is now out too, so I'm migrating the installation from 10.6, and updating some things along the way.

Why would you want do this? If you're into developing software you need source control. There are lots of other options, but subversion suits my workflow, and allows previous versions of source to be pulled, and allows work to be done and synchronised from multiple computers. I'm in a development team of 1, so I don't need multi-user features, but it's for that too. Redmine is a tool that can do a heap of stuff - I just use it for logging 'issues' with the software I'm working on (bugs, new feature requests...). I find myself using it as a sort of to-do list that I can access and update over the web. It runs using ruby on rails, so it's a good excuse to look at that too.

Starting conditions:

I have had redmine and subversion running under Mac OS X 10.6.8 on a Mac Pro. I'm migrating to an iMac running Mac OS X 10.7.2, with all updates current to December 12, 2011. Developer tools are installed, but there's nothing yet in /usr/local/. apache2 is installed on Mac OS X by default.

Here's some terminal output showing versions of relevant software components. You can see that mysql is not installed, and no ruby gems are installed:

server:~ user$ ruby --version
ruby 1.8.7 (2010-01-10 patchlevel 249) [universal-darwin11.0]
server:~ user$ gem --version; rake --version; rails --version;
1.3.6
rake, version 0.8.7
Rails is not currently installed on this system...
server:~ user$ mysql --version
-bash: mysql: command not found
server:~ user$ gem list

*** LOCAL GEMS ***

server:~ user$

Before we start, some of the more useful websites I've consulted are:

The Redmine Install wiki.

The Redmine Mac OS X - Snow Leopard installation notes.

A page describing moving redmine from one server to another, and a page with some info about installing MySQL on Mac OS X 10.6.

I'll break this into 4 parts: mysql (required for redmine), redmine, subversion, and configuring apache.

Installing MySQL:

Mac OS X doesn't come with MySQL, although I think the Server version does. Good installers are provided at www.mysql.com. I chose to install from a disk image using a GUI installer. The file I downloaded is mysql-5.5.18-osx10.6-x86_64.dmg.

Here are the steps:

  • install the mysql binary
  • install the startup item
  • install the prefPane in /Library/PreferencePanes

    It used to be necessary to set permissions on the startup item to get the prefPane to work. The code to do it is provided here for completeness, but with this version of mysql I found it was not necessary.

    In a Terminal:

    sudo chown -R root:wheel /Library/StartupItems/MySQLCOM

  • use the preference pane to start/stop mysql. Hopefully you get some green text that says "The MySQL Server Instance is running".

  • add mysql to the PATH. Do this by adding the following line to ~/.bashrc:

    export PATH=${PATH}:/usr/local/mysql/bin

    Then start a new terminal to load ~/.bashrc, or type

    source ~/.bashrc

    at the command line. You can check that mysql is installed correctly by running it (as superuser).

Installing Redmine

There are lots of docs around on how to do this - although there are some subtle differences between different operating systems. They tend to end at 'testing the development setup using webrick', which is not intended for production use. Here I'll describe setting up redmine to run through apache and be world visible.

I started by downloading the latest stable version:

svn co http://redmine.rubyforge.org/svn/branches/1.2-stable redmine-1.2

On Mac OS X the root of the web server is at /Library/WebServer/Documents, so a good place for redmine is in /Library/WebServer/Documents/redmine-1.2, with a symlink to /Library/WebServer/Documents/redmine. This is a restricted access directory, so all the terminal commands will need to be run as superuser. The following terminal commands assume you've cd'd into the redmine directory.

Configure database access

Copy config/database.yml.example to config/database.yml

cp config/database.yml.example config/database.yml

Edit config/database.yml, removing all but the 'production' section. In my case, I've edited it to look like this:

production:
  adapter: mysql
  database: redmine
  host: localhost
  username: redmine
  password: redmine_pass

Create the log file if necessary:

touch log/production.log

Change permission for the log file [may need to be created first]:

sudo chmod 0666 log/production.log

Change the owner for the files directory, where redmine will save files:

sudo chown -R _www files

NOTE: If migrating an existing redmine installation, the contents of the files directory need to be copied across, and the appropriate permissions set.

Create the mysql database for redmine

Run mysql as superuser

sudo mysql

Run the following commands in mysql to create the database. Note that the values are the same as those in config/database.yml

create database redmine character set utf8;
create user 'redmine'@'localhost' identified by 'redmine_pass';
grant all privileges on redmine.* to 'redmine'@'localhost';
exit

Again, if you're migrating an existing redmine installation, the database content needs to be copied across. To do this the mysql database needs to be 'dumped' from the old server with the following command (the password will be requested by mysql):

mysqldump -u redmine -p redmine > /location/of/dump/file.sql

The dumped database can be reloaded on the new server:

mysql -uredmine -p redmine < /location/of/dump/file.sql

Set up the necessary Ruby gems

i18n

sudo gem install -v 0.4.2 i18n

rails

I started by installing the v 2.3.5 version, but had some issues and ended up going to v 2.3.11 (see below).

sudo gem install -v 2.3.5 rails

mysql

Redmine needs this to access the mysql database where it stores its stuff. I started by not specifying a version number, which installed 2.8.1. If you read on you'll find that this is a problem, and version 2.7 is the magic version.

export ARCHFLAGS="-arch x86_64"
sudo gem install --no-rdoc --no-ri mysql -- \
    --with-mysql-dir=/usr/local/mysql \ 
    --with-mysql-config=/usr/local/mysql/bin/mysql_config

In order to ensure ruby can access the mysql libraries execute the following (first check the paths are relevant for your versions). This assumes your current directory is /usr/local/mysql/lib.

sudo install_name_tool -change libmysqlclient.18.dylib \
    /usr/local/mysql/lib/libmysqlclient.18.dylib \
    /Library/Ruby/Gems/1.8/gems/mysql-2.8.1/lib/mysql.bundle

You can then check things are OK by running irb and checking that mysql can be accessed:

> irb
require 'mysql'
true
exit

If you see 'true', you're good to go.

Versions... here's where the fun stuff begins...

The next thing to do is go into the redmine directory, and run a rake command to generate the session store. This is to generate a session store secret. Not sure exactly what that means, but it sounds important.

> sudo rake generate_session_store

Now, if you're lucky a heap of text will fly by, and there are no errors. More likely there will be some problems... Here's what I got:

Missing the Rails 2.3.11 gem. Please `gem install -v=2.3.11 rails`, update your \
RAILS_GEM_VERSION setting in config/environment.rb for the Rails version you do \
have installed, or comment out RAILS_GEM_VERSION to use the latest version installed.

So, I installed the 2.3.11 version.

sudo gem install -v 2.3.11 rails

Now, try again...

sudo rake generate_session_store

And once more, crashed and burned:

WARNING: 'require 'rake/rdoctask'' is deprecated.  Please use 'require 'rdoc/task' (in RDoc 2.4.2+)' instead.
    at /Library/Ruby/Gems/1.8/gems/rake-0.9.2.2/lib/rake/rdoctask.rb
WARNING: 'task :t, arg, :needs => [deps]' is deprecated.  Please use 'task :t, [args] => [deps]' instead.
    at /Users/wpowell/Development/redmine-1.2/lib/tasks/email.rake:170

I looked around and concluded that the version of rake was too new, so downgraded:

sudo gem uninstall rake

Select gem to uninstall:
 1. rake-0.8.7
 2. rake-0.9.2.2
 3. All versions
> 2
Successfully uninstalled rake-0.9.2.2

So now we're running rake 0.8.7, let's have another go...

sudo rake generate_session_store

Woo Hoo! Seems to have worked. Next step is to migrate the database:

sudo RAILS_ENV=production rake db:migrate

And there's another error...

rake aborted!
Object is not missing constant Issue!

(See full trace by running task with --trace)

Cryptic. Running the full trace didn't really help. This error has bugged me in the past, and I eventually figured out that it's related to the version of the mysql gem. So, uninstall version 2.8.1 and install 2.7...

sudo gem uninstall mysql
export ARCHFLAGS="-arch x86_64"
sudo gem install -v 2.7 --no-rdoc --no-ri mysql -- \
    --with-mysql-dir=/usr/local/mysql \
    --with-mysql-config=/usr/local/mysql/bin/mysql_config

Again, make sure ruby can access the mysql libraries...

sudo install_name_tool -change libmysqlclient.18.dylib \
    /usr/local/mysql/lib/libmysqlclient.18.dylib \
    /Library/Ruby/Gems/1.8/gems/mysql-2.7/lib/mysql.bundle

Now let's try that rake command again.

sudo RAILS_ENV=production rake db:migrate

...lots of printing out... looks promising. Now test with the WEBrick web server built into rails

ruby script/server webrick -e production

Open a web browser to http://127.0.0.1:3000... and it works!!

Moving to apache

This can be covered in another post...

Tuesday, January 25, 2011

code blocks in blogger

Something I've just come across is the formatting of code blocks in blogger:
 here is some code

There are plenty of sites out there with information on how to do this [Dashboard->Design->Template Designer->Advanced->Add CSS], but what they fail to mention is that the CSS is not correctly implemented in blogger's preview pane [Edit Posts->Preview].
I spent ages testing out CSS changes, expecting to see the effect in the Preview pane, but nothing seemed to be working. After some frustration I decided to post anyway, viewed the post and there was the CSS formatting implemented correctly. If only someone had mentioned this sooner...

adding an smb share in linux

Another post of something I do regularly but can't quite remember...

Running df gives the mount point:
> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda1 72G 4.7G 64G 7% /
tmpfs 253M 0 253M 0% /lib/init/rw
udev 10M 768K 9.3M 8% /dev
tmpfs 253M 0 253M 0% /dev/shm
/dev/sda1 917G 13G 859G 2% /mnt/raidk
/dev/sdb1 230G 11G 207G 5% /mnt/alfred

In this case it's /mnt/alfred I'd like to share.

Now to add the entry to /etc/samba/smb.conf:
> sudo gedit /etc/samba/smb.conf

I tend to copy an existing section of the config file and change the relevant parts. In this case, the new entry looks like:
[alfred]
path = /mnt/alfred
available = yes
browsable = yes
writable = yes
valid users = joeblo
admin users = joeblo

Once the changes are made, samba needs to be restarted:
> sudo /etc/init.d/samba restart
Stopping Samba daemons: nmbd smbd.
Starting Samba daemons: nmbd smbd.

Now that samba is restarted the new share is available on the network.

Monday, January 24, 2011

connecting a new drive to linux

This is one of those things I do infrequently enough that I'm not quite able to remember all the details. So here's the procedure documented so that I can do it more quickly next time. Maybe someone else might find it useful too.

FYI, the linux distro I'm running is Debian, and I'm connected to it from Mac OS X via ssh.

1. Connect the drive, power it up, and see where linux has put it.
> sudo /sbin/fdisk -l
Disk /dev/hda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xdd27dd27

Device Boot Start End Blocks Id System
/dev/hda1 * 1 9541 76638051 83 Linux
/dev/hda2 9542 9729 1510110 5 Extended
/dev/hda5 9542 9729 1510078 82 Linux swap / Solaris

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 1000.1 GB, 1000153686016 bytes
255 heads, 63 sectors/track, 121595 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sda1 1 121596 976712583 ee EFI GPT

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdb: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sdb1 1 30402 244198583 ee EFI GPT

In this case the drive I'm interested in is the 250 Gb one, and it is at /dev/sdb1.

2. Run gparted to partition the drive.
> sudo /usr/sbin/gparted

gparted is a gui application. I'm running it over an ssh connection (using the -Y option), so it will display on my Mac OS X machine using the X11 server that's part of Mac OS X. Choose the relevant drive (/dev/sdb), and delete all the current partitions. All the available space should be shown as "unallocated" when you're done. Click Apply to make the changes to the disk - note that this will destroy any data on the drive!
The new partition needs a filesystem before it can be used. This can also be done with gparted: click New, change to ext3, primary partition should already be selected and the other defaults are OK. Click Apply and wait a while. The bigger the partition, the longer the wait. Once this is complete the drive is ready for use, but the operating system needs to gain access by mounting it.

3. Now to define a mount point. In my system, these are at /mnt, and running ls -l /mnt shows the default permissions:
> ls -l /mnt
total 12
drwxr-xr-x 2 root root 4096 2010-04-30 14:18 cuda_80
drwxr-xr-x 2 root root 4096 2010-09-06 15:16 five_hundred
drwxr-xr-x 4 root root 4096 2010-12-23 13:59 raidk

cuda_80 is the drive that I'm replacing - the physical drive has already been removed from the system, and I've removed it from fstab and smb.conf. The new drive I'm calling alfred (I watched Batman last week), so I'll rename cuda_80 to alfred rather than deleting it and creating a new one. This will maintain the correct permissions, too.
> sudo mv /mnt/cuda_80 /mnt/alfred


4. I need to know the drive's UUID so I can add it to the fstab file. Using the UUID makes it independent of the /dev/* information, so that if other drives are added or removed and the drive ordering changes, this drive will still mount OK. The UUID is retrieved with:
> /sbin/blkid
/dev/hda1: UUID="0521aa47-4fdb-424e-bd83-0f6b164e55f5" TYPE="ext3"
/dev/hda5: TYPE="swap"
/dev/sda1: UUID="637d0461-72d5-40c2-9446-04a45912eb8e" SEC_TYPE="ext2" TYPE="ext3"
/dev/sda5: TYPE="swap"
/dev/sda2: UUID="13A3CE68BA98A5AF" LABEL="Untitled" TYPE="hfsplus"
/dev/sdb1: UUID="a5c5ba62-3c58-4ff4-9d1c-6d754f4430f0" SEC_TYPE="ext2" TYPE="ext3"

Using the information from step 1, I know the drive is at /dev/sdb1, so the UUID is the long string starting with a5c.

5. Now to edit fstab. I'll do this in gedit, which has a gui interface so I can cut and paste.
> sudo gedit /etc/fstab

I'm duplicating a fstab line that works, then changing the UUID and mount point to the values determined in the previous steps. My extra line in fstab is:
UUID=a5c5ba62-3c58-4ff4-9d1c-6d754f4430f0 /mnt/alfred ext3 defaults,errors=remount-ro 0 2

The line is made up of 6 fields separated by tabs. The UUID and the mount point are the first two. The third is the filesystem type (ext3 in this case), and the fourth is a collection of options (see the fstab or mount man pages for details), the fifth has something to do with dumping the filesytem (I've never really read the finer details) and the final field "determines the order in which filesystem checks are done at reboot time." Root filesystems should have the value of 1, but this is not a root filesystem so it has the value of 2. This also gives the USB drive time to get going during startup so that it's ready by the time the OS looks for it.

6. Now that the fstab has been edited, it should be possible to mount the filesystem. This can be done as follows:
> sudo mount -a

There's a little bit of noise from the drive, and running df -h shows that the drive is now mounted (the -h option causes the drive sizes to be shown "human readable").
> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda1 72G 4.7G 64G 7% /
tmpfs 253M 0 253M 0% /lib/init/rw
udev 10M 768K 9.3M 8% /dev
tmpfs 253M 0 253M 0% /dev/shm
/dev/sda1 917G 200M 871G 1% /mnt/raidk
/dev/sdb1 230G 188M 218G 1% /mnt/alfred

It should also be mounted automatically when the system restarts.