Thursday, February 27, 2014

BBEdit/TextWrangler Codeless Language Module for IDL


UPDATE April 2015: I've now updated the CLM for IDL vesion 8.4, for use with BBEdit 11.x. You can download it here.

For nearly a decade I've been using IDL, a programming language that's good for visualising lots of data. Recent versions ship with an IDE based on Eclipse, which is either horrible or great depending on whether you're an advocate of the Eclipse environment. I'm not.
Call me old fashioned, but I like to code in a simple text editor, and debug from a terminal. I work on Mac OS X, and my choice of text editor is BBEdit or TextWrangler, both from BareBones. TextWrangler is free and should be on every Mac, and BBEdit costs money but has some great additional features. I've tried to find a text editor as useful on Windows, but there just isn't one.
A few years ago I wrote a Codeless Language Module (CLM) for BBEdit (and TextWrangler), so that it can apply syntax highlighting to IDL source code. The CLM is a .plist file, which is a flavour of XML that contains keys described in BareBones' documentation that allow lots of cool stuff to happen. This includes:
  • syntax highlighting (comments, quotes, etc)
  • keyword highlighting (and predefined names)
  • autocompletion of function names
  • function scanning (makes function names apear in the function pop-up menu for easy navigation)
  • automatic recognition of language based on filename extension
  • code folding
Here's what it looks like:


The IDL Configuration.plist file is provided here for you to download. There are some comments at the top of the file about how it was generated - mainly for my own memory so I can regenerate it when a new version if IDL comes out. It refers to some custom functions I wrote to extract information from the IDL documentation and help files, and generally make life easier. If you'd like to get hold of these functions let me know.

Using CTAGS with IDL and BBEdit/TextWrangler

As an added bonus, we can also use Exuberant Ctags to scan IDL code files and generate 'tags' files. These are used by BBEdit for autocompletion. The use of ctags is well documented in the BBEdit manual, so I won't go into detail here, except to say that they're great, and I have written a routine to generate a 'tags' file for IDL codefiles, which you can download here. You also need to have the appropriate stuff in ~/.ctags for ctags to scan your *.pro files correctly. These are the lines that do the magic:
--langdef=IDL
--langmap=IDL:.pro
--regex-IDL=/(^[ \t]*(pro|function)[ \t$]+)([A-Z_a-z0-9:]+).*$/\3/d,definition/i
If you have any problems or suggestions, please let me know.

Connecting to an SMB share...

How to connect to an SMB share...

with spaces in its name...

from Mac OS X using the Terminal or a bash script

This is something I needed to do in order to back up some Windows systems with rsnapshot, a python script that uses rsync. Getting it to back up Windows systems over the network requires that the Windows computers share the directory or directories that need to be backed up, and the Mac OS X box mounts the shares for the backup to run. There may be other ways, but this works well for us at the moment. The only issue has been working out how to mount Windows shares with spaces in their names from a bash script, so the backup could be scheduled to run automatic and unattended.

It took a while to figure out. Googling didn't really help much - there were plenty of suggestions but nothing actually worked. These ranged from quoting the space with backslashes, replacing the quote with its octal representation (\040), using double-quotes. In the end, many people suggest that renaming the share to get rid of the space is the way to go.

In the past this is what I'd done: altered the name of the Windows share to remove the spaces, but in the back of my mind I knew it should be possible. Finally I got around to working it out properly...

The following are known to work in Mac OS X 10.9 with shares from Windows 7, XP and Vista. There are two steps to mount the share.

Create the mount point

This is pretty straightforward. If this is done in the bash shell the spaces can be quoted with a backslash, or the entire name can be double-quoted. The mount point can be created anywhere in the file system, but if it's in /Volumes the mounted volume will be available in the Finder.

mkdir /Volumes/Share\ Point

or

mkdir /Volumes/"Share Point"

Mount the share

This is done at the bash prompt or in a script using the mount command. You must specify the username and password, the address of the server, the name of the share and the mount point.

The path to the mount point is fairly simple. In typical bash fashion it can be specified with double quotes or the space could be preceded with a backslash - either will work. The real trick is getting the path to the SMB share exactly right. My first (failed) attempts were to use double quotes as follows:

> mount -t smbfs "//username:password@ip_number/Share Point" "/Volumes/Share Point"
mount_smbfs: URL parsing failed, please correct the URL and try again: Invalid argument

which clearly did not work. Next try was escaping the space with a backslash:

> mount -t smbfs //username:password@ip_number/Share\ Point "/Volumes/Share Point"
mount_smbfs: URL parsing failed, please correct the URL and try again: Invalid argument

which also failed. Similarly, using the octal representation of the space:

> mount -t smbfs //username:password@ip_number/Share\040Point "/Volumes/Share Point"
mount_smbfs: mount error: /Volumes/Bruker Backup: Unknown error: -1073741412

this gave an exciting new error message, but still failed :(

Finally, I was successful when I replaced the space with %20 its URL-encoded representation:

mount -t smbfs //username:password@ip_number/Share%20Point /Volumes/Share\ Point/

Phew! The share will now be available in /Volumes, and also visible in the Finder. When you're finished with it and it's no longer needed, it can be unmounted in the Terminal with

umount /Volumes/Share\ Point

or by Ejecting it in the Finder.

Getting it to work in a bash script

I use a function within a bash script to do the mounting of smb shares. I pass in the username and password, the server name and the share name and the function does the heavy lifting. If the share name to be mounted has spaces in its name, the spaces must be replaced with %20 in the function call. Within the function the share name is used with the %20 in the call to mount, and the %20 is replaced with a space and the share name is double-quoted when used to create the mount point, specify the mount point in the mount call, and to unmount the share when it's no longer needed. A proof of concept script would look something like:

#!/usr/bin/env bash

IP="192.168.0.1"
USERNAME="username"
PASSWD="password"
SHARE_NAME="Share%20Point"

# bash can replace the %20 with a space...
MOUNT_POINT="/Volumes/${SHARE_NAME//\%20/\ }"

mkdir "${MOUNT_POINT}"

mount -t smbfs "//${USERNAME}:${PASSWD}@${IP}/${SHARE_NAME}" "${MOUNT_POINT}"

# do something useful with the share at this point

umount "${MOUNT_POINT}"

Saturday, May 25, 2013

encoding video for topfield pvr

Sometimes we use our Topfield PVR to play back video from 'other' sources. What I mean by this is video that it didn't record itself, which might be video from the web (eg. youtube), or recordings made on a linux box with a DVB card. The PVR is a TF 7100 HD PVRt Plus.

The manual for the PVR is a little vague on the subject. It says: "You can enjoy video files in DivX, vob, mkv, or mp4 format on the digital receiver." Not really any mention of codecs or container formats, and nothing on audio encoding. The unit has a USB port on the front (and another on the back) where an external hard drive can be attached, but according to the manual: "To copy files to the external hard drive, it must be formatted through the Topfield in the JFS file system. After this, the external hard drive can only be used with the Topfield PVR." Bad news for anyone not running linux (even heard of JFS??), but after some experimenting, here's what I've found actually works...

Video Encoding

Video formatted for iOS seems to work. I have a bash script that does the encoding using ffmpeg, but anything that can produce mp4 formatted video for iOS should be able to produce video that the Topfield can display. I experimented with DivX avi files as suggested by the manual, but as far as I know aspect ratio is not encoded into avi files, so the video was display in the incorrect aspect ratio. It would be possible to correct this during playback in the PVR, but I found that mp4 files displayed with the correct aspect ratio with no adjustment necessary.

Disk Filesystem

I use a USB stick, formatted in EXT2 format. The Topfield box reads this OK, using the USB port on the front. To write the video files to the disk you either need access to a linux box, or a linux virtual machine. I use a VM running Debian, on Mac OS X. It's a little slow, but it works. This should work just the same with an external USB hard drive, although I've not tried it.

Permissions

Last point, which seems to be crucial, is that the Topfield can only see folders and files that have root for the owner/group. It may not be necessary to have *both* owner and group set to root, but this is what I've found works. For example, here are the permissions on a demo video file supplied on the PVR:

---------- 1 root    root    65337872 Feb  6  2009 Alterna_Compilation.divx
Very limited, but the Topfield seems to like it. Hope it works for you.

For those that are still reading...

Here's ffmpeg's info on the video file supplied with the Topfield PVR. Make of is what you will:


~ > sudo ffmpeg -i /Users/foo/Alterna_Compilation.divx 
[mpeg4 @ 0x7fc78a029600] Invalid and inefficient vfw-avi packed B frames detected
Input #0, avi, from '/Users/foo/Alterna_Compilation.divx':
  Duration: 00:02:05.46, start: 0.000000, bitrate: 4166 kb/s
    Stream #0:0: Video: mpeg4 (DX50 / 0x30355844), yuv420p, 720x400 [SAR 1:1 DAR 9:5], 23.98 fps, 23.98 tbr, 23.98 tbn, 30k tbc
    Metadata:
      title           : Video 
    Stream #0:1: Audio: mp3 (U[0][0][0] / 0x0055), 48000 Hz, stereo, s16p, 160 kb/s
    Metadata:
      title           : Audio 
At least one output file must be specified

Tuesday, January 10, 2012

a ruby wrapper for 'svn switch'

A while ago our local network admins decided static IP numbers were a Bad Thing, causing chaos for me when trying to use a machine as a server for subversion (among other things).

I found myself trying to check changes in subversion repositories back in to the server, only to find that the server was no longer at the IP address from which the source had been checked out. The answer (apart from attacking the network support people) is the svn switch command. This is a multi-purpose tool with lots of options, so I wrote a wrapper to simplify the process of changing the repository's root URL and nothing else. It was also an excuse to learn a bit of ruby, which is way cool.

#!/usr/bin/env ruby

    new_server_address= ARGV[0]
    fail "Specify the new server name or IP number" unless new_server_address

    svn_info = %x{svn info}
    fail "'.' is not a working copy" if svn_info == ""

    url_regex = Regexp.new('^URL: (.*$)')
    repository_url = url_regex.match(svn_info)[1].to_s
    fail "Unable to find the URL" unless repository_url

    split_repository_url = repository_url.split('/')
    split_repository_url[2] = new_server_address
    new_repository_url = split_repository_url.join('/')
    puts "Switching from \n#{repository_url} \nto \n#{new_repository_url}"
    puts "Continue? [Y]"
    confirm = STDIN.gets.chomp
    if confirm.upcase == "Y" then
        system("svn switch --relocate #{repository_url} #{new_repository_url}")
    end

Subversion setup

Start by editing /etc/apache2/httpd.conf so that the httpd-subversion.conf file gets included.

Include /private/etc/apache2/extra/httpd-subversion.conf

There's no httpd-subversion.conf file by default, so we need to make one. On my system, it needs to contain the following:

LoadModule dav_svn_module libexec/apache2/mod_dav_svn.so
LoadModule authz_svn_module libexec/apache2/mod_authz_svn.so

<Location /svnrepos>
    DAV svn
    SVNParentPath /usr/local/svnrepos
    # the following svnindex.xsl (and svnindex.css) are located in the
    # server DocumentRoot dir, which is defined in httpd.conf with the following line:
    # DocumentRoot "/Library/WebServer/Documents"
    SVNIndexXSLT "/svnindex.xsl"
    # require SSL connection for password protection
    SSLRequireSSL

    # user authentication
    AuthType Basic
    AuthName "Subversion Repository"
    AuthUserFile /etc/apache2/subversion.auth

    # only authenticated users may access the repository
    Require valid-user
</Location>

The .xsl and .css files were copied from my old server. They're just to make browsing the subversion repo more nice.

The subversion.auth file is generated using htpasswd as follows, and stored in /etc/apache

sudo htpasswd -cm /etc/apache/subversion.auth my_user_name

After the command is issued, a password must be entered (twice).

Test the install by trying to access a subversion repository (presumably in /usr/local/svnrepos, according to the above config). You should be required to use a https connection, and be prompted for your username and password.

Generate SSL keys on Mac OS X Lion

I've done this successfully using a script I wrote a while back. One of the gotchas seems to be that the "Common Name" should be the server name (or static IP number if you don't have one). This must match the ServerName directive in httpd.conf.

#! /bin/bash
echo; echo -e "Generating Certificate Authority (CA) [1]:"
openssl genrsa -des3 -out ca.key 4096

echo; echo -e "Generating Certificate Authority (CA) [2]:"
openssl req -new -x509 -days 365 -key ca.key -out ca.crt

echo; echo -e "Generating Server Key:"
openssl genrsa -des3 -out server.key 4096

echo; echo -e "Generating certificate signing request (CSR):"
echo -e "       ** name entered for Common Name (CN) should match your server name **"
openssl req -new -key server.key -out server.csr

echo; echo -e "Signing CSR with CA:"
openssl x509 -req -days 365 -in server.csr -CA ca.crt \
        -CAkey ca.key -set_serial 01 -out server.crt

echo; echo -e "Making insecure version of server.key for apache startup"
openssl rsa -in server.key -out server.key.insecure

echo; echo -e "Renaming server secure/insecure keys"
mv server.key server.key.secure
mv server.key.insecure server.key

sudo cp server.key /etc/apache2/server.key
sudo cp server.crt /etc/apache2/server.crt

Still seems to work...

In order to get apache to use the ssl certificates it's necessary to change httpd.conf

First uncomment the LoadModule line to get ssl_module to load:

LoadModule ssl_module libexec/apache2/mod_ssl.so

Uncomment the line in httpd.conf to get the httpd-ssl.conf file loaded:

Include /private/etc/apache2/extra/httpd-ssl.conf

Now make some changes to httpd-ssl.conf. This file is in the extra/ directory.

Set the ServerName correctly (and the ServerAdmin email address, if you like)

ServerName name.of.your.server:443
ServerAdmin email.address@nowhere.com

Set the SSLCertificateFile and SSLCertificateKeyFile to the correct file paths:

SSLCertificateFile "/private/etc/apache2/server.crt"
SSLCertificateKeyFile "/private/etc/apache2/server.key"

And that should be it. Try loading https://localhost in a web browser. Check the logs if there's a problem.

Moving redmine to apache

This post follows the last one about installing or migrating redmine and mysql. Picking up from the point where redmine was working through WEBrick...

A couple of extra ruby gems are required to get apache to serve redmine to the world. Passenger does most of the hard work, and when it installs it provides it's own config instructions. Neato!

sudo gem install passenger
sudo passenger-install-apache2-module

The last command is really cool, it does lots of work compiling the apache module. I guess it's something from the rails community. The instructions tell us to add some lines to the apache configuration file. I'm putting them in /etc/apache2/other/passenger.conf. After I'm done, the passenger.conf file looks like this:

# settings from passenger-install-apache2-module
LoadModule passenger_module/Library/Ruby/Gems/1.8/gems/passenger-3.0.11/ext/apache2/mod_passenger.so
PassengerRoot /Library/Ruby/Gems/1.8/gems/passenger-3.0.11
PassengerRuby /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby

# additional settings
PassengerUserSwitching on
PassengerUser _www
PassengerGroup _www

Set user:group on /etc/apache2/conf/extra/passenger.conf

sudo chown _www:_www /etc/apache2/conf/extra/passenger.conf

And get apache to load the passenger.conf file by adding the following line to httpd.conf

Include /private/etc/apache2/other/*.conf

Redmine runs on apache using a virtual host. Include the httpd-vhosts.conf file by uncommenting the relevant line in httpd.conf:

# Include /private/etc/apache2/extra/httpd-vhosts.conf

Then edit the httpd-vhosts.conf file to get things working. Comment out the example configs (anything referring to *dummy-host*), and add the following config sections:

<VirtualHost *:80>
        DocumentRoot "/Library/WebServer/Documents"
        ServerName localhost
</VirtualHost>

<VirtualHost *:80>
        ServerName your.ip.number.here
        ServerAlias redmine.local
        ServerAlias your_server_name.local
        <Directory /Library/WebServer/redmine>
                Order allow,deny
                Deny from all
        </Directory>
        <Directory /Library/WebServer/redmine/public>
                AllowOverride All
                Options Indexes FollowSymLinks -MultiViews
                Order allow,deny
                Allow from all
        </Directory>
        RailsBaseURI /redmine/public
        RailsEnv production
</VirtualHost>

Restart apache in the terminal with sudo apachectl graceful, or in System Preferences->Sharing (toggle Web Sharing off and on).

Go to http://your.ip.number/redmine/public in a web browser to check if all is working. The first time you load redmine, the server needs to start ruby so it takes a moment, but the response should be quicker after that.

If things go wrong it might be helpful to open the Console to look at relevant logs.