Utility Maintenance

Everyone has their own way of working, and their own needs for tools to support their approach. This is how I'm dealing with the question of utility code for APL (specifically Dyalog APL) application development.

Why Utilities

I use utilities (pre-written functions and operators) to:

Some people prefer to write all of their code inline – that's fine – but I like to build on what's gone before. Some of my utility functions are trivial – they could as well be coded inline – but others are more complex, and it's handy to have them available on-demand and ready to plug into applications

Development Environment and Methodology

I'm working as a solo developer, I don't have any “team” issues to deal with.

I have several discrete applications under development at any one time – and I want changed utilities to percolate effortlessly into each. I'm willing to deal with integration problems when they arise (for example, if the arguments to a utility function change), but I don't want to find myself with different versions of what's supposedly the same object in different applications. And on the converse side, if I change a utiity while working in one application I want the modified version to be saved back into the utility library automatically.

I'm not bothered about version control – these utilities are supposed to work – if they don't I fix them there and then, I don't need an audit trail of changes. If a change looks like it might be “controversial”, then I make a new utility with a different name – later I can do some rationalisation and either settle for one or the other or decide that I need both.

My current situation is one in which continuous development is the order of the day, there are no requirements for a “stable release version”. But there's a switch built in which lets any application workspace (one application – one workspace, that's my rule) be decoupled from the utility library and become static – if that's the need, then the workspace can be handed off for formal quality assurance and packaging with an invariant set of utilities.

Technicalities

What controls everything is a command line argument called umode; if umode='d' then utilities are loaded from a library file and saved when modified – if umode isn't there then the workspace is static (the utilities it needs have already been put in place, and any changes will not go back to the utility file (unless the developer decides to be really insistent, and bypasses the code that's supposed to do the job). umode is a command line argument because I'd rather do it that way than fool around with Registry entries (my world is – regrettably – a Windows one, but I want to leave my options as open as I can – command line arguments and files are simple and universal).

Utility File Format

I agonised over this, because deep down I wanted to maximise my flexibility and develop an approach which would let me share code between APL/W and other APLs (principally APLX and APL2). I knew from earlier explorations that it would be possible to tuck APL code away inside relational databases which could be read by various APL implementations (sure, there'd be some character translation issues, and some code compatibility – but it could be done). Using the same utility code from APLX as from APL/W would be even more attractive than reusing it across APL/W applications.

But I haven't been doing much outside of APL/W, and – when I did begin to develop an application recently with “flat APL” the loss of namespaces was so painful that I switched it back to APL/W. My priority for utilities was to solve the “easy propagation” issue, so I decided that I'd make something to work with APL/W. If the need arises to make it work with another APL at some time in the future, then that'll be an opportunity at that time.

Having made the “APL/W-only, for now” decision it then became more attractive to think in terms of ŒOR rather than ŒCR, so that utilities could be organised into namespaces, with these saved into components of an APL/W component file as complete entities rather than deal with individual functions and operators (variables may also be parts of utilities, but because they aren't time-stamped I tend to not use them unless it's inescapable)

Namespaces and Paths

All utilities are held in subspaces of a namespace called #.UTILS. For example, if I have a utility called PRINT it's going to be loaded into a namespace within the application namespace called #.UTILS.PRINT. There's only this level, no further sub-spaces (this seems to be working out in practice – most of my utility namespaces are at most a couple of dozen objects).

When a utility is loaded the namespace is added to the ŒPATH variable. I know some people don't like application code to depend on ŒPATH resolution, but that's their choice – it works well enough for me. Doing it this way does mean that I need to take some care to avoid name clashes, but it hasn't happened yet and the applications are non-trivial.

Utility File Components

Utilities are held in a component file with the following components

1

General file description

2

Index, five-column matrix where:

[;0] Utility space name
[;1] Last-modify timestamp
[;2] Component containing utility space
[;3] Comment
[;4] Required utilities

3

Free component list

4

This description

5-10

Not used

11 on

ŒORs of utility namespaces



Which should be self-explanatory. Notice the “required utilities” column – when a utility requires another (for example REPORTS requires PRINT) an entry in this column means that the application code only needs to get top utilities explicitly – asking for REPORTS will automatically get PRINT, and anything that PRINT is noted as requiring. Multiple requirements are catered for (the “required” list is a blank-separated list).

Utility Loads and Saves

Within an application there has to be a #.UTILS namespace, which contains three objects (and it's only these three objects that need to be manually copied into new applications, or if they're changed – nothing's perfect, but I can handle manual synchronisation of three objects).

UtilFile

The name of the utility file (I work with one, but if versioning and so forth was to rear its head, everything's in place to handle multiple utility files).

I haven't found any need to split utilities across multiple files – maybe I will one day.

UtilLoad

Takes a list of utilities and loads them into the appropriate subspaces of #.UTILS; in detail...

Resolves any “required” items

For each namespace, checks whether there's a later version already in the workspace

If the later version is on file, either create a new namespace or refresh the existing one

Return a list of all utilities loaded (which may be a superset of the input list, because of requirements). Note that this results list will contain entries for utilities not loaded because newer veraions are already in the workspace – but utilities not found on file are not in the returned list.

There's some chitchat sent to the session so I see that it's done what it was supposed to (this is a residue from early development, before I was sure it was working properly).

The “test for newer” is crude – it looks at ŒAT of objects in existing namespaces and compares to the timestamp in the utilities file – that's one problem with variable, they don't have a ŒAT value.

UtilSave

Takes a list of utilities and saves them back to file (if they're newer than the version on file).

Normally the list of utilities is the one returned by UtilLoad, but a developer might have added something new (but there are “more approved” ways to add new utility namespaces).



So, the latent expression of an application looks something like this...

Weather w;ŒIO;ŒML;ŒPATH;utils;umode
© Driver for the weather application
ŒIO ŒML ŒPATH„0 3 '† #.UTILS'
umode„#.GetEnvironment'umode'
utils„'ARRAYS' 'BLANKS' 'DATES' 'DOGON' 'FILE' 'GEN' 'GUI' 'HELP' 'PRINT' 'STRINGS' 'TODO' 'WHOLES'
utils,„(umode='d')/›'TOOLS'
utils„umode UtilLoad utils
#.MAIN.Main''
umode UtilSave utils

Which should be self-explanatory (TOOLS is a set of utilities which developers would use rather than the application code, and all the application business is taken care of by #.MAIN.Main.

Utility Changes On-the-fly

While working with and developing an application new functions and operators might be required in a utility namespace, or existing code might be changed.

Break (accidentally or deliberately) the application – make the changes – carry on. If you )RESET after changing code then restart UtilLoad will see that the workspace contains a newer version of the namespace and not refresh from file (but watch out for changing variables, you may have to make a fake function edit). If you don't )RESET (or even if you do) next time the application comes to a tidy end UtilSave sees that you've made some changes and saves them back to file.

Only one developer, only one application being changed at that moment – it just works (and your changes get picked up by the next application just as soon as you start using it)

The only thing that isn't handled cleanly here is creation of new utility namespaces. You can do it, but it's easier to use the utility maintenance application (later) – and probably more appropriate if you're making those sort of changes.

Utility Examples

Some utilities pop up everywhere, think of them as extensions of the APL language (stuff like remove leading blanks); others are more complex, for example...

Application Configuration

Each of my applications has a configuration file (remember, I don't fool around in the Windows registry). It's a component file and applications all need to do the same sort of things to their configuration files (read them, let the user make changes, save the results).

The CONFIG utility lets all these things happen without having to write custom code in each application. In addition to read/write it can build a little (modal) dialogue which is available to the application for configuration changes – it's controlled by the individual configureation file (which is itself different for each application).

Entries in a configuration file are four-element vectors:

Developer Tools

Things that developers use, for example

ToDo Lists

Another common application component; each application has its ToDo list and this utility offers a simple dialogue which lets developers review and modify the list.

Utility Maintenance

While on-the-fly code changes work as intended, there's also a need for more “serious” utility maintenance, which is where the UtilMaint application becomes useful.

Everything works from a main form...






What the form lets us do is

Notice that there's a function called desc. UtilMaint adds this automatically to each utility namespace, and amends it when objects are changed. This gives an automatic (if simple) audit trail for changes when UtilMaint is used – but if you make changes on the fly you'll have to deal with desc yourself.

Technical Description

#.UTILS.adhocs

Namespaces loaded for display/examination. They won't be saved, but they do have to be erased when the utility maintenance application closes

#.UTILS.loaded

Namespaces that have to be loaded to make the utility maintenance application work (it's the same global that's used for the same purpose in other applications which use utility code). The usual saving rules apply as in other applications – but really they shouldn't get invoked because any changes will be made to the namespace as a modified namespace and handled as below.

#.UTILS.modded

Namespaces that have been loaded, modified, but not saved.



Support and Queries

Dogon Utility Maintenance is available on a no-charge as-is basis; there is no formal support, but queries and suggestions may be sent to Dogon Research.

The APL code (which may be obtained on request from Dogon Research) is copyright © Dogon Research 2003-2004, but may be copied for personal use; commercial use is strictly prohibited. The executable version may not be reverse-engineered.

Copyright © Dogon Research Ltd., 2003-2004