Recording Local History – the 4th Statistical Account 2.0

Last summer I did a project with the good folk of the East Lothian Antiquarian and Field Naturalists Society to revamp the East Lothian Fourth Statistical Account website. I had previously made the web presence for the society, which included all the published volumes of the transactions since 1937 and other manuscripts and books, around 50 volumes all told, which are now indexed and searchable c/o our friends at Google.

The Fourth Statistical Account was only 7 volumes, but there was already a website with all the seven volumes and much more which never made it into the printed version.

It didn’t make a lot of sense to simply convert the print versions to pdf nor to fundamentally deviate from the foundational work undertaken by Peter Gray, who created the original website, but now out of date.

Background

The first online edition of the Fourth Statistical Account of East Lothian Online ran to just over 600 pages, included over 400 images, and hundreds and hundreds of relative hyperlinks, footnotes, and tables. Page navigation largely followed the structure of the original printed edition, except the online addition had additional material not available in the printed, though this was (and remains) a little hard to distinguish. Arguably the site was tricky to navigate as the menus were not alphabetical and it was easy to get lost.

Transferring a book in 7 volumes online actually presents unique issues, which are not easy to resolve. Long chapters are a case in point. The site had not been designed around modern devices e.g. the need to adapt to screen size. Nor had the technologies been updated recently, presenting a potential security risk. The website pages were not served securely using the latest TSL technologies. It worked, but only just. The first problem we encountered was that we didn’t have access to the site server and no member of the society could correct, amend or update the website. But never mind that, we liked the idea of a challenge.

As the Society behind the creation of EL4.org.uk had been legally wound up, East Lothian Antiquarians felt it was time to find a more permanent and durable solution, administratively closer to home and technologically more secure for the foreseeable future.

The requirement was to give ELASFN full control and “local” management of the website, along with a modern template that would adapt well to different viewing devices, with improvements to navigation and search, as well as promotion of ELAS and its publications. A number of options were examined.

Options

Do nothing

We like doing nothing. Sometimes the simplest and most obvious and efficient solution can be to do absolutely nothing.

This was discounted for security reasons; the lack of a responsive template; the lack of site access by ELAS without reference to the webmaster, who was now working remotely in New Zealand.

Copy site files and database and replicate ‘as is’

While this options would give site access to an ELAS finding a suitable hosting arrangement willing to hold insecure content might be problematical. This was discounted as even if the site access problem were resolved, the site administration panels were not user friendly and we had found very cumbersome to use even for the relatively small ELAS website, so with 1000s of content items, ongoing management would remain unnecessarily complex.

An alternative would be simply to transfer the hosting account to ELAS, but again this would not resolve the technology problem as an update would be required anyway at some point soon (and there were several potential upgrade paths).

There would have been a need to recruit a competent individual to look after the website platform and hosting environment, but in all likelihood the site would atrophy, never get security updates, eventually become a security risk.

Integrate into the new ELAS site

On the face of it, this appears be a highly desirable solution, delivering full control and access to an ELAS member/s on a local platform, with the benefit of local and accessible support.

However, the current EL4 site is very large and integrating the complex navigation into the new ELAS site could easily create confusion for website users AND for website managers alike.

The ELAS template was selected for its simplicity and around an expectation that the site would be retained for the foreseeable future offering a relatively straightforward structure, with a few publications added every year as well as regular articles.

A simple integration is not impossible, but it would require the site to be recreated, perhaps with a new template, or taken offline for a few weeks or face the consequences of a cowboy import being undertaken ‘live’.

There might be additional work to separate or segregate the website admins from the EL4 area, so as not to cause confusion. Not impossible, but additional work nonetheless. Issues with the navigation of the current EL4 site would require changes to the ELAS template, or the adoption of a new one.

PDF scan each EL4 volume

The online version includes additional content, so this option would only partially meet the requirement, without the benefits of on-site search (although Google Search works well, it doesn’t navigate the user elegantly straight to the page in question). At some point this idea could be activated anyway, should not be excluded.

Create a separate microsite

By creating a microsite for el4.org.uk one would get all the benefits of quasi seamless integration (mainly in the backend), with a distinctive and unique template and prominent attribution of el4.org.uk as an ELAS inspired and hosted enterprise.

Hosting

In a practical sense it makes some sense to host the site alongside the current website, hosted by Our Locality, the local publishing platform. While this is not the only solution, it makes administration altogether simpler. An alternative hosting solution to consider would see the site sitting alongside the John Gray Centre, but with ELAS copyright and branding (they were quite keen).

This would require appetite and willingness of ELC to co-operate on the project and the technical migration. This would almost certainly take some time to execute and might not be a frictionless experience. Another option would be to create a virtual server specifically for this site, this would be desirable if the team anticipated significant growth in demand for the the sites’ content, or the settled view was that technical independence from the ELAS site or autonomy was more important than being part of an East Lothian community-orientated platform (the Our Locality option).

This would add a number of extra days to the costs, as the whole environment would have to be setup. While the recurring costs of virtual hosting are quite low, you need a mechanic on hand to manage and maintain it perhaps an hour or so a month.

Solution

We proposed migrating the EL4 website to a separate site on OurLocality, which uses the modern platform WordPress, employing a simple responsive template. The technology is to a very large extent future-proofed as WordPress is a tremendously popular and stable platform, with an extraordinary community of users. Content is very portable, so doesn’t preclude moving the site easily or exporting it to some other content management system in the future, which gives ELAS an insurance policy built in at low cost.

The work

The costs included creating an archive back up the old site (a safety precaution to ensure verification and resolve disputes), prior to  exporting the old site to html, parsing the html content to make it ready for the import stage, initially via an intermediate site, attending to broken links, manually adding the tabular information which was broken, manually fixing the footnotes and images, resolving duplicate menu items, creating new menu structures and developing & testing intelligent navigation structures, a small amount of templating and styling, creating a more attractive front page and redesigning the parish pages (and using existing images but more more creatively).

Task	Estimated Hours
Backup of existing, obtaining copy of code & database and archiving the old as html	1
Export as html locally (this step creates a full copy but without the database or the search facility and is a useful backstop and backup should access be delayed)	2
Parse the html to enable smooth WP import, namely so the page content has a unique id, and we can strip out the header and footer elements easily (we a use a text replace tool that can span multiple files at once)	2
Import html to WP on a local machine to do heavy lifting before the final import	2
Manually copy content to folder pages (blank interstitials due to the structure of the old site)	1
Fix titles to avoid duplicate title problem (when we import from html, there is no unique id so the importer gets confused over duplicate titles and fails – quite a lot of the titles were effectively the same in the parish pages – yawn)	2
Move image files, so that they are accessible from.the web	2
Export local xml file and change references to reference an image store somewhere on the web	2
Import xml to the web	3
Author headings – these should be styled differently (County chapters)	2
Fix tables that fail to import (around 100)	3
Fix menus to override alpha sorting in County section (around 600 items in toto)	2
Add side navigation menus to improve navigation (for County and Parish pages)	2
Non article pages – interstitial nav pages (remove hard links and add automatic navigation)	2
Fix footnotes, these used an old technology  (200 or so)	3
Regex broken links (there are around over 1000 internal and external links, which are assumed can be semi-automatically fixed … tedious)	2
Regex broken images (there are over 300 images, which should import correctly and the residual problems caught quickly)	2
On the final site – simple css only templating, using a free open source theme	4
Create a new parish pages navigation	2
Create new front page	2

While we are estimating around 80% of the content will import correctly, the remaining 20% will need manual attention, which is where the main migration costs lie. There was no separate contingency for problems identified along the way, except that built into the estimates. As most projects do, this one took twice as long.

Additional work included, the late discovery of a parallel website that duplicated most of the main site content, perhaps designed for a CD/DVD version. Unbeknownst to us this was ranking well in search results, despite being a potentially very major duplicate content hazard. Our last minute strategy was to redirect those old links via a rather large concordance table, though we could have just let the 404 or reported them for delisting. Over time a do nothing strategy would have seen them disappear quite quickly.

New Look

---

---

---

---