Publications
Byte-ing Off What You Can Chew: Electronic Records Strategies for Small Archival Institutions
Terry Cook
University of Manitoba
Winnipeg, Canada [Originally published in Archifacts April 2004]
Introduction
The archival profession is obsessed with electronic records.[1] Conferences, workshops, journal articles, and major research projects seem to address issues around electronic or digital records more than any other. Most of the presenters, examples, and rhetoric in these forums come from large archival institutions dealing with large governments or business corporations. Yet many archivists work in smaller archives, for smaller sponsors. What solutions can they embrace for electronic records? What priorities should they set? What arguments can they use to convince sponsors for action and funding? Where do they start in capturing, appraising, and segregating the records for eventual archival preservation?
Certainly a starting point is getting past a fear factor, and recognizing that the whole solution is not resting on their shoulders or actions. Indeed, I would confidently assert that no one is qualified to speak about electronic records with full authority. No one yet has the solution, the one answer, for ensuring the reliable creation or capture now of electronic records, or their maintenance and preservation as authentic evidence across centuries. Bill Gates does not have the answer, although we know now that major software companies are at last recognizing the dual problem of capture and preservation of transient electronic data, and are beginning to develop solutions. Archival electronic records researchers working within archival circles, do not, especially given the significant disagreements between them – although their research to date undoubtedly contributes some important pieces to the puzzle’s eventual solution. Margaret Hedstrom has very wisely suggested that, in fact, there should not be one solution.[2] There will be many answers depending on the size and complexity and type of digital information that is created; the organisational culture and partnership possibilities between the creating agency and the archive; and the size, maturity, staff competence, and technical and financial resources of the archive itself. In light of these circumstances, we therefore need, as implied in my title, to bite off what we can reasonably chew. And we will certainly need a vast array of tools, large and small, in our professional toolkit, rather than just ten identical hammers to beat the same nail over and over.
The point of this paper, then, is to suggest some introductory strategies that will enable small archives to cope with appraising electronic records, which used to be called machine-readable records, and now, increasingly, are called digital records, both "born digital" for those first created on a computer and "made digital" for those which began life on a paper, photographic, film, or sound medium and then later were scanned or otherwise converted to a digital format, usually for placement on a CD-ROM or website.
* * * * *
Horror Stories
Let me start with two graphic stories from recent months. In April, Eduard Mark, a senior historian with the United States’ Department of the Air Force, took a significant personal risk by publicly posting his thoughts on two listservs for historians of war and of diplomacy about the state of the federal government’s recordkeeping system. Readers should remember that he is describing the records of military forces which, with the police, are usually the twin bastions of the most efficient recordkeeping in society, because personnel there are trained to follow orders and procedures, much more than public servants working in more liberalised and individualistic bureaucracies, or even businesses. Keep in mind, also, that this recounting by Eduard Mark relates to the most powerful, well-resourced military force in the history of the world. It is a relatively long quotation, but its words are haunting even after numerous readings:
On the subject of the current state of federal records I will be brief . . . I hold . . . that the crisis is so great that I cannot in good conscience say nothing when an opportunity presents itself to call attention to the problem . . . The federal system for maintaining records has in many agencies – indeed in every agency with which I am familiar – collapsed utterly.
The basic reason for the collapse of recordkeeping is that the computer and electric records were introduced into the federal workplace in the 1980s in a fashion that destroyed the old system . . . . Before the computer every office had a typing pool. The typists made copies of their work – first carbons, then photocopies. These were collected and filed according to the agency’s rules and ultimately retired to the Archives. With the computer almost all officials became responsible for their own typing. Out went the typing pool and no other system for collecting documents took its place, whatever the regulations may say. The appearance of electronic mail has only compounded the problem in ways too obvious to require mention. In the Department of the Air Force virtually no records are saved except for what passes over the desk of the Secretary and the Chief of Staff and what the historians of my office manage to save for an annual history of the Air Staff (a history which, I should add, we have resumed writing only this year since abandoning it in 1993). So far as I have been able to observe a similar situation exists throughout the Department of Defense. In 1990-1991
I wrote a history of the invasion of Panama, which remains classified. I began my research within weeks of the operation and found that many electronic records had already been purged from computers – not from calculation or malice but because the culture of recordkeeping had even then so broken down that the staff officers involved had no idea at all that they were supposed to preserve records. In the years since then several other agencies . . . have approached me to see if I had copies of various records of theirs relating to Panama and others matters, they having lost all their own copies.
I will mince no words. It will be impossible to write the history of recent diplomatic and military history as we have written about World War II and the early Cold War. Too many records are gone . . . I have long since grown weary of historians who forever and anon bleat about access to still classified records from the 1950s but who remain obstinately deaf when told of the daily hecatomb of contemporary records. When in decades hence they go to the Archives and find decades compressed into single boxes they will not be able to claim they were not warned. History as we have known it is dying, and with it the public accountability of government and rational public administration.[3]
Despite much advocacy and significant research by some archivists in the past decade or so, that revolution that Eduard Marks laments still has not permeated very deeply into the archival profession in terms of establishing viable, practical, operational programmes for the actual capture, contextual description, appraisal, acquisition, processing, and preservation of archival electronic records. Meanwhile the technology marches relentlessly onward.
Will our generation of archivists mostly still mired, alas, with Paper Minds as they face Electronic Records, be seen, one hundred years from now, as vandals of neglect for not forcing this issue more aggressively, starting right now, even in small archives?[4] Will we be viewed as the modern equivalents of those barbarians who destroyed classical Rome or burnt medieval monasteries, thereby wiping out generations of records, history, and civilisation, because of our continued inaction? We knew, but we did not act, or did not act wisely or decisively, or did not change priorities (and funding) from our comfortable curatorial practices to meet this recordkeeping crisis?
The challenge to each archivist is the same: to ask of her or himself what each will do differently starting next Monday morning in his or her archival jurisdiction – whether large or small – to avoid that charge, and to help rebuild that "broken culture of recordkeeping" of which Eduard Mark speaks so poignantly? What two or three practical steps can we take right now, even if the large overall solutions are still some time away. How can we now stop the blood-letting, even in small archives? How can we start making a difference rather than continue making excuses?
Small archives I would define as an institution or programme with one or two archivists, with perhaps some access to support staff, perhaps these shared with other operational units, plus an occasional summer student or volunteer helper. Often, the small archive is a one-person shop – the classic Lone Arranger. This seems a shared assessment of the nature of the small archives in New Zealand as it would be in Canada. Small archives so defined exist (in Canada at least) for schools, universities, and colleges; religious orders, dioceses, and churches; businesses, museums, and galleries; non-government lobbying, charitable, cultural, or ethnic organisations, and local or regional governmental authorities. For all these sponsors or jurisdictions, computers are now very widely in use, or they very soon will be. Yet small archives often do not know where to start in reversing Eduard Mark’s lament that computerised records are not being captured in reliable recordkeeping systems at the point of creation or contemporary use. Archivists from small institutions have often complained to me in various workshops that they are overwhelmed with too much to do, pulled in too many directions, with too little funding, and without an IT (information technology) infrastructure, and so starting up an electronic records programme just seems beyond them – usually coupled with the assertion that we folks from big archival institutions simply do not understand their plight.
I have a fair degree of sympathy with this, but also some lack of patience, because for large archives as much as smaller ones, here is the stark bottom line: unless you can get substantial new financial and human resources, you will need to stop doing something important that you are now doing, and reallocate significant resources to electronic records, period. There is no other way. That requires, in the first instance, an act of personal will and professional commitment, not technological infrastructure or digital expertise. The will to change must come first, and I suggest that it will be your hardest decision.
However, small archives do have certain advantages as they contemplate such a change. The big archives that pioneered archival electronic records programmes in the United States, Canada, and later Australia, Britain, and Netherlands, and in some of their states and provinces, did not have the models of successes and failures, the results of major archival electronic records research projects, and the burgeoning archival literature on the subject that small archives’ archivists can now easily access. While archivists in larger archives do have colleagues down the hall to share ideas and develop best practices as teams, those in small archives now have access to the incredible resources and international conversations of the internet, with hundreds of relevant sites, created by archivists and recordkeepers in many jurisdictions, containing models, policies, standards, bibliographies, case studies, and research results, as well as listservs, to all of which the electronic-records pioneers in big archives did not have access, save by rare international travel. The internet with e-mail also allows those in smaller archives to build inter-institutional virtual teams, and to engage in co-operative ventures, perhaps building a shared IT infrastructure or service bureau contract for computer processing where several small archives could pool together their needs and resources. In short, archivists in small archives need not be in the isolation many now feel. In fact, they have the great advantage of being now the third generation of archivists dealing with electronic records, with many useful precedents already established.[5] What no archivist has, in large or small institutions, is the luxury of time to remain inactive, hoping for some magic solution to appear, or to wallow in self-pity just because they are in a small institution.
With readers now thoroughly offended, here is my promised second graphic story, which is much shorter, coming from this past July, and needing no further introduction:
In 1986, the British Broadcasting Corporation created the Domesday Book Mark II, an electronic version of the original record of English lands that was written at the instigation of William the Conqueror in 1086 [nine hundred years earlier]. The BBC’s version contained 25,000 maps, 50,000 pictures, 60 minutes of video and millions of words. It cost 2.5 million pounds to create.
Only 17 years after its creation [in 2003], the Domesday Book Mark II can’t be read. The BBC computers used for the project no longer work and the disks on which it was stored are not readable by other computer systems. But the 917-year-old original is still available to researchers in London’s Public Record Office.[6]
This story underlines the second great archival challenge with electronic records: preservation across time. Whether made-digital records, like the Domesday Book, that so many archives are busily creating by scanning originals from their collections, or born-digital records like Eduard Mark’s office systems in the US Air Force, once these digital records have been created or captured, they must then be preserved across centuries – across centuries when each has an optimistic shelf life in digital format of perhaps 20 years before significant archival intervention is needed to refresh, migrate, or emulate the record to new formats, before it either disappears as unreadable or self-destructs physically.
The first axiom of recordkeeping from an archivist’s perspective, then, is a simple truth of continuum thinking: the better the record is created and captured; the richer the contextualised metadata or description surrounding the record during its pre-archival use; and the more astute the appraisal "keep-destroy" decisions; then the easier the long-term preservation of the record – or digital object – will be, once it is later transferred to an archival institution, or at least under the control of the archives. But even a perfectly captured and maintained digital record still presents preservation challenges across a time-frame of centuries which must always be the archival perspective.
At this point I offer the somewhat controversial assertion that the main focus of small archival institutions for the next ten years should be on recordkeeping activities up front, including appraisal, and less on the preservation of archivally valuable digital records across the centuries. Why? Four main reasons. First, if the archivist with other recordkeepers does not intervene effectively in the up-front or inner dimensions of the records continuum, there will be few reliable or trustworthy records created that will be worth preserving at all in archives. Secondly, effective appraisal (especially functions-based macro-appraisal) allows the archivist typically to authorise the destruction of 95% or more of the total record, including the electronic record, and thus reduces the amount (and costs) of digital "stuff" to be preserved by an equivalent large percentage.[7] Thirdly, effective recordkeeping regimes "up front" should include encapsulating electronic records with rich descriptive and functional metadata, including instructions for conversion to, or creation in, effective standardised formats, like XML, that will – if done properly – very much aid later preservation of such encapsulated digital objects, if not for centuries, at least for decades by your archives. Finally, solutions for preserving digital media without loss of information through changes that will have to be made to the "original" record, because of fragile physical storage mediums and technologically obsolete software instructions, will not come from efforts made by small archives, or indeed by any one big archives. Multi-million dollar projects using super computers and strategic precedents hard-won by national and international research consortia will provide answers, that once generalised into available commercial software will give individual archives the preservation strategies and standards they need. Preservation, then, should be limited now to storing electronic records transferred to archival control, whether in native or original software format like Word or Power Point or Lotus, or in software-independent formats like ASCII, TIFF, Rich Text Format, or XML, and maintaining these, in those same formats with their software, in appropriate environmental, refreshing, and handling conditions. The definitions, standards, and procedures for such conditions are readily available from numerous web-based sources and need not detain us here.
* * * * *
Strategies and Solutions
Turning then to explore some strategies and solutions for the pre-preservation, recordkeeping dimensions of digital media, and with apologies to Gertrude Stein, the first thing to grasp for beginners in small archives is that an electronic record is not an electronic record is not an electronic record. There are at least four distinct organisational environments (or cultures) in which digital media are created, and each involves very different players with different perspectives and expertise. Let us look at these four communities.[8]
First, there is the traditional world of structured data, the systems world, the big databases, often located on mainframe computers or powerful workstations. This was the primary focus of the first generation of the machine-readable records archivists of the 1970s and 1980s, yet such systems have not gone away, but have evolved into multi-functional relational database systems often containing mission-critical information about the organisation, and its clients, as well as payroll, finance, inventory, and general administrative data. There are generally clear policies and good standards in place for the very efficient management of the data in these systems, but rarely for issues of data alteration, deletion, retention and disposal, and archival transfer. The focus is on reliable, accessible, efficient, up-to-date data, not time-bound records wrapped in context-rich transactional metadata reflecting changes over time. Data in these systems is continually updated, altered, merged, and deleted, which in effect, in all such cases, destroys the previous version of the data, or record. Such changes under most archival legislation amount to de facto illegal destruction without authorisation, yet records managers or archivists rarely challenge this reality, despite it being their responsibility to do so. The systems world is composed of IT and database management specialists, where records managers, let alone archivists, are rarely consulted or involved as part of the normal business practices of that world.
The second electronic records environment is the unstructured world of the automated office – the PCs and desktop computers, whether stand-alone or connected in one or several internal networks through centralised servers. This is the automated workplace with which we are all familiar, which seemed so pioneering in the mid-1980s and which had become so pervasive by the 1990s – the world of e-mail, word processing, spreadsheets, small end-user databases, graphics, presentations, automated calendars and address books, and all the other features common to "office suites" of integrated software. This is also primarily the world so out of control that Eduard Mark laments for the US military. This is the world that my former National Archives of Canada colleague, John McDonald, has characterised as the "wild frontier," where anything goes, with few rules and no corporate-wide perspective on managing the records in these office systems, a process not helped by massive resource cuts in "support" areas like records management in the 1990s, as governments retrenched and downsized, at the very time that these new automated office tools were being most widely implemented.[9] This second environment is the world facing the second generation of electronic records archival pioneers, as they have attempted, from the early 1990s, to impose some order for the chaos "out there." Examples include the pioneering Pittsburgh Project, New York State, Indiana University, University of British Columbia, Monash, Michigan, Cornell, and similar research projects, as well as major research and implementation initiatives at National Archives in Ottawa, Canberra, London, Washington, The Hague, and now Wellington, at least as reported in the English-language literature. All these projects attempt in different ways, often using different assumptions and theories, to discover and promulgate concepts, policies, rules, guidelines, prototypes, and standards that will turn the unstructured data and diffused information of the individualised desktop of this second generation or environment, into a structured, well-controlled, corporate-wide resource of contextualised, trustworthy records that will serve as reliable evidence for business operations and for legal and public accountability, and, of course, from a longer perspective, that will also serve as the basis for a fuller, richer archives, rather than the bleak a-historical wasteland that Eduard Mark sees before us.[10] In this second environment of the automated office, the primary recordkeeping contact for archivists is with the records management community, although many of that community’s policies and standards still address (or reflect) paper records only. While there is now a growing and impressive number of good recordkeeping tools available for controlling this second generation of electronic records, and model policies and best-practice guidelines exist aplenty, as a result of the many research projects and field trials noted above, it is still sadly a big struggle to convince senior management that these policies and tools are worth the expense of implementation, and then implementing them consistently as an integrated part of all work processes and business functions, across the entire organisation, using retrained or newly acquired staff capable of doing this job. And so, too, is it a struggle convincing many records managers and archivists that they have to abandon their comfortable Paper Minds and accept, indeed embrace, the challenge of managing this automated office information across the whole continuum.
The third environment is that of the World Wide Web, often building elaborate digital interfaces for the many publics that your sponsoring or parent institution wishes to reach with its message, products, holdings, or services. Websites can draw information from both the previous two worlds of large databases and the automated office, and then convert and export these, redesigned and connected in multiple ways, for viewing on a website, whether one restricted to internal business use only, for a limited number of external partners (such as suppliers or registered clients), or for the entire world. The key communities for this third environment are communications, publishing, and public relations people, and often the departmental librarians, who together usually manage the website(s) for an institution. There are frequently good policies in place governing web masters and web content, and senior managers are certainly sensitive to the external web-based image presented of their institution and the possibilities of web-based enterprise and profit. However, few in this community recognise the relevance of recordkeeping approaches for the website, issues of authenticity and accountability for its content, or requirements for retention, disposal, and long-term preservation, even though web content has rapidly evolved from publicity and image to include business transactions and operational guidelines. As with the systems world, the first environment above, records managers are rarely consulted or involved as part of the normal business practices of the web world, and so again, archivists are also left out of the loop.
The fourth electronic records environment encompasses specialised digital applications of numbers one (the systems world), and especially two (the office suite software world), that in turn are frequently partly re-presented in number three via the website: examples include digital film and photography, special digitally designed software graphics for such functions as mapping, architecture, engineering, design, or forms management; special database applications like geographical information systems, records management or library control systems, and so on. These specialised applications tend to be attached to the operational unit directly involved and are found scattered throughout the organisation. Not surprisingly, these applications share all of the problems already outlined for one or more of their three originating environments, and, save for records management software, attract very little contact with either records managers or archivists.
So, my essential point is that the electronic or digital records of the parent, sponsoring, or other organisations, groups, and associations, that the small archives will appraise and acquire, involve many communities that do not talk to each other, and, save for some recordkeeping initiatives beginning in the second world of the automated office, these communities have little awareness and less interest in addressing recordkeeping issues: authenticity, metadata contextuality, retention periods, authorised and systematic disposal, and archival transfer. Three of the four categories rarely fall under the operational purview of the records manager, and for the fourth that does, this presents challenges so immense that those records managers, even with some answers, do not get the senior support or resources needed.
* * * * *
What to do now that we are facing this bleak environment where, make no mistake, as Eduard Mark details, the records of tomorrow’s history and today’s accountability are de facto being destroyed every second by our collective failure to intervene effectively? The first thing is for the archivist in small institutions – and I would assert in any-sized institution – to search for partners, and to identify and then build bridges between these four separated communities of systems and IT people, records managers, web-content communications and PR experts, and specialist groups of users. Here the archivist may be able to act as an honest broker to influence bringing these four communities together so that a corporate-wide approach to electronic recordkeeping is developed in your sponsoring or parent institution. The growing appointment of Chief Information Officers (CIOs) in government and business is a welcome initiative to unify these disparate environments. Beyond these four core communities of specialists closest to the creation and daily management of the records, there are other, natural allies. Auditors, programme evaluators, accountants, lawyers and legal advisors, freedom of information, privacy and copyright officers, ombudsmen and similar people, all have a vested interest in having accurate, reliable, comprehensive, timely, retrievable, usable, and secure records – not forgetting either your potential allies of other archivists in other smaller institutions. Often there will also be experienced programme or "line" managers who will be interested because they have been burned in the past by not having such reliable information at hand when they needed it, as well as naturally any senior CIO that may be present.
Cook's Top Ten
Once you have this alliance, this partnership in place, what is the message that you deliver to both senior managers and every line worker? It is that contextualised, reliable records are essential to the organisation for positive or work-enhancing reasons and for negative or risk-avoidance reasons. This message we have to sell, uncomfortable as that may make the more traditional "paper minded" custodians among us who prefer to be processing old records in the stacks. For such selling, I commend Cook’s Top Ten List of why records are important to any organisation, which you should customise for frequent repetition to your partners and the senior managers, in person, in print, in handouts, in screen-savers, in training – not once and you’re done with it, not twice, but over and over and over again.[11] Records are important to your organisation in order:
1 To support your own and others’ ability now to make good decisions based on reliable information, and thus to deliver effective programmes and services;
2 To help you to document how or why decisions and actions were undertaken on a particular case or project, thereby permitting you and the department to meet the accountability requirements and performance benchmarks associated with sound decision-making, the achievement of stated goals, and the use of institutional resources, whether these come from the public, legislators, shareholders, or private wealth;
3 To provide you with information about similar previous projects and past activities, in order to help you avoid "re-inventing the wheel," thus making for better decisions now in similar cases;
4 To save (as a corollary of number one and number three), significant staff time and thus monetary resources, since data exists to show that professional and managerial staff spend 20 to 50% of their time searching for misplaced or missing information, when a reasonable amount would instead be 10%.
5 To provide the corporate memory and operational continuity required to permit those who follow you to understand how your programmes and services were developed and delivered;
6 To permit you and the department to meet the requirements of various laws, regulations, and policies, that necessitate good records be created and maintained, as, for example, laws governing workplace health and safety, environmental quality, taxation, employment equity, human rights, and many more, including permitting you to be in compliance with archives laws and regulations, and international records management standards,
like ISO 15489, that require effective recordkeeping, retention, destruction and disposal authorisation;
7 To ensure the protection of legal rights and moral entitlements of citizens, members, clients, customers, and other relevant publics, including the public’s right of freedom of information and the protection of personal information, which are guaranteed rights in law. The failure to facilitate or comply with rights and entitlements, because of poor recordkeeping or lost or destroyed records, could lead to negative public relations nightmares or a backlash against your organisation;
8 To enable your department to implement effective security and disaster readiness or recovery for its information resources against sabotage, technological failure, or accident;
9 To allow your department’s programmes, activities, and decisions to be understood better by New Zealanders at large, thus strengthening public support for your activities, and for the general process of public administration in a democracy, thereby reducing cynicism and detachment; and
10 To contribute to the archival record so that your, and your institution’s, role in the historical process, is not lost, and the country’s (or region’s, city’s, church’s or university’s) heritage and culture is enhanced now and for future generations.
Note that only number ten and a passing reference in the second part of number six mention archives at all, or the heritage, cultural, and historical purposes of long-term preservation. All the others are directly relevant to the organisation, or required of it, morally or legally, even if no institutional archives existed at all for the jurisdiction. The point is to make good recordkeeping appeal to their self-interest, in order to set up a win-win situation for the creator and the archive. But do not be afraid to remind them that the organisation is dangerously exposed legally and morally, in public relations and profits, if guilty of not exercising due diligence by changing its decontextualised transient data and its scattered unmanaged information into reliable, corporate, secure records.
Seven Step Plan
Why have I spent so much time on these so-called "soft" people and "soft" environmental issues? Recent research suggests that the electronic records dilemma will not be solved so much by implementing some electronic document management system software, although that is a useful step, as it will by solving the people and commitment issues. Surveys and discussions with recordkeepers in the "real world" indicate that authenticity of records is achieved more by policy statements, best practice examples, international ISO standards, consistent and normal business practices, and committed people and managers, in a positive organisational culture, rather than by software magic based on theoretical precision.[12] After all, even the best electronic document management software only contains an empty series of boxes or templates that must be filled in by record creators – consistently, accurately, reflecting corporate classification and work processes within the relevant business functions. For recordkeeping software, as for any other computing application, the now-classic mantra applies: garbage in equals garbage out. That is not a technical issue, or message, therefore, but one relating to organisational behaviour and organisational culture, and its capacity for change. Within the modern workplace, we need to re-create Eduard Mark’s missing "culture of recordkeeping." Unless you do that, your electronic records programme in a small archives, as much as in a large one, will be limited to an ad hoc collection of snap-shots (one-time downloads) of data from compliant corners scattered across the parent or target organisation. While this is not utterly worthless, it is far from meeting the requirements for authentic and reliable records as trustworthy evidence that our researchers require, or for constructing the archive that posterity will rightly expect as a legacy from our profession. So, back to those practical action steps for you for next Monday morning, to help stop the bleeding that Eduard Mark laments. Here are seven possibilities for you to try, perhaps a lucky seven?
1 The very first thing you do next Monday morning, after taking a deep breath, is consider how to build these effective alliances I have been mentioning and how to tailor and then deliver a message – your customised version of Cook’s Top Ten List above – that will generate a commitment in parts, and later all, of your parent or sponsoring organisation to facilitate a recordkeeping culture. And while you are at it, start rearranging your priorities: do we really need better indexes, right now, of paper or photographic records safely in our custody, while Rome burns?
2 Make a commitment to recognise and then adopt (or learn about so you can) the power of function- or work-process-based macro-appraisal in your archives, which is fully consistent with functional classification of records based on a DIRKS-like methodology. The rich contextualised information that the archivist or recordkeeper uncovers through her or his research in doing macro-appraisal creates a body of knowledge about organisational behaviour that can very much assist the records creator to establish a good functional and work process-based records-classification system, simply because the core task in macro-appraisal is to understand the functional whole into which all the organisation’s programmes, activities, and transactions fit. In good continuum manner, researching macro-appraisal’s functional knowledge does not just provide essential contextual knowledge for the archivist to make good, defendable "keep-destroy" decisions, or offer better access points and context for our researchers at the so-called "back end", but also, potentially, to records creators and users at the "front end". Such front-end functions-based knowledge will leave us with much less garbage-in garbage-out among the recordkeeping systems employed by our creators. Remember too, that macro-appraisal is media-neutral, and so the appraisal of electronic records is less the issue for you as appraisal of organisational functions, activities, and processes, in all media. E-mail is a tool, just like the typewriter. We do not appraise typewriters (at least not as archivists!). We macro-appraise the functions, programmes, and business activities that typewriters and e-mail technologies support.[13]
3 Recognise on Monday morning that there are vast quantities of digital "stuff" that have no archival value, and very little corporate value after a short period of time, just as in the paper world. By granting your sponsoring agencies authority to destroy normal administrative practice electronic records, or general administrative electronic records, as these are called in various countries, you eliminate some 40 to 50% or more of the total record; you permit your agency to save costs and gain efficiencies; you allow your agency to be in compliance with various laws, regulations, and policies as outlined earlier; and you assert your presence and interest – and your credibility – in helping to manage their electronic records, and reduce their burdens. Expand this authority beyond administrative records to so-called transitory records created for all operational functions – such as copies, working drafts of documents, reference manuals, research and downloaded materials, personal stuff, system backups once superseded, and old batch and audit tapes – and by doing so, you once more allow for the generic destruction of large quantities of digital information while doing your sponsor a favour, and again building your own credibility, and not incidentally your own contacts, confidence, and comfort level with digital media.
4 Recognise that you do not now have the infrastructure or capacity of acquiring many types of data and their subsequent processing, preservation, and access. Pending that IT and human capacity in your own archives, or a regional alliance of small archives, or even when once acquired using the capacity in the most efficient way, you negotiate leaving electronic archives appraised as having long-term archival value in the custody of their creator, with appropriate preservation guidelines and monitoring. This is especially relevant to mission-critical databases, to cumulative and longitudinal data not subject to change or alteration, to data requiring very expensive and proprietary software licences, or to data still subject to very heavy reference use by the creator. There are models and standards for such distributed custody of electronic records, including one that has been operational for a decade at the National Archives of Canada.[14]
5 Your principal, specific action-step Monday morning in the database world, the first of our four environments, is to capture transactions of system processing rather than just taking annual or periodic snapshots of the data. Old data is not bad or redundant data, as the IT world believes, but evidence of change and programme activity. When an immigrant’s or taxpayer’s occupational code changes on 14 July 2003 at 2:57:04 p.m. from 39 (plumber) to 67 (teacher), that change – and when it occurred, under whose authority – needs to be captured, not just blown away into cyberspace. To do so means you have a time-bound recordkeeping system rather than merely an accurate-at-the-moment-only information system. Such a change, from plumber to teacher, without retaining the sequence of such changes over time, is a de facto illegal destruction of the portion of that person’s case-file record. For the vast majority of systems and data elements, this is not important – we do not need a record of six previous telephone numbers. But for those that have long-term value, as a whole, or as a part, for those data tables within the system that document processes and activities that have been appraised as archivally important, or as samples, extracts, or aggregates, such recordkeeping functionality must be insisted upon by the archivist.
6 In the second environment of the automated office, pending the implementation of DIRKS-like functions-based classification, and of records management software that has automatic triggers to dispose of records when their retention needs are finished, and to transfer to archival control automatically those records relating to work processes and activities already appraised as having long-term significance, pending that "ideal," there are three useful short-term action steps. One is to have workers create tree-like directory structures based on the 10 or 15 major programmes and their activities and sub-activities for their functional area of the organisation, that would be consistently named for all workers across that unit, and to file all documents – e-mail, word processor texts, spreadsheets, data, graphics, images, etc. – in that structure. There would also be folders – again consistently named – for transitory records and for non-corporate, personal material, thus allowing these to be easily segregated and destroyed. A second step is to implement within this directory structure a consistent naming and dating convention for all office documents that designates and distinguishes a memorandum from a report from a letter from an e-mail, spreadsheet, presentation, or downloaded research material, as well as version control and fixed dating. And the third step is to ensure that any electronic document management software adopted by your record manager allies not only supports all these conventions, but also includes automated retention, disposal, deletion, and archival transfer functionality.
7 In the web-based environment, as well as that of specialised applications like digital photography, GIS, and so on, the message is similar to that given to the database managers: maintain all older versions of the website pages appropriately, time and date stamped, allowing for a reconstruction of the site as it was seen by both the organisation’s workers and its clients at any given time when they made decisions based on its ever-changing content. Of course, this again applies only to those portions of the website appraised as being of long-term value to the organisation or to its archives.
These seven steps are not an ideal recordkeeping regime, but they are, in my view, practical and achievable steps in that direction, and they will staunch the bleeding of records and evidence now going on. They do not require awesome levels of technical expertise, but rather people skills and recordkeeping knowledge, and good archival research skills into functional context, administrative history, and macro-appraisal thinking. In adopting such approaches, you as recordkeepers will do much more good rather than harm, as we await more standardised, commercial, and lasting solutions.
I began trying to inspire you with specious – perhaps spurious – comparisons between archivists of our generation and ancient barbarians destroying the records of civilisation. Let me end by returning to this image of civilisation. The National (then Dominion) Archivist of Canada, Sir Arthur Doughty, said in 1924, in a phrase that appears on posters and coffee mugs found in many archivists’ offices in Canada, as well as being carved into the base of the only statue ever officially raised in Ottawa to honour a civil servant, that, ‘of all national assets, archives are the most precious. They are the gift of one generation to another, and the extent of our care of them marks the extent of our civilisation.’[15]
What digital gifts, to evoke Sir Arthur’s phrase, will you, each of you, be leaving to the next generation? And how civilised will be judged your care of them? This is not an academic question, but, rather, an immediately relevant issue about your heritage here in New Zealand, your culture, your identity, your history, that is now critically threatened.
Society, collectively, has entrusted us as archivists the task of preserving these very things on its behalf. Will you let it down? You need first to decide that over the weekend before you start back at work on Monday morning!
Endnotes
1 This paper is a reworked version of a speech I gave to the Twenty-seventh Annual Conference of the Archives and Records Association of New Zealand, held in Dunedin, 4-6 September 2003. I have deliberately retained the conversational tone of the original presentation, but added more context and some suggestive (but not exhaustive) footnotes. I wish to thank Stuart Strachan especially, and my sponsors (ARANZ, Archives New Zealand, and the Canadian High Commission in Wellington), for so efficiently arranging my attendance and for the warm hospitality they and other New Zealanders offered to me.
2 See Margaret Hedstrom, "Building Record-Keeping Systems: Archivists Are Not Alone on the Wild Frontier," Archivaria 44 (Fall 1997): 44-71; as well as her "Electronic Archives: Integrity and Access in the Network Environment," American Archivist (Summer 1995): 312-24.
3 Eduard Mark, posting on 24 April 2003, to H-WAR discussion network listserv, under the subject heading, "COMMENT: Collapse of the Federal Recordkeeping System," at: http://www.h-net.org/logsearch/ [Accessed 3 March 2004].
4 The allusion here is to a speech I gave ten years ago on my first visit to the Antipodes in 1993, and published the following year. See Terry Cook, "Electronic Records, Paper Minds: The Revolution in Information Management and Archives in the Post-Custodial and Post-Modernist Era," Archives and Manuscripts 22 (November 1994): 300-29. While there has been significant progress in the intervening decade, not enough has occurred inside and outside the recordkeeping professions, as Eduard Mark’s lament clearly shows, to alter the overall analysis I made then, or the factors for it, some of which Mark now rehearses.
5 On the nature and character of the first two generations of archivists, dealing with machine-readable and electronic records archives, see Terry Cook, "Easy To Byte, Harder to Chew: The Second Generation of Electronic Records Archives," Archivaria 33 (Winter 1991-92): 202-08, and passim. See also on such characterization, Richard Cox, The First Generation of Electronic Archivists in the United States: A Study in Professionalization (New York et al.: The Haworth Press, 1994).
6 Gary Frost, "Digital Domesday," in the on-line journal, futureofthebook.com: preservation and persistence of the changing book, first posted 28 July 2003. http://www.h-net.org/logsearch/ [Accessed 3 March 2004].
7 My keynote address to the ARANZ annual conference in Dunedin, which preceded by a day the address on electronic records that is the substance of the present article, was entitled "Two Solitudes?: Appraisal and Access in the Postmodern Archive." The essence of that address, and my latest thinking about macro-appraisal, is being published as "Macro-appraisal and Functional Analysis: Documenting Governance Rather Than Government," Journal of the Society of Archivists (UK) 25 (forthcoming Spring 2003). I first articulated macro-appraisal in 1989-90 while writing The Archival Appraisal of Records Containing Personal Information: A RAMP Study With Guidelines (Paris: International Council of Archives, 1991), and later used its insights to develop, with the help of colleagues, the macro-appraisal programme at the National Archives of Canada, where it was formally launched across government in 1991 as the intellectual core of a new planned approach to the disposal of records. I was later the director responsible for the appraisal and disposal programme for government records in all media from 1993 to 1998, when I left the institution. The earlier articulations of the theory, strategy, and practice of macro-appraisal include Terry Cook, "Mind Over Matter: Towards a New Theory of Archival Appraisal," The Canadian Archival Imagination: Essays in Honour of Hugh Taylor, ed. Barbara Craig (Ottawa: Association of Canadian Archivists, 1992), 38-70; Terry Cook, "‘Many are called but few are chosen’: Appraisal Guidelines for Sampling and Selecting Case Files," Archivaria 32 (Summer 1991): 25-50; Richard Brown, "Macro-Appraisal Theory and the Context of the Public Records Creator," Archivaria 40 (Fall 1995): 121-72; and Richard Brown, "Records Acquisition Strategy and Its Theoretical Foundation: The Case for a Concept of Archival Hermeneutics," Archivaria 33 (Winter 1991-92): 34-56. For placing macro-appraisal within Australian continuum-based thinking about recordkeeping, see Terry Cook, "Beyond the Screen: The Records Continuum and Archival Cultural Heritage," Beyond the Screen: Capturing Corporate and Social Memory, ed. Lucy Burrows (Melbourne: Australian Society of Archivists, 2000): 8-21. For situating macro-appraisal within the history of archival thinking about appraisal, see Terry Cook, "What is Past is Prologue: A History of Archival Ideas Since 1898, and the Future Paradigm Shift" Archivaria 43 (Spring 1997): 17-63; and within the postmodern archives and our conditions of postmodernity, see Terry Cook, "Fashionable Nonsense or Professional Rebirth: Postmodernism and the Practice of Archives," Archivaria 51 (Spring 2001): 14-35, especially 30-35.
8 The idea of these four sectors builds on my "generations" approach, where each generation of electronic records is overlapped, not replaced, by its successor(s), (see note 5 above), but also borrows in its details and examples from some course materials shared with me by John McDonald, my long-time National Archives of Canada colleague, and now an independent (and international) information management consultant, who I gratefully acknowledge.
9 John McDonald, "Managing Records in the Modern Office: Taming the Wild Frontier," Archivaria 39 (Spring 1995): 70-79.
10 To give citations for all these national, state-based and university-centred research projects into identifying, capturing, and preserving authentic, reliable, usable digital records would amount to article-length footnote. Every project mentioned has both published articles, sometimes books, and rich websites with reports, studies, and models. The diligent reader will easily be able to locate these.
11 I am happy to acknowledge again John McDonald’s input to some of the factors on this listing.
12 On this, see the important conclusions from the InterPARES project that the authenticity of electronic records is not assured, as earlier assumed, through diplomatics theory, or related templates or models, but "in all cases . . . mainly through procedural means." Furthermore, "the diplomatic approach, with its focus on the individual record or document, and the functional analysis approach (as in macro-appraisal or computer systems design), with its focus on business processes, can both yield valuable insights, but are incompatible as complementary research approaches." The majority of world recordkeepers through international records management standards have opted for a DIRKS-like functional analysis approach that I am advocating with macro-appraisal. See Anne J. Gilliland-Swetland, "Testing our Truths: Delineating the Parameters of the Authentic Archival Electronic Record, American Archivist 65 (Fall/Winter 2002): 206, 210 for quotations, and passim. For some parallel conclusions based on real-world practitioners, see Eun G. Park, "Understanding ‘Authenticity’ in Records and Information Management: Analyzing Practitioner Constructs," American Archivist 64 (Fall/Winter 2001): 270-92. Margaret Hedstrom anticipated such conclusions some years earlier (see note 2 above).
13 For a flavouring of published case studies of actually implementing macro-appraisal (and thus complementing the more theoretical and foundational references in note 7 above), see Jean-Stephen Piché, "Macro-Appraisal and Duplication of Information: Federal Real Property Management Records," Archivaria 39 (Spring 1995): 39-50; Catherine Bailey, "From the Top Down: The Practice of Macro-Appraisal," Archivaria 43 (Spring 1997): 89-128; Candace Loewen, "From Human Neglect to Planetary Survival: New Approaches to the Appraisal of Environmental Records," Archivaria 33 (Winter 1991-92): 87-103; Jean-Stephen Piché and Sheila Powell, "Counting Archives In: The Appraisal of the 1991 Census of Canada," Archivaria 45 (Spring 1998): 27-43; Sheila Powell, "Archival Reappraisal: The Immigration Case Files," Archivaria 33 (Winter 1991-92): 104-16; Ellen Scheinberg, "Case File Theory: Does It Work in Practice?," Archivaria 38 (Fall 1994): 45-60; Jim Suderman, "Appraising Records of the Expenditure Management Function: An Exercise in Functional Analysis," Archivaria 43 (Spring 1997): 129-42; and Peter Botticelli, "Records Appraisal in Network Organizations," Archivaria 49 (Spring 2000): 161-91. On functional analysis and the DIRKS methodology, see Catherine Robinson, "Records Control and Disposal Using Functional Analysis," Archives and Manuscripts 25 (November 1997): 288-303; as well as Paul Sabourin, "Constructing a Function-Based Records Classification System: Business Activity Structure Classification System," Archivaria 51 (Spring 2001): 137-54. A number of other case studies of implementing macro-appraisal are now being edited for publication. For a comparative perspective, see John Roberts, "One Size Fits All? The Portability of Macro-appraisal by a Comparative Analysis of Canada, South Africa, and New Zealand," Archivaria 52 (Fall 2001):
47-68.
14 For this policy, its background, and cross-references to other relevant archival literature, see Terry Cook, "Leaving Archival Electronic Records in Institutions: Policy and Monitoring Arrangements at the National Archives of Canada," Archives and Museum Informatics 9 (1995): 141-49.
15 Arthur Doughty, The Canadian Archives and Its Activities (Ottawa: Public Archives of Canada, 1924), 5.