Archiving

[ned c]

This is really an extension of the discussion which is on the Sarnia thread but addressing the interesting subject of archiving.

I archive for two reasons; a master from which future copies can be made and protection of video/audio assets for possible further use. .mpeg2 is fine for creating simple masters from which future DVD copies can be made but is not a good CoDec for archiving assets. I agree with Michael that ProRes HQ files stored as data along with the original camera material makes future work easy; OK like Michael I work on a Mac so I assume some other format will be used by PC users. I use external HDDs for most archiving but for smaller projects I use BluRay discs storing the information as data. There are now triple layer BD discs with a 100GB capacity; not inexpensive but will store one of my largest projects.

I also have a mountain of miniDV camera assets and masters in both DV and HDV and as I have a deck can play them but obviously if they are to have a future will eventually have to be transfered to a more appropriate archive media.

ned c

[Dave Watterson]

As with all electronic archive processes ... the trick is to keep copying them onto the "newest big thing" in storage media every couple of years.

a) hard drives all fail, no exceptions.
b) all home-burned DVDs, Blu-Rays and CDs fail eventually, no exceptions.
c) the kit to play back "outdated" storage media vanishes very quickly into the junk yards.

For those operating as a business it may be worth preserving assets for future re-use. For films of family there may be a purpose in preserving images for the next generation. But for the other stuff?

Who wants our documentary on this, or our travelogue about that, in five years time, let alone a hundred years time? As for our club "film in an evening" ...

Dave

[Michael Slowe]

As always, Dave puts things into perspective. Archiving for most of us might not be such an issue as it is with the production companies and broadcasters. His suggestion of transferring to newer drives every few years is however sensible, these are getting cheaper by the year and they can also be used for more mundane storage and back ups from our everday computers.

I don't necessarily agree that no one will want our documentaries in a few years time. How often are we asked for old material by various organisations? Speaking personally, my documentary on the groundsman at the old Arsenal football stadium, made over fifteen years ago now, is often asked for and indeed, will eventually be quite a valuable item of interest. I currently only have it archived on DVCAM tapes, apart from DVD's which, I agree, cannot be relied upon for longevity.

As for the kit to play tapes, well, decent professional decks such as the Sony DSR series will run for many years yet, both in our hands and production houses.

[ned c]

I agree with Dave's comments about the transitory nature of storage media although solid state drives will eventually be the answer I suppose. I disagree with his comments on the unimportance of archiving our assets. With two professional historians in the family we are constantly reminded of how the modern world of e-mail, twitter, face book, few letters and journals, are ephemeral and the paper chase they depend upon is vanishing into the electronic ether. So, those old family movies, odd documentaries and holiday records may well become the historical references of the future!!

ned c

[Dave Watterson]

Ned makes an interesting point about historians ... though in the long-term how important is their work? Famously we do not learn from history and much of what we think we know about the past is wrong or just a partial truth. History IS interesting and enjoyable. Our film/video records WILL enrich it in future but history will not necessarily make the world a better place.

At present solid state storage devices have an expected life that is shorter than regular hard disk drives, but who know how they may develop.

Part of me hoped that the very existence of "the cloud" - of all the information stored in databases around the world and copied on servers would give information a longer life. Many a club has very little in the way of history because the papers and reports were not kept safe and handed on from one officer to the next. Emails are much harder to destroy. Even defunct websites remain somewhere in the ether.

Just ask someone who has been in legal trouble because of an intemperate or ill-judged email. He or she cannot know whether or how often the original was copied. Every recipient may have a copy on their hard disk. All the various mail servers along the way may have copies on their files and their backup tapes.

Yes, if Amazon goes out of business masses of information might seem lost. (In addition to their own business records, Amazon rent out storage for many other companies large and small.) But there will be copies all over the place. Think of the old radio and tv shows which turn up in private collections decades after the original broadcast tapes were wiped.

But every time I watch another travelogue about the Galapagos islands, a documentary on the restoration of a steam locomotive, a comedy based on a bad one-line joke I rather hope archiving will not preserve those things.

Dave

[col lamb]

Archiving is an emotive subject with everyone having their own ideas and solutions.

I agree with Dave, keep the materal you want to archive on external hard drives (I recommend two drives, with the second hard drive being an exact copy of the first). Again agree to update storage medium when technology changes, remember IDE, in new PC's those drives do not have inbuilt connections as they are all SATA.

For PC users the best format to keep tape based material as an AVI, (avoid specialist AVI's like the Canpopus HQ family).

For AVCHD if you have backed the SD cards up to both the external drives they are catered for.

Storing completed movies as MPEG2 or MP4 is a compromise on keeping the file size down, but as the original was in MPEG4 there seems little point in storing the end movie in any other format.

It would be nice to have a secure "Cloud" service, but me I would not trust them. A couple of years ago my ISP went under and I lost all my "centrelinevideo.co uk" web site and a mass of data I had stored on my allocation of their server hence the never again ethos.

For you guys with Mac's I think that you are taking a risk keeping the files in ProRes format, with Apple's seemingly diminishing interest in video the day will come when they stop supporting it, and it is far better to store files in a universal format. I for one would certainly not archive PC derived files to a format such as Canopus HQ, like ProRes it is far better to keep storage to standard formats.

Tape based storage - we have been around this many times, one day the tape player will not work, what then. Electronic devices do wear out, soldered joints dry and fail, components break down, drives start to stiffen up. Whilst you still have a working player why not copy the tapes that really matter onto a hard drive, that is just what I did so I have all my MiniDV which in turn are backed up to the two external hard drive.

[ned c]

ProRes has now passed into the general world of acquisition as well as editing and is being used by the Alex and is availble on most external SDI/HDMI capture drives. i think that the Mac and FCX have been declared dead too quickly; no doubt Apple misfooted when they introduced FCX and left off some vital elements but are now updating. I was an early objector as it left out a vital capability that I used but I am watching with interest to see how it develops.

On the other hand film is now in its death throes; Kodak into Chapter 11 bankruptcy and many more features shot digiitally. Thanks Kodak; it's been a great ride.

ned c

[Michael Slowe]

I've just returned from the Production / Broadcast show at Earls Court (where by the way I met many IAC people, notably Tom Hardwick & John Astin) and every Professional film maker I spoke to uses and archives in the Apple ProRes codec, so I don't think that Col's pessimistic view is justified but we'll see in ten years time! I do share his disgust at Apples behaviour generally. It is rumoured that they may be planning to stop making the big MacPro computers - ridiculous since every graphic house and advertising agency worldwide relies on them.

A successful archiving method is the LTO tape system but this is really only suitable for professional outfits because the cost of a Record /Player is about £3,000 but it is becoming popular in that world. Solid state may be the future but we are in 'the now' so need advice!

[col lamb]

Michael

Yes, I am pessimistic, but there again the MPEG's I made in Premiere 15 years ago still play, enough said.

Then again what is the point of offering advice if it is rubbished, yet despite that you still state and ask.......but we are in 'the now' so need advice!

Store in Pro Res if you wish but I will always recommend that you also store in MPEG2 or MP4.

Solid state cards are NOT reliable for long term storage owing to the way the memory modules function.

Good advice has been given, this is my last post on this subject.

[Willy]

In ten years' time!
A few weeks ago I told you that I have plenty of cardboard boxes full of super VHS-cassettes in my cellar. Not only those big videotapes but also those fantastic old 8mm films. I have not watched them for ages. My filmclub is gone. It was even difficult to find people who wanted those projectors (Bauer, Elmo) for free. Actually they were already good for the junk yards. Now I realize I should have kept one. How stupid of me! I made such a big mistake by giving them to someone else. I don't even remember who he is.

Imagine...
But imagine that I do my best to copy my films on the "newest big thing" (David's expression). What will happen to them?
I don't think I am a pessimist or a defeatist. But I am a realist. We are all going to die. That's life. I love my wife (Today is St. Valentine's). She loves me. She also loves my films. But I am not sure that she will ever watch my films when I am in the Elysian Fields. And my daughter? She is not interested in filmmaking at all. I don't blame her because I am not interested in her hobbies.

Wedding film
I have a film on videotape about her wedding. Mind: she, her husband and children still form the happiest family in the world after 17 years. That's very exceptional these days. I asked Kathy if she wanted to have a copy of the film on a DVD. She said : "Oh, yes, dad. I would love it!", but really, I didn't hear any enthusiasm. We must be honest: she is not the only one. Are all your members of family different? I don't think so. We must accept that filmmaking is our hobby and that it is not anybody else's hobby.

Garbage
I am sure that all my films will be thrown into a garbage truck one day. "How often are we asked for old material by various organisations", says Michael. He gives the example of the documentary on the groundsman at the old Arsenal Football Ground. Yes, such documentaries are very useful for the future. For history in the future.

Last week I watched the film you made about BIAFF at Royal Tunbridge Wells, Michael. How sad it would be (for myself and my descendants, not for anybody else) if that film would disappear in a garbage burner?

Make as many films as you can
And thinking about fiction films? Is it useful to archive them for the future? Forgive me, my friends, this message is maybe a bit provocative or discouraging... If there is anyone who would like a copy of my funny films like 'Magic Clown' 'Occupied' or ... please ask me. I do not feel like a poor soul. I am making films because I enjoy it and I always hope that other people (will) enjoy them ... now ... and hopefully also in the future.

[Dave Watterson]

Col - I and many other readers value your detailed technical help and advice. Without you and Tom Hardwick the IAC would be much the poorer. All too often other forums and magazines speak in generalisations or at the other end of the scale in technospeak gobbledegook. You write from practical experience and in a way we can understand. And if other people have different points of view, so be it. On archiving, we can revisit the issues in a hundred years time and see if one approach was better than the other!

The Tonbridge film Willy refers to came about because Willy had been invited to propose the toast to the IAC at that year's gala dinner. I knew that family commitments meant Vera, his wife, could not be with him. So I asked Michael Slowe if he would mind making a recording of the speech for her.

To my astonishment (and some embarrassment on my part) Michael turned up in dinner jacket carrying a very large, heavy tripod and his main camera. Not only did he record Willy's speech but he also shot the audience listening and applauding. To see the finished work you would swear he had at least two cameras on the go. Watching him work that evening was a minor masterclass. Someone else might have used a small camcorder, handheld and pointing only at Willy. Even with an audience of one person in mind, Michael shot it to his highest standards.

Sometimes I cannot get over how lucky we are to have so many wonderful, generous people in our movement.