AVCHD Archiving (first principles)

A forum to share ideas and opinions on the equipment and technical aspects of film, video and AV making.
User avatar
ADBest
Posts: 79
Joined: Wed Jul 22, 2009 3:04 pm
Location: Sunderland

AVCHD Archiving (first principles)

Post by ADBest »

Having recently started to use my Panasonic HC-X920 I am just getting around to archiving my original footage from the SDHC cards; this being the equivalent of storing away my old HD tapes.

When I look at an SDHC card using Windows Explorer it would appear that the actual video content is stored at Private>AVCHD>BDMV>Stream as .MTS files.

Still picture capture is stored at DCIM>100CDPFQ as .jpg files.

There are a host of other files on the card.

Before I start deleting things and archiving the useful .jpg and .MTS files I would like to be sure that I am not throwing away anything vital.

I would be grateful if someone could either provide me with reassurance or warn me of potential disaster.

Arthur
ned c
Posts: 910
Joined: Mon Jan 29, 2007 3:39 pm
Location: Dammeron Valley USA

Re: AVCHD Archiving (first principles)

Post by ned c »

I import the files into my editing system and can then recognise them as individual clips in a folder; I use Final Cut 7 so they have to be converted to ProRes422 files but most edit systems will import the .mts files. When my edit is completed I then write the clips from the storage folder ("capture" in FC7) to a BluRay disc as DATA FILES, I do this using Toast on a Mac I am sure that other DVD/Blu Ray writing programs will write data files as well as .mpeg files. The data files can be dragged from the disc back into an edit program when required. I have used HDDs for archiving but one went down and left me concerned about their long term stability. Other archive media would be USB drives now that the prices are down, or even SDHC cards.

ned c
col lamb
Posts: 680
Joined: Tue Oct 16, 2007 9:51 pm
Location: Preston, Lancashire

Re: AVCHD Archiving (first principles)

Post by col lamb »

First, read the sticky post on AVCHD.

It advises briefly on storing.

Basically create a new folder, name it what you want, then copy the whole of the card content to this folder, repeat folder creation and copy process for each card you use. Then you will have an exact copy of the material on your PC. This is the backup, you would not previously delete a dv tape so you should backup the sd cards.

Only copy the files in the stream folder to the hard drive if you are using the Panasonic utility.

There can be issues if you shoot a long continuous shot c30+ minutes and the clips will be split and need to be stitched together
Col Lamb
Preston, Lancashire.
FCPX, Edius6.02, and Premiere CS 5.5 user.
Find me on Facebook, Colin Lamb
User avatar
Dave Watterson
Posts: 1872
Joined: Sun Jan 28, 2007 11:11 pm
Location: Bath, England
Contact:

Re: AVCHD Archiving (first principles)

Post by Dave Watterson »

This is the backup, you would not previously delete a dv tape so you should backup the sd cards.
I am guessing what you mean by this, Col, is that just as you would in the past have kept the DV tape with everything on it, so you should now store all the content of an SD card to allow for maximum flexibility if you ever need to return to that movie project.
User avatar
ADBest
Posts: 79
Joined: Wed Jul 22, 2009 3:04 pm
Location: Sunderland

Re: AVCHD Archiving (first principles)

Post by ADBest »

Thanks for the information.

I am just about to archive my first cards and I suppose I over-complicated the question as all I actually needed to know was whether the only files I need to archive from the plethora of files found on my memory card were the .MTS files for video and the .jpg files for stills.

I'm pretty sure that is the case, but before I start formatting cards to use again I need to be certain.

Arthur
col lamb
Posts: 680
Joined: Tue Oct 16, 2007 9:51 pm
Location: Preston, Lancashire

Re: AVCHD Archiving (first principles)

Post by col lamb »

Arthur
Just to confirm. Copy the whole card, including its folder structure to the hard drive.

I would not format the card, just delete all the files in the camcorder.

Repeatedly formatting cards is supposed to shorten their life.
Col Lamb
Preston, Lancashire.
FCPX, Edius6.02, and Premiere CS 5.5 user.
Find me on Facebook, Colin Lamb
User avatar
ADBest
Posts: 79
Joined: Wed Jul 22, 2009 3:04 pm
Location: Sunderland

Re: AVCHD Archiving (first principles)

Post by ADBest »

Thanks Col. I do keep your AVCHD sticky always to hand and consult it every time I forget the specific resolution of a particular format.

Since posting my first question I have had a better look at the actual content of the memory card and find that the .MTS files represent about 99% of the data stored. The remainder, .MTL, .CPI etc etc are very small files indeed and whether they are archived or not has little bearing on storage requirements.

I had hoped to be able to create separate folders for each project on a multi-project card but it would be difficult, if not impossible, to separate the non-.MTS files into their appropriate folders. At this point I realised that archiving a complete multi-project card was no worse than sticking a DV tape in the cupboard and keeping a paper data-base of it's content.

I can now get my external drives sorted out and get my footage safely stored away.

Arthur
User avatar
TimStannard
Posts: 1225
Joined: Fri Feb 11, 2011 5:20 pm
Location: Surrey

Re: AVCHD Archiving (first principles)

Post by TimStannard »

col lamb wrote: <sound advice followed by> .. This is the backup, you would not previously delete a dv tape so you should backup the sd cards.
Just with my IT hat on here and possibly sounding more than a little pedantic, once you have deleted the material from everywhere else (on your hard drive, on your SD card) this is NOT a backup. It is the ONLY copy (exactly as your Mini DV tapes are). Ideally you should take a copy of your archive and store it elsewhere (ideally a different house). THEN you have a backup.
Tim
Proud to be an amateur film maker - I do it for the love of it
col lamb
Posts: 680
Joined: Tue Oct 16, 2007 9:51 pm
Location: Preston, Lancashire

Re: AVCHD Archiving (first principles)

Post by col lamb »

Arthur

The small files are used when the card is written to and read hence they are important.

By full copying all the folders, files and folders to what I describe as a backup location, there is the possibility of copying the content back onto a card and it works.

If software is used to copy the video only from a card it does some reprocessing to each file and changes its extension to m2ts from mts.

In practice I have a dedicated video store drive on my PC, this is where I store all my SD and Compact Flash cards contents to this drive and it is also where I store the made movies and Disc Images. Then i backup this whole drive to an external hard drive and keep it remote from the PC when it is not in use.
Col Lamb
Preston, Lancashire.
FCPX, Edius6.02, and Premiere CS 5.5 user.
Find me on Facebook, Colin Lamb
Michael Slowe
Posts: 807
Joined: Mon Jan 29, 2007 4:24 pm

Re: AVCHD Archiving (first principles)

Post by Michael Slowe »

Tim's right, one copy is not a back up.
Do you really want to keep everything you shoot? Certainly during an edit I keep two copies (on separate stand alone drives well away from the editing external drives), but once a project (film) is completed I delete everything, having first exported the film as a QT.mov file, on to two separate drives. I can retrieve the film in minutes and re import into the edit suite for more work or making additional DVD's and BD's. Ned's method is similar except he is probably being even more secure by keeping his files on disc. Drives fail, there's no doubt about that and they should be run every few months to stop them seizing up.

Col is right about keeping the whole file structure if you've shot on to cards. I shoot XDCAM on to the Sony S X S cards and unless I have kept the whole file structure intact I can't import the media, the Sony Transfer software won't recognise the media unless the whole file is there. It seems, from what Col says, that other systems are similar in this respect.
User avatar
ADBest
Posts: 79
Joined: Wed Jul 22, 2009 3:04 pm
Location: Sunderland

Re: AVCHD Archiving (first principles)

Post by ADBest »

Thanks everyone for your helpful ideas. I have spent today settling on an archiving procedure which should satisfy all the comments. I repeat it here for the benefit of anyone else in circumstances similar to mine.

The method creates two copies on separate external drives which are not kept connected to the PC, (in an attempt to thwart the latest evils of the ‘crypto locker’).

I have always kept a log describing the content of each DV tape so that I could use the content again if required. I will continue to keep this log with the content of each card. The only difference being that the card will be reused and its current contents saved on the external hard drives.

Procedure is as follows:-

With the card still in the cam delete any clips which are not required. I think this is important because if you try to do it at the back-up drive level then the associated .CPI and .MPL files are not updated.

Create a new folder on the external drives with a unique title which links back to the log reference.

Copy the entire card contents to these new folders. These folders thus become the archive for the card and the card contents may then be deleted, (using the software within the cam), but note the next paragraph contents first.

The video clips may then be imported into the editing system either from the card via the cam or directly from one of the back-up drives.

If importing from the back-up drive it is necessary copy the ‘DCIM’ and ‘PRIVATE’ Panasonic folders to the top level of the back-up drive file structure, as the 'HD Writer AE5' software only looks for these files at the first level. It ignores any copies buried deeper in the file structure.

Perhaps others will find this useful.

Arthur
col lamb
Posts: 680
Joined: Tue Oct 16, 2007 9:51 pm
Location: Preston, Lancashire

Re: AVCHD Archiving (first principles)

Post by col lamb »

One problem.

Import the video files into editing programme from ....... the card via the cam........not what anyone should recommend.

Copy files onto a dedicated internal video editing only hard drive, also it is a good idea to have the editing software write all its files to the same hard drive and not to the boot disc.

Ideally an editing programme like Premiere will have many seperate hard drives, at least one each for:- source video, cache, preview, output.

What I suggest that you do is: -

1 - Make a full copy of a card on both your external hard drives (these are your backups)

2 - Use HD Writer to copy the files from the camcorder to your specified location on the dedicated video hard drive

3 - Once the files have been copied disconnect camcorder by ejecting it via the Windows programme shortcut which is located in the bottom taskbar...........Safely Remove Hardware and Eject Media

4 - Turn off camcorder and then disconnect it.

5 - Delete all files on your SD card only when you have checked that the backup copies and the working copies are OK
Col Lamb
Preston, Lancashire.
FCPX, Edius6.02, and Premiere CS 5.5 user.
Find me on Facebook, Colin Lamb
User avatar
ADBest
Posts: 79
Joined: Wed Jul 22, 2009 3:04 pm
Location: Sunderland

Re: AVCHD Archiving (first principles)

Post by ADBest »

Col, thanks for your patience.

I can see that my choice of words may have given the impression, (at your para (2) stage, that I was importing only the .mts files when in fact I intended to indicate that HD Writer be used to copy all the files). I have reworded my description and I think it now fully mirrors your method. Perhaps using a few more words but intended to be of use to even less experienced club members than I. If I could impose further on your patience it would be nice to have your seal of approval on the description below before I release the words to my club compatriots.

Revised wording:-

I have always kept a log describing the content of each DV tape so that I could use the content again if required. With the introduction of the new memory card based storage I will continue to keep this log, cataloging the content of each card. The only difference being that the card will be reused and its current contents saved on the external hard drives on the computer as opposed to DV tape in the cupboard.

Archiving and capture to a video editing system

1 With the card still in the cam delete any clips which are not required. This is important because if you try to do it at the back-up drive level then the associated .CPI and .MPL files are not appropriately updated.

2 Create an identical new folder on each external drive with a unique title which links back to the log reference.

3 Copy the entire card contents to each of these new folders. The folders thus created become the archive for the card and the card contents may then be deleted, (using the software within the cam), but read paragraph (4) before deleting.

4 A complete copy of the card contents may then be imported into the editing system using HD Writer either from the card via the cam or directly from one of the back-up drives.

5 If importing from the back-up drive it is necessary copy the ‘DCIM’ and ‘PRIVATE’ Panasonic folders to the top level of the back-up drive file structure, as the 'HD Writer AE5' software only looks for these files at the first level. It ignores any copies buried deeper in the file structure.

Once the contents of the card are deleted then the back-ups are the only way to access the stored material at a future date.

Thanks again

Arthur
col lamb
Posts: 680
Joined: Tue Oct 16, 2007 9:51 pm
Location: Preston, Lancashire

Re: AVCHD Archiving (first principles)

Post by col lamb »

Good job Gromit............to borrow an apt well known phrase.

Now you have a very similar system to myself.

On my PC the dedicated video hard drive not only has my 1st backup of my cards it also has the completed movies and related video tutorials and in fact every thing video I have made or used.

Now on my 2nd and 3rd backups (the two external drives) after I have archived the cards to the folders I run a free piece of software called FileSync. This as the names suggests syncs all files on a specific location to another so my video drive in its entirety is copied to the external drives. The drives are all 2Tb so fairly large and still have masses of free space.

Incidentally whilst I still had my DV camcorder I captured all my DV tapes to the video hard drive so there they are waiting to be edited where I have not yet used them.

Now is the time to put all those files to good use, happy editing.
Col Lamb
Preston, Lancashire.
FCPX, Edius6.02, and Premiere CS 5.5 user.
Find me on Facebook, Colin Lamb
User avatar
ADBest
Posts: 79
Joined: Wed Jul 22, 2009 3:04 pm
Location: Sunderland

Re: AVCHD Archiving (first principles)

Post by ADBest »

Well that’s archiving sorted.

Now to consider output from the editing process.

Up to present, although I have captured in HD, I have edited in standard definition and all my videos are saved and backed-up as .avi files and burned to DVD for distribution. I settled on this method as very few of the friends who would normally receive copies possess Blue-ray players and it was too much of a hassle to cart my BD player about with me.

I have now purchased a WD TV Live media player which is about the size of a VHS video tape and therefore not a problem to transport.

The idea is to save my 1080i edited footage as an mpeg4 file, suitably backed-up, and then be able to copy any movies I wish to show as files on a memory stick using the, very portable, WD TV Live media player to play them.

I also thought I would be able to connect my external drive, with all my historic .avi movies, into the media player and hence into my TV and enjoy my archive at leisure.

HOWEVER, the .avi files created by Premiere CS5 do not play on the WD device. To play them I need to use ‘Format Factory’, (as recommended by Mike Shaw), to transcode the files to .mpeg ((2) I think), which then play happily on the media player. I was reconciled to either converting all my .avi files, or, more likely, continuing to rely on my DVD copies.

I happened to download a demo video from Vimeo in .avi format and surprisingly it plays perfectly well on the media player without any further action. This is where I begin to get out of my depth. I know that the .avi format takes a number of forms and that this gets into the realms of Xvid etc.. It would appear that Vimeo uses a form of .avi that the media player is happy with but that the .avi files produced by Premiere are not compatible.

The question is; “can Premiere be set up to output compatible .avi files – eg Xvid, which the media player would accept?”

Hope someone can help.

Arthur
Post Reply