How might BIAFF entries be judged?

A forum for sharing views on the art of film, video and AV sequence making as well as on competitions, judging and festivals.
User avatar
Dave Watterson
Posts: 1872
Joined: Sun Jan 28, 2007 11:11 pm
Location: Bath, England
Contact:

How might BIAFF entries be judged?

Post by Dave Watterson »

In the discussion of BIAFF 2011 some concern was expressed about the judging. That always happens! But let us see if we can come up with some positive ideas about how the assessment of 200 or more films ranging from under 1 minute to 2 hours long might be done better.

For information, this is how it was done in 2011:

The competition officer received 214 entries and checked them all for playability etc. Then on a weekend at the end of January 18 people gathered in a hotel near Durham.

15 were split into panels of 3 to view the entries.
  • After every block of films (about 90 minutes worth) the panels regrouped so that the same 3 people did not watch more than one block together.

    They decided on the star rating for each film, from 1 star to 5 stars. The 5-star films went on to the final panel.

    The Competition Manager ensured that so far as possible no one judged a film from their own IAC region. If a judge had made a film, or assisted in the making of one, he or she did not judge it.

    If a panel could not agree a rating the film was passed to another panel ... though this was very rare.

    After each film, one panel member was asked to write notes on it for sending to the film maker, incorporating the views of the whole panel.


3 were the final judges who examined those entries passed up to them by the first stage panels. They decided the Diamond awards and special prizes.

On the Friday evening all the 18 judges assembled and watched a selection of entries chosen by the Competition Manager to present a range of quality. This attempted to establish a common standard. All of Saturday and most of Sunday was spent viewing films.

After the weekend each judge went home and wrote up the notes they had agreed to do and sent them to the Competition Officer.

Over the weekend meals and accomodation were paid for, but no travel expenses were paid.

-Dave
tom hardwick
Posts: 914
Joined: Mon Jan 29, 2007 9:59 am

Re: How might BIAFF entries be judged?

Post by tom hardwick »

Your description might apply to 2011 Dave, but it's been like this for very many years, hasn't it?
User avatar
Dave Watterson
Posts: 1872
Joined: Sun Jan 28, 2007 11:11 pm
Location: Bath, England
Contact:

Re: How might BIAFF entries be judged?

Post by Dave Watterson »

It has been like this since 2005 - with minor variations. Before that judging took place over two or three weekends in the homes of various IAC Council members and their friends, with judges sleeping in spare rooms and with other friends in the area.

I gave the description only to serve as a starting point for discussion.

Dave
Mike Shaw

Re: How might BIAFF entries be judged?

Post by Mike Shaw »

It would be difficult, I think, to come up with a more 'balanced' approach.

Just goes to show though, how difficult judging can be - we all have our 'prejudices' and look for different things when watching a film (in judgemental mode).

I'd wager too that if you showed the same movies to audiences at different locations and used a 'clapometer' method for judging the best film shown, the results would be different each time.

Judges can never win!
Ken Wilson
Posts: 164
Joined: Mon Jan 29, 2007 11:17 pm

Re: How might BIAFF entries be judged?

Post by Ken Wilson »

As I started the current judging discussions from BIAFF 2011, perhaps I should start the ball rolling. I appreciate Dave posting this new thread to hopefully come up with something. I do remember Dave starting something similar in the days of letter writing a few years back, though I don`t think we managed to change anything that time. Let`s do better this time. I said to Mike at Harrogate that I agree with much of what he writes on the forum, but please don`t be negative here that we cannot look at new ideas and arrive at a better system.
It may be a coincidence, but I keep records of all my films, results, judges comments and all the rest. Quite a few of my films over the past 5 or 6 years have fallen below expectations. I may be getting worse or my competition getting better...or it may be that the system isn`t working (?) The judging methods in the days of Mike Donlan did spread things over a few weekends as Dave says. I wasn`t involved in it and it may have proved to be less that ideal for the comfort of the judges, but it certainly allowed more time and therefore gave breathing space to get the decisions right.
So enough of that. Here are my ides to be considered.

Blank DVD discs are very cheap and we assume that most, if not all film makers can put a film on to a disc. I prefer to enter my films on mini DV as there seem to cause less problems for other people (though I can play just about all DVDs from anywhere nowadays without all the problems we used to have) but mini DV films would have a DVD with it.

So: 1/ ALL films entered into BIAFF on whatever your own chosen format, would be accompanied by a standard play DVD.

2/ The closing date is adjusted to allow enough time between the close and the BIAFF weekend to do what follows.

3/ A panel of judges chosen from those who are accepted as competent by most of the known IAC authorities (proven skilled judges and those who have appropriate credentials) are posted the copy DVDs to their home. This panel should be as many as reasonably manageable, but say 6 to spread any preferences on genre/ subject matter and personal likes/ dislikes.
In other words, more judges give a more balanced view.

4/ These judges have lots more time and watch the films in comfort and without any pressure from perhaps one strong minded individual. (Similar to the Cotswold method.)

5/ ALL judges see ALL the films and any chosen judges should be aware of what is involved. Their task is to supply an initial grade of each film from 1 star to Diamond (in their opinion.)

6/ These judges write a few basic comments on all films below a certain level, say 1 star to 3 stars. Al those above are passed on to the judging weekend session.

7/ The judging weekend takes place with the chosen 70+ movies (based on this years entry) to be graded into 4 star to Diamond and the final trophy winners. This means that only about 1/3rd of the total entry remain and are more easily managed on that final session so that more time/ more judges can see the films in a less rushed environment.

The benefits of all the above is that each film is seen by more people and each is given more time for consideration.

The only extra costs are 1 blank DVD per film paid for by the entrant and some extra postage for the discs. I would guess that the savings on the final weekend would likely cover the postage costs.

Please discuss. Ken
User avatar
fraught
Posts: 567
Joined: Thu Mar 08, 2007 11:54 pm
Location: Basingstoke
Contact:

Re: How might BIAFF entries be judged?

Post by fraught »

I have to say Ken, that sounds like a very sensible approach to me, and one that would be worth trying.
Only Boring People Get Bored
http://www.fraught.net
Mike Shaw

Re: How might BIAFF entries be judged?

Post by Mike Shaw »

There is another aspect which I have heard well-seasoned international judges say ... a film looks very different in the 'silence' of one's own home, compared with seeing the same film in a large audience that reacts (noisily perhaps) to the film's events. One such judge commented that he had not rated a film very high when seen with one or two other judges 'in private' - but when seen in the company of a large audience, it looked very different - and better!

I think that there is a huge difference in seeing a film in an isolated situation, and seeing the same film in the company of a large audience. Especially where comedies are concerned. Audience reaction - whether of laughter or 'horror' - is contagious, and contributes appreciably (in my view) to a film's enjoyment.

I'm all for a better system of judging - one which completely isolates human (and hence personal) judgement would be ideal were it even remotely possible.

Even if one filled a theatre with 100 people (as judges), the winner is still likely to have achieved less than 50% of the 'votes'. Look at the results of the 'audience vote for the most entertaining film at the festival' as a pointer and re-inforcement to that.

A knotty problem. I'm not sure that seeing the films in the isolation of one's own home is the answer to it either, I'm afraid, no matter how many individual people so see.
Chrisbitz
Posts: 198
Joined: Thu Feb 07, 2008 2:31 pm
Location: Orpington, Kent

Re: How might BIAFF entries be judged?

Post by Chrisbitz »

At the risk of being negative, I don't believe there is a way to "do judging" correctly.

At the end of the day, it's opinion not fact, and one person might like a film and another person might hate it. Unfortunatley that's always going to happen as long as you are forced to use mere humans.

I just take all Judges comments with a pinch of salt, bearing the above in mind, and loudly declare that ALL JUDGES ARE IDIOTS.

Unless I win of course! :-)
I like to make films, this is- my Youtube account. What's yours?

"all of the above is nothing more than nonsensical ramblings, and definately should NOT be misconstrued as anyone's official policy"
tom hardwick
Posts: 914
Joined: Mon Jan 29, 2007 9:59 am

Re: How might BIAFF entries be judged?

Post by tom hardwick »

I'm with Ken - if at all possible I ask to judge films at home.
User avatar
Dave Watterson
Posts: 1872
Joined: Sun Jan 28, 2007 11:11 pm
Location: Bath, England
Contact:

Re: How might BIAFF entries be judged?

Post by Dave Watterson »

We all know that judging any work of art - including videos - is subjective. But the practical questions to address here are how best to organise the judging of BIAFF so that there is a good chance of avoiding anomalies.

IDEAS PLEASE!

And thanks for those that have already been sent in.

Dave
Peter Rouillard
Posts: 3
Joined: Wed Feb 07, 2007 2:15 pm
Location: Guernsey, Channel Islands
Contact:

Re: How might BIAFF entries be judged?

Post by Peter Rouillard »

So far, Ken's suggested system is the closest to the system we used for the Guernsey Lily competition in the recent years. The three preliminary judges viewed ALL the films submitted ( up to 150 ) in their own time over a period of several weeks. If any of the chosen judges did not have the facilities to watch Mini DV, for example, then I would send them a copy on DVD, - in the right format and with no loss of quality.
Each judge would write their comments separately, and also give it their own grading. These would then be assessed after all the judging has been completed, and if there are any discrepances of opinion between the three, then we would discuss them and decide the rating.
With this system the judges are not rushed over the process and may watch the films more than once if they wish to clarify one or two points in the films plot, for example. The films which are rated as ' Special Commendation ' ( the equivalent of BIAFF's 5 Stars ) are then put forward for final judging.
So yes, Ken's idea of all entrants also submitting a standard DVD copy should work very well. Also, the entrant would receive individual comments from the three preliminary judges, rather than condensed comments from one judge writing on behalf of three.
We have had VERY few complaints on our judges comments and ratings since the Lily started. There were many at this year's BIAFF who could not believe some of the star ratings given - including me !!

Peter Rouillard
Mike Shaw

Re: How might BIAFF entries be judged?

Post by Mike Shaw »

Well, I always thought that the Lily judgements were very fair, very balanced, and the judges comments positive and constructive. So, Peter's 'Lily' method method works for me. And I stand down on the idea that "it would look different seen among an assembled audience".

I also think the ability to watch a film through a second time - particularly if one has to write a critique - is invaluable: some may want to look again at all or part of a film, others not at all. Hence, home viewing makes absolute sense on that score.

I can also see the benefit of comparing 'judgements' - with discussion on those where there is a disrepancy of opinion.

Presumably it also means that all the judges see all the films, making 'comparative' judgements easier. But at the same time, it does tie each judge down to hours - days - of viewing.

I would presume that the options would be to send all the judges all the films (lots of copies need to be made), or for each judge to get a batch at a time and to pass them on to the next judge (lots of postage perhaps).

There is still the requirement perhaps that judges 'shouldn't see their own films or films from their own region'.

One must bear in mind that whatever method is used - even using an 'audience vote' - there will always be some that disagree with the verdicts. But as the Lily verdicts were always well received as far as I'm aware, that method seems to offer a very acceptable approach.

Athletes have it easy, don't they! (Just kidding!)
User avatar
Howard-Smith
Posts: 298
Joined: Tue Apr 20, 2010 12:11 am
Location: Wolverhampton
Contact:

Re: How might BIAFF entries be judged?

Post by Howard-Smith »

I believe that Ken and Mike's suggestions provide the most satisfactory solution to the problem. The judges should have DVDs of all the entries to watch at home in their own time. I for one would be totally prepared to pay a higher entry fee to cover the cost of the blank DVDs needed to make the judges' copies.

Howard-Smith LACI
Mike Shaw

Re: How might BIAFF entries be judged?

Post by Mike Shaw »

...actually, Ken's and Peter's suggestions. I was merely agreeing after having suggested it was better to see the films 'in a large crowd or audience' as they seem to take on a different 'flavour' then to when seen in isolation. The advantages of 'solo' viewing at least in the first 'round' far outweigh - and are far more practical - than any group viewing.
Ken Wilson
Posts: 164
Joined: Mon Jan 29, 2007 11:17 pm

Re: How might BIAFF entries be judged?

Post by Ken Wilson »

Many people reading this will know that I am a bit obsessive about movies. Movies are the equivalant of my sport where people live and breathe it and "getting it right" is very important. So I was awake for hours last night going through the logistics of the idea I posted on here, which so far seems to have gathered some support-thank you all.
I was about to say, before others did, that yes, I agree, NO system of judging an art form can be perfect as it will not be proveable which (film) is best. Unlike sport, we cannot prove who is fastest, or who can jump the highest, so it will always be subjective, but not having a perfect system, doesn`t mean that we shouldn`t try to have a better system. After all, nothing in life is perfect anyway.
So I read Peter`s comments about the "Lily" system with great interest and he has already talked to me about how this worked.
I was on the final panel at Guernsey one year, so saw for myself the fairness of how the final winners were chosen. I also have judged in a number of other competitions, such as "The Burnley" festival, "CEMRIAC," "Dundee club", "The Mersey Ten", "North vs South" and a number of others, apart from our own fiction competition 5 times. Having the films at home, certainly does mean that I watch them more than once. Sometimes seeing them three or four times if I need to verify something, is not unusual.
So to expand on what I said last time, it really could be simple number crunching when deciding the rating a film should receive.
If we had 6 preliminary judges at home (or 3 in Peter`s example) there are a group of people who will see EVERY film. Unlike at present where apart from David Newman who is not judging, NO-ONE sees all the films entered.
So based on my figure of 6 ( which can be adapted to suit chosen number of judges) here`s how it works out.

1/ Closing date to BIAFF weekend date adjusted to allow sufficient time.

2/ 6 judges= 6 parcels of plain (in wallet) DVDs posted out. Each parcel contains around 35 films =210 films. Judges would get at least a week to see each set of films. This would equate to watching an average of 5 films per day. Not an unmanageable number.

3/ Parcels are passed around by post from one judge to the next. Allowing a week each, means a 6 week+ process. This would likely run through January and the first half of February.

4/ Each judge writes a few basic notes (meaning each film gets 6 sets of fairly basic comments) and gives the film a score of 1 (1 star) to 6 (Diamond.) Max score 36. (6 x 6)

5/ A film with a score above 18 at the end of this (6 x 3 or an average of 3 stars per judge) moves on to the second session at the judging weekend. A film scoring 6 would be a 1 star; a film scoring 7 to 12 would be a 2 star (meaning at least ONE judge thought it deserved 2 stars) and 13 to 18 would be 3 stars (meaning at least ONE judge thought the film deserved 3 stars.)
As we should encourage and support film-making, we should always defer to the higher score.

6/ Films above a 3 star rating (above 18 points) automatically go through to session 2. This could be the remaining 9 judges (as BIAFF uses 15 judges and 6 of these have been working from home) divided into 3 panels of 3 as it is now.
This is the Saturday session and each panel of 3 now have to see about 25 films each. (75) These scores are added to the first judges` scores.

7/ This Saturday judging session selects the 4 star films. Each film arriving here has now been seen by 9 judges!!! Time constraints would probably mean that this session still needs 3 panels, but each film is being seen by many more people anyway.
Again the scores would show that up to 36 are 4 star films. (This is 9 x 4 the total added together from sessions 1 and 2.) Any score above 36 means 5 star and Diamond and the final award winners.

8/ The Sunday session now has only around 15 films to view and discuss on the final day to put into place the 5 star films, Diamonds and winners. This could be done by the current sysem of 3 or by all the judges at the weekend who have stayed over and may be the total 9 from Saturday.

This means that EVERY film will have been seen by at least 6 people. Some films may have been seen by all 15 judges!

In practice, judges at home (session 1) can begin when enough films have been sent in and wouldn`t need to wait for the closing date.

A "level-setting" DVD could be made and sent out to the judges by David Newman ( similar to the Friday evening session at the judging weekend) to agree star rating levels.

If in doubt, any film should be graded upwards. (We are here to promote film-making and encourage it.)

The final decisions should remain with the competition officer; David Newman.

So that`s it. Just my thoughts and a way that this system could work. Perhaps a bit more effort, but I think as with the Guernsey Lily, many fewer complaints and though we will not all agree, it`s a much more balanced system in my view.
There are areas where fine tuning and/ or some elements could be changed as required, but It`s no use moaning and not having any suggestions to back up a new system.

I hope that David Newman reads this or we may have to let him know what is being discussed on the forum.
Post Reply