How might BIAFF entries be judged?

A forum for sharing views on the art of film, video and AV sequence making as well as on competitions, judging and festivals.
User avatar
fraught
Posts: 567
Joined: Thu Mar 08, 2007 11:54 pm
Location: Basingstoke
Contact:

Re: How might BIAFF entries be judged?

Post by fraught »

Excellent piece of work there Ken... i agree with it too! :-)
Only Boring People Get Bored
http://www.fraught.net
Arthur Bates
Posts: 29
Joined: Wed Feb 25, 2009 12:16 pm

Re: How might BIAFF entries be judged?

Post by Arthur Bates »

May I enter the ring? I can only thank the judges for their time and patience in grading our efforts. It must be a gruelling task and I don’t expect they go home laden with riches. It must be a bit depressing to face so much dissatisfaction. Thank goodness I will never be a judge; anyway they would never get my wheelchair up the hotel steps. I have always wondered if any of the entrants ever get no stars. I am absolutely against judges viewing films at home. This approaches the idea of a single judge this I do not support. I have submitted my film to club competitions with a single judge and have never won anything, worse; I received reports one of which was unhelpful and rude, the other downright insulting. The same films got 3 stars in the subsequent BIAFF and similar credit at Guernsey. The films may be seen by other judges in solitude but I think that discussion between them is important. Would they all have the correct equipment for viewing? Setting a standard is easier if one is marking an exam where the answer is to a set question but how is it possible to set a standard when there are so many variables? Audience participation as suggested seems to me to indicate one not unimportant factor; entertainment value and surely that is confirmed by some of the remarks made after the festival. To be cynical, I wonder how much of this criticism of the system arises from dissatisfaction of personal results. This may sound rather opinionated especially as I am no longer mobile and am unable to get to the Festival to my great regret as I have enjoyed it so much in the past. Thankfully I can still make films and thought I had done fairly well this time till I heard everyone commenting on how low the standard was. You just can’t win. Arthur B
Mike Shaw

Re: How might BIAFF entries be judged?

Post by Mike Shaw »

Arthur, I don't think having 'a' judge see the movies at home means there'd only be one judge - that isn't what Ken was suggesting at all. He was suggesting six judges are on the home circuit, and that their individual assessments are then also assessed at a further 'panel' of judges. So I think your fears on that score are unfounded. We've all suffered, I think, from very poor judgements - one competition I entered the critique was 8 words long and simply decribed what the movie was about! That cost me £5!

There is a lot to be said for being able to see films in one's own home - my own feeling, if you read back in this thread, was that it would be better to be in an audience and get the 'feel' of an audience's reaction to a film. However, that isn't a possibility really for judging something like 250 films. Being able to stop a movie and go back - especially when writing a critique - is really invaluable and not quite as possible when several people are watching at the same time. From Ken's proposal - which as far as I can gather closely follows the very successful Lily methods - the judge's have plenty of opportunity to discuss their assessments. And Ken also advises 'marking up' as a form of encouragement to film makers, which I also strongly agree with - so if a result is poor, then maybe the film maker should start looking at his work rather than at the inability of the judges to recognise 'what was intended'. Sometimes what we think we have made isn't what the judges see - and recognising this is an important factor in improving our own work I think. I made a film which I thought was easily going to be a 4-star - but it wasn't, it was a 3: when I read the crits, I knew exactly why. And they were right of course.

Judges are always criticised: someone, somewhere will always disagree with a judges decision. I think we all agree on that. I believe that Ken's plan does try to iron out and smooth down the discrepancies between judges markings, just as the Lily always seemed to 'get it right'. Audiences do react differently - even the same film shown to different audiences at different times or at different locations will get different reactions.
ned c
Posts: 910
Joined: Mon Jan 29, 2007 3:39 pm
Location: Dammeron Valley USA

Re: How might BIAFF entries be judged?

Post by ned c »

First a question. What qualifies a person to be a BIAFF judge and how are they selected/appointed for BIAFF?

My thought is that the judges for the BIAFF will be representative of the age/class of the members of the IAC and will judge in a predictable way, the occasional award to a film that does not fit the expected criteria causes a storm of comment and complaint. I suspect that it is possible to set out to make an BIAFF award winning film based on examination of the past winners rather than making a film that meets a personal passion.

No matter how the process is organised the judging is subjective and the recipient of the awards/comments must receive them with a large pinch of salt and disagreeing with judges is a waste of time.

The Academy Awards are based on sending out films to the various grades of members of the academy and them voting; there are no comments supplied but look at the results almost invariably cause of much discussion and dissent.

To me the weakness of the present system is that it depends on people who have the time to devote to the concentrated sessions; which leaves me suspecting that many of the judges are retired people who by dint of their age and experience will favor certain types of films. But however you slice it someone will be disatisfied.



ned c
Brian Saberton
Posts: 355
Joined: Fri Apr 27, 2007 8:00 pm
Location: Scotland

Re: How might BIAFF entries be judged?

Post by Brian Saberton »

Ken's suggestions have a lot of merit, and the system worked well for the Guernsey Lily, but bearing in mind that this year there were 214 entries in BIAFF (249 in 2010) and, unlike the Guernsey, there is no time limit on length it could add up to a considerable amount of viewing hours for the judges, especially bearing in mind that there were quite a few films this year that were rather long. Add in the time to write up notes and it becomes quite a large undertaking. Speaking for myself I wouldn't be able to commit the amount of time required. However the value of the first round judges seeing every film is clear and I think if you could get enough volunteers willing to do it, the system could work well.
Brian Saberton
User avatar
Dave Watterson
Posts: 1872
Joined: Sun Jan 28, 2007 11:11 pm
Location: Bath, England
Contact:

Re: How might BIAFF entries be judged?

Post by Dave Watterson »

If I have added correctly this year's entries totalled 48 hours 16 minutes ... 48 hours is a lot of film to watch and to comment on.

I like the Lily system where the same people see all the entries. This at least makes a consistency of ratings more likely.

There is a potential problem if those judges decide on levels (e.g. 1 star to diamond). How do we know judge A sets the same levels as judge B?

When I did the first stage of the Lily each judge ranked all the films in order from 1 to xxx. (Often there might be 3 or 4 films at the same level "52nd equal") When the judges compared lists there was very little variation in ranking. At the end of the process we went through the list to pick how many of the top films could be shown to the final panel, bearing in mind that they have a limited time to work in. Then we moved down the list to find the points at which we agreed there was a notable change of quality, so that we could separate "highly commended", "commended" and so on.

I think it is important to recognise that this year the feeling among many people was that something had gone awry in the judging. It was more than the usual disagreement with judges or any sense of individuals being marked lower than they anticipated.

Judges

As to who the judges are ... most are retired people because they are the ones who can give up the time. When we were working we had to calculate the loss of Friday and Monday earnings when figuring out whether or not to take part. (Arthur, no one even gets expenses, so no judges make money. They all pay their own travel.)

As any competition organiser will tell you, it is not easy to find good judges.

IAC rules require judges from each region. Many have taken part before so their ability is recognised and respected. Newcomers are carefully scheduled to work with more experienced people. But it is hard to find suitable people. How can a Competition Officer know what someone is like as a judge until he or she has tried using them? There are usually last-minute panics when people have had to drop out and someone else has to be contacted at short notice.

Ideally no judge would have a film in competition ... but in practice that has to be modified to: no judge assesses his or her own film.


Dave
Ken Wilson
Posts: 164
Joined: Mon Jan 29, 2007 11:17 pm

Re: How might BIAFF entries be judged?

Post by Ken Wilson »

There is apparently a saying which people e-mail to me, which says: "If you always do what you`ve always done, then you`ll always get what you`ve always had." In other words, if we don`t change any part of the judging system, it will always remain as it is now. HIGHLY controversial.
I accept that there is no perfect system, but there is certainly a lot of room for improvement.
May I say to Arthur to allay his fears, as Mike said on my behalf, the single judge idea sat alone at home is NOT what I was saying at all. We have a panel of judges, who have the abilty to watch a film as often as the feel they need to. They would of course have the equipment as they are viewed on a simple standard DVD disc.
I have done this many times judging our own competition especially when writing up the critiques. Again I say that I know this is for a much smaller competition, but that`s why I put in all the numbers for a supposed entry of 210 films to see how it all worked out.
Dave says:
If I have added correctly this year's entries totalled 48 hours 16 minutes ... 48 hours is a lot of film to watch and to comment on.

I like the Lily system where the same people see all the entries. This at least makes a consistency of ratings more likely.

There is a potential problem if those judges decide on levels (e.g. 1 star to diamond). How do we know judge A sets the same levels as judge B?


Taking the figure Dave gives of 48 hours and based around my proposed system of a 6 week viewing period, this is 8 hours a week or in other words, little more than ONE hour a day. I think there would be many judges who should be able to spare ONE hour a day. Also I should clarify that these judges DO NOT write up a full page of critiques for each film. I said BASIC comments meaning a couple of sentences, but for the film makers, it`s from 6 people!
Films which rate more than 3 stars in round one don`t need any comments at all from the panel of 6, just a score. They will receive their comments from the second round judges. I believe this always happened in the system used in IAC MOVIE xx competitions.

The point about judge A not working at the same level as judge B is exactly the problem we have now. It`s a random system, a lottery in fact, that the panel of 3 who rate your film will most likely give a completely different result to another panel of 3. The idea here is of course than at least some people will see ALL OF THE FILMS and more judges will see more films.
The idea of using more judges, levels out any of these differences and in any case, any judge working to a slightly different level, will at least still be consistant to themselves.

In our fiction competition I tried using 6 judges the first year. Scores varied but the number of people involved evened things out. The problem was that these were actors who are usually highly critical and blunt to the point of rudeness. Highly unsuitable for we sensitive non-commercial film makers. I had to tone down their remarks somewhat. But for the past 3 years we have had 4 film judges who have seen EVERY film entered. Some film makers may not agree with all the final results, but the system is consistent. In 2010 we also had to send 3 longer films on DVD to the other judges for a viewing at home prior to the main session. It all seemed to work well, so it`s really just working out the best way to scale things up in numbers.
User avatar
Willy
Posts: 711
Joined: Sun Jan 28, 2007 10:42 pm
Location: Antwerp Belgium

AFF

Post by Willy »

Ken Wilson wrote: If we don`t change any part of the judging system, it will always remain as it is now. HIGHLY controversial.

.
Kill me if I am wrong : I think that in England people are more conservative than in any other country. It gives England some charm. Driving on the left side of the road, the condiment set (in the UK a salt shaker has one hole, a pepper shaker has numerous holes), preserving the old architecture, etc... For instance : Harrogate has still its beautiful Victorian architecture ... There are enough examples.

However, it is sometimes necessary to change something. Rules for instance. Because life changes very quickly. Now there were more than 200 entries. But also ten years ago ? 15 years ago we still used 8mm films that were very expensive. For three minutes of film we had to pay £8. We were careful and thought twice before shooting a scene. Our films were shorter because we had to keep them short. Maybe at that time there were not 48 hours of film to judge at BIAFF.

I must still have my BIAFF-notes of last century. At that time it was called "Movie 1999" I remember. Perhaps I am wrong again, but I remember that I received notes from every judge individually. Perhaps the judges of the 20th century are also the judges of the 21st century because we don't find any young people anymore. That's an other problem.

In my opinion you are right, Ken and other friends. "It is time for a change" sometimes. Perhaps now. "Time for a change" ... it was also a political slogan at elections in your country some decades ago. It is not wrong to send films on DVD to judges. That's also my opinion. I am also saying these things because one of my Belgian friends was a member of the final BIAFF-panel some years ago and he told me how it worked. He said it was very stressful. For everybody. To be honest I didn't agree with his criticism on some films either. One film only received a few stars, but it won a gold medal at Unica!

In 2008 I was a judge in the final Guernsey Lily panel. We were 7. Old and young. We watched the films that had been chosen or selected by other friends at home. Peter Rouillard already explained the system. There was only one bad thing. The sea was not kind. I still felt seasick when judging the films the day after, but luckily there were still 6 other judges. It only happens once in a century that there is such a heavy storm at sea at the end of July. But : the more judges in a final panel the better I think. It may happen that a judge is not in the mood because of unforeseen circumstances. Imagine : the night before he or she didn't sleep well because ... he or she has a headache and is asked to see 20 films one after another!

After this excellent judging-experience in Guernsey I went back home. I received DVD's from Mary and Peter and I could watch the films again in order to write the notes. I had enough time to do this. The Lily-secretary Mary received letters of thanks from the filmmakers and she sent them to the judges.
Willy Van der Linden
User avatar
Dave Watterson
Posts: 1872
Joined: Sun Jan 28, 2007 11:11 pm
Location: Bath, England
Contact:

Re: How might BIAFF entries be judged?

Post by Dave Watterson »

Quickly: you are right, Willy. For many years every LAFF/BIAFF judge wrote notes on every film they viewed. The catch is that it became harder and harder to find judges willing to write 30 - 40 or more crits working from their memories of seeing each film once. Many fine film judges do not find it easy to write about movies. Hence the current system whereby only one member of eacxh panel writes about a film, so each judge writes about 18 crits.
-Dave
User avatar
Dave Watterson
Posts: 1872
Joined: Sun Jan 28, 2007 11:11 pm
Location: Bath, England
Contact:

Re: How might BIAFF entries be judged?

Post by Dave Watterson »

Two questions:

Should we abandon the IAC practice of not having a film assessed by someone from the same region?

Various ideas floated here aim for consistency of grading by having judges see all or a very large proportion of all the entries.

Where do we find judges?

Any one IAC officer may only know only a handful of people in the movement ... and may have no idea how most of those people would be as judges.

Dave
Mike Shaw

Re: How might BIAFF entries be judged?

Post by Mike Shaw »

My view is that judges shouldn't see films from their own Region - to eliminate any possibility wahtsoever of 'favouritism'. That means the judges will not see all films - but I don't see that as a particular problem: they will see most since, from the statistics in this year's programme, no Region seems to send in no more than around 40. What one region doesn't see is well covered by the other 7 regions I'd say. (I am assuming no overseas judges...)

Where do you find judges? That is the $64,000 question.

How about - outside of the IAC altogether - if you can persuade the pro directors ...

The OSCARs don't have judges as such do they? What is their method of judging - short-listing, and so on.
Peter Copestake
Posts: 340
Joined: Sun Feb 11, 2007 2:23 am
Location: Colne, Lancashire

Re: How might BIAFF entries be judged?

Post by Peter Copestake »

I would like to add my support for the Lily or Cotswold method to be adopted by BIAFF and other IAC competitions. What I particularly liked about the Cotswold was that we knew what weight was given to different aspects, so that, for instance, my particular niggle - whether a film is suitable for a competition/general audience - can be ignored by the entrant who wants to know if he/she has made the best use of the camcorder/editor, in the view of the judge, knowing that even this is a subjective judgement.

I might be able to judge - I've been film making for 30 years - but only for an hour or so at once, so would definitely be in favour of DVDs circulating to the preliminary judges so they have time to check what they see as mistakes were really there.
Peter Copestake
Ken Wilson
Posts: 164
Joined: Mon Jan 29, 2007 11:17 pm

Re: How might BIAFF entries be judged?

Post by Ken Wilson »

I would like to thank recent messages added on here. At least we have some serious discussions now going on which is good.
To give further food for thought, each year in May, Wakefield has a week long drama festival. There are 7 plays, a different one each night performed by 7 different drama groups. An adjudicator who I believe to be professional (I think he is paid) gives a summing up at the end of each show and gives comments on directon, set design, writing or adaptation, acting and the rest. On the Saturday, he sums up that evening`s play and also all the rest from the week, before awarding his prizes. The point of all this re:the IAC, is that he is completely independent and impartial. He judges all aspects of each play and has been trained to do so. It would be great if we could do something like this, but 7 plays and 200+ films are two entirely different things of course. But our judges seem to be made up partially of club members who have volunteered for the task. Hardly a system to gain respect for the quality of the judges comments from the film makers. We spend weeks, months or sometimes years working on a film. The end results deserve to be treated with some respect when judged. It`s far more than just an extra star or more which matters, though this can be painful to the maker if it is judged to be unfair, but the fact that the film is then denied a screening.

Point number 2. As some contributors on this forum seem worried by the numbers of films involved in regard to holding a pre-judging session by people at home, I offered some ideas and ways this would/ could work as an opening gambit. The basis of this was that more films would be seen by more judges and this evens out any genre or theme bias by one individual. But of course often compromises have to be made.
You could modify my original proposition by having the 6 judges involved in round one being split into two groups of three.
One group would be asked to see and judge ALL of the non-fiction (Travelogues/ documentaries) and the other group ALL of the fiction films. This incorporates some of Willy`s ideas of seperating different types of film.
But at judging session 2, the films again are judged together as they are now. The plus points of this change would be (assuming a roughly 50/50 split between fiction and non-fiction) that each judge would see about 100 films instead of 200. It would give more time for consideration or would reduce the time needed for this first round.

As mentioned previously, it`s playing around with numbers and working out the final details, but it seems many people favour a pre-judging session, based on what has been written here.

I was assuming that the first round judges would be largely found from those who exist already. Ideally, yes, it would be preferable to avoid judges from your own region, but I don`t see this as such a major problem if it adds complexity to the system.
Brian Saberton
Posts: 355
Joined: Fri Apr 27, 2007 8:00 pm
Location: Scotland

Re: How might BIAFF entries be judged?

Post by Brian Saberton »

Ken has raised something I've mentioned before concerning the training of judges. Admittedly I'm coming at this from the point of view of judging competitions in camera clubs, and I don't know what happens in England, but here in the West of Scotland we have an organisation for photographic clubs called the Glasgow District Photographic Union and they maintain a list of approved judges that affiliated clubs can access for both internal and inter-club competitions. The point is that you only get on the list if you successfully undergo a days training session on judging techniques. For inter-club competitions run by the GDPU it is compulsory to select judges from the approved list, although clubs are free to choose judges from elsewhere for other competitions if they wish. This system is designed to ensure that the approved judges have demonstrated a level of skill in which clubs can have confidence. However, just as with the amater film movement, there is a big shortage of people willing to judge and I know that the GDPU experiences problems in getting enough people to put their names forward. How practical this would be for the IAC to organise I don't know - perhaps some of the Regions already do something?
Brian Saberton
Ken Wilson
Posts: 164
Joined: Mon Jan 29, 2007 11:17 pm

Re: How might BIAFF entries be judged?

Post by Ken Wilson »

I just thought that I would now add here that I have drawn the attention of the competition manager, David Newman, to this thread on the forum. I don`t know if he follows material written on here, so thought it only right and proper that I should e-mail him directly to let him know about it. I have also e-mailed it to Ron Prosser as our current Chairman. I think that`s about as far as it goes now. These situations tend to die down after each BIAFF until the next time, but hopefully there is some food for thought amongst what has been written and some good may come of it.
At the present time I have no enthusiasm for making new films. It seems fairly pointless with so few places now where we can get our films seen, with the Guernsey Lily not taking place this year and the Morecambe competition not being repeated due to lack of support, BIAFF seems to be our last hope in the UK. I have always enjoyed the festival weekend, so let`s hope that something can be done about the judging situation.
Post Reply