An armed guard strides out of the dark, ready to defend the city. For the first time in 300 years, the public can see Rembrandt’s dynamic painting of The Night Watch as the painter intended it, now that artificial intelligence has been used to reconstruct four missing panels.

When Rembrandt created The Night Watch in 1642, as a commissioned group portrait of Amsterdam’s civic guard, it was both larger and had the main characters off-centre. But in 1715, when the painting was moved from the organisation’s club house, the Kloveniersdoelen, to the city’s town hall, someone cropped it on all sides, putting the action at the centre.

Now, as part of a massive restoration project, carried out in a specially built glass box, the Rijksmuseum is for the first time presenting the painting as it would have been, thanks to the powers of modern computing.

Famous

‘Some pieces were cut off, when in 1715 – around 70 years after it was painted – it was moved…when the marksmen were no longer charged with keeping the peace in the city and defending it against the Spanish,’ said museum director Taco Dibbits at a press conference. ‘It had become a hotel room. Rembrandt was famous, and it was thought too risky to let the painting hang there.

‘In the city hall in the Royal Palace on the Dam, it needed to fit through two doors, it didn’t fit, and the movers got out their scissors, you might say, and snipped off strips. We don’t know why they trimmed the top and bottom, because the room was high enough… We know how large the strips were because copies were made of the Night Watch in those 70 years since it had been painted.’

With the help of one copy in particular, attributed to Gerrit Lundens and on long-term loan from the National Gallery of London, the Rijksmuseum has been able to recreate an impression of how the original painting looked in its entirety.

Robert Erdmann, senior scientist at the Rijksmuseum, said that with the help of artificial neural networks, a computer could be ‘trained’ to translate the missing panels into the style of Rembrandt.

‘The goal was to take the copy by Gerrit Lundens and use it as the basis for making the reconstruction,’ he said. ‘The copy is different in many ways. First it’s only 20% of full scale, second, it has a different style, and third it has a different palette.

‘The way the figures are painted is different, the geometry is a bit different and even the overall placement of figures is quite different. It would not be acceptable to simply take the Lundens copy and blow it up and print it out: it wouldn’t match geometrically, it wouldn’t have the right style or the right palette.

‘So we can use new technology, machine learning, artificial intelligence to help us achieve this task.’

Three different neural networks were developed so that the computer could ‘learn’ the differences in style between the Lundens copy and Rembrandt original, to teach it to translate the missing strips from the copy into the style and palette of The Night Watch.

These were then printed out, varnished, mounted on metal plates and set just slightly in front of the painting to give the impression of the completed work.

Sense of space

‘We tried with a lot of effort to make sure the pieces would be believable but we wouldn’t want to trick our viewers into thinking it was the original Night Watch,’ he told DutchNews.nl. ‘But it’s really striking to see how different the painting feels with the missing parts attached. It completely changes the composition, it’s more spacious and the sense of movement is really enhanced.’

Dibbits said that the fact that the main characters now appear off-centre in the reconstruction is vital to the sense of movement, and the dynamism of the painting.

‘In the original picture, the captain and lieutenant were just off-centre,’ he said. ‘At a time where you couldn’t make a film, you needed to ensure that you painted a story, just before the climax or just after. For the Night Watch he chose the moment just before so that our brains start to wonder what will happen and where they are going, and he achieves a sense of movement.

Man with drum

‘Whoever snipped off pieces clearly thought about it a lot, decided they should be in the middle and neatly snipped off some bystanders but kept the man with a drum, cutting off his back. More dynamic has returned to the painting, they are walking forward, the troops are beginning to march.’

In addition, he added, we can see the railings of a bridge, making clear that they are marching across it, the painting can ‘breathe’ into the sky at the top and the shadows at the bottom, and we see the fuzzy figures of onlookers on the left hand side again.

While the computer may not be able to learn to paint like Rembrandt, in all his masterful characterisation, it does have one advantage: since it can predict how aging looks, the newly restored missing sections were reproduced, complete with cracks.