Translated material
Museums in 4D
These are difficult times for museums, especially small and private ones. You have to pay rent, and the money has to come from visitors who are not there. Even if, with the easing of quarantine, they are allowed to open, the previous flow of guests will not return - the pandemic continues, it will last for a long time, but in the minds of the authorities and ordinary citizens, museums are still not among the vital elements of infrastructure... In addition, a fair share of visitors are tourists from other regions or from abroad, and closed borders negate all the efforts of museum workers to get at least some income.
There are sad cases when interesting projects are deprived of their rented premises. In such cases, it is logical to seek support from the public, but the level of this support depends on how popular the museum is, and support outside its region also depends on how present the museum is on the Internet and whether its employees are sitting on their hands during forced times of emergency. holidays
Perhaps there is a chance to make money by organizing virtual tours. And large and state museums face essentially the same problem: they cannot afford to forget about themselves and cut off funding. How do they deal with this? What can you do? How have museums been “promoted” on the Internet before, here and abroad, and what is being done now?
The recent Museum Day (which essentially turned into a demonstration of Internet projects), as well as a very unusual event - the opening of a new exhibition of the Pushkin Museum exclusively in the virtual space - gave us the idea to talk about this with Dmitry Lobachev , founder and CEO of Navigator4D [ 1 ] , which has been creating digital replicas of museums around the world for many years and has actually released a program for virtually visiting the above-mentioned exhibition. The conversation was conducted by Maxim Borisov .
— What are some ways to virtually walk through museums? From the recent newsletter of the Ministry of Culture of the Moscow Region, from those links that turned out to be intact and working, I found two almost obvious ways: filming of excursions posted on YouTube (this, of course, is useful, but something similar could have been achieved a hundred years ago by ordinary movie camera) and 360° panoramas of museum halls, where you can look around, turn your head, zoom in on some fragment of the situation, move to the next room and even go outside, look at the building from the inside (in the spirit of 3D panoramas on Google and Yandex maps). An example is the Melikhovo estate, the museum-reserve of A.P. Chekhov [ 2 ] . Relatively cheap and cheerful. To be honest, the web studio where I worked in the last century did exactly the same thing.
— There are similar solutions, there are solutions from the Matterport company [ 3 ], they make good panoramas based on 3D photography, it’s quite “dirty”, of course, but the transitions between locations are quite smooth. All this cannot claim the status of a replica and the status of a virtual copy, of course. It is necessary to repeat all the textures, shoot most of the objects in 3D around, remove objects from display cases...
To do this, you need to combine several complex areas - 3D scanning, gaming technologies... Real culture and games (associated with big money) - they are on different planets. It is difficult to combine them and create a competitive solution that will not die when your oxygen is cut off by the organization financing your project, which tomorrow will finance something else - and the previous project is finished... We need a solution that will feed itself - and this is ours The task is to make it pay off.
- Thus, we are all, in principle, familiar with the simplest 3D panoramas; sometimes it’s enough to use maps from Google and Yandex... But Google still has a technology where the raster is stretched onto the outlines of houses and the terrain - so they can see for example, most of Paris, some cities in America... As I understand it, you still went the other way? What kind of 4D? Sounds pretentious...
— Why is the company called 4D? This is 3D plus information, a huge amount of information - layers that contain the most important information and, let's say, complement the 3D. Why "navigator"? Because we are talking about navigation—moving through the halls of virtual museums.
There is enormous potential and a huge reservoir of knowledge in museums, which act as archivists and guardians of very important information - historical, cultural, scientific, etc. When I realized that science and technology had already reached such a level that it was possible to create visual clones - in fact digital replicas of these museum spaces, almost no different from the original, from a regular museum visit (except you can’t touch anything) - I created this company in 2014. In museums, once you make a copy, you can get a basis for further development...
But I created my first web studio back in 1993 in Moscow. Over the past time, we have gained enormous experience in computer graphics, animation, multimedia, television graphics, film graphics, and scientific graphics. But the most important thing is that we have been working with museums for a long time and know how they work from the inside. For education, for further development, for free access to unlimited people from anywhere on the planet, exactly this format is needed - it is necessary to develop tools, an internal platform that allows you to create these museums with high quality and quickly enough. And so that we can create museums of any level - with complex architecture of rooms, spaces and, perhaps, even architectural ensembles, scanning of any exhibits. Our technology itself has undergone four modifications. At first we made the online part on Macromedia Director, the offline part was done on our own engine, which was written in C++. And he and I created, I think, six museums. This is from 2010 to 2012. It was the Vernadsky Geological Museum, the Borodino Panorama, it was the entire Museum of the East, and several other less significant projects. There were also architectural projects. But in fact it was just a simulation, a copy. In the Museum of the East these were scanned objects.
- Is it possible to watch this somewhere now? Or is everything already closed?
- Funny thing. We have all this working, but of all these projects, only the Museum of the Orient lasted several months. In their foyer. And on their website, as well as on the websites of the Geological Museum and the Battle of Borodino Panorama Museum, sections with our first virtual museums lasted about a year, after which everything was removed. I saw that museums do not really use the created products and do not reveal their potential. It was very demotivating. Perhaps the level of photorealism of those products was still insufficient - after all, in fact, our technology itself was just developing. Then we moved to another hypostasis. I realized that we needed a work model in which we could develop products ourselves and own them. I moved to working with museums as a partnership. We have actually stopped asking museums for money (although there is an opportunity to add more). We started looking for interesting museums. But we own the product ourselves - it’s a completely different model, and it worked. But in parallel, we still did work with the Paleontological Museum, it is still on display at their place, they show it at exhibitions, inside the museum, there is a huge screen, 1x2 m, standing in the foyer, you can spin everything on it. We completely recreated the museum building with all the halls and scanned all the individual exhibits, this is more than 50 exhibits in 3D detail. This can be viewed on YouTube at our channel ( youtube.com/watch?v=zDDRSPBSQ6k ).
— A meaningful application that I heard about is to present a museum in this way at exhibitions... You can really install the necessary equipment there, a computer...
— At exhibitions, individual exhibits are shown, when you can spin the exhibits themselves. Because the exhibition itself, the museum itself, is a single space, cultural, historical and scientific... It is organized by eras, periods of the development of the Earth, the geology of the Earth is linked, etc. Why is the museum itself interesting, as a whole? Because this is a well-developed structure, a unified format for presenting information, where the exhibit is built in, etc. This is an interesting thing, it’s like a book, a finished project. Therefore, if we are offered to make only one room in a museum, but they don’t give us the whole museum, then we refuse such work. We are interested in making the entire museum in order to get a complete replica of it and open it to a large number of people, then do translations, etc. We started working with foreign museums and have already done more than ten European museums.
When we start working with museums, we plan to cooperate with them for a very long time, decades. Our task is to set very high standards, to create a complete replica. Accordingly, immediately after the first version we include the entire space, the entire exhibition of the museum. We scan everything that stands alone in 3D. And the result is a finished product in several languages, it can be purchased at the museum. That is, in fact, it is “take the museum with you in your pocket” - after the exhibition, after the visit. You can spend two hours, but you will only see part of the exhibition (this is a problem especially in medium and large museums). People try to return to the museum many times. But this is difficult to do if we are talking about another country, especially an expensive one - you cannot fly to France many times from Russia, from the USA or from Japan, it is clear that this will then be an excellent solution to the problem - take this museum with you, and everything will be in your native language.
At any time on the big screen, with grandparents, friends, you go, without crowds of tourists, with an audio guide, with detailed explanations, you can stay at each exhibit for as long as you need. You can come back any time. In fact, the museum completely moves to your bookshelf, from which you can always get it.
Not only that: the latest iteration of the technology - we are adding VR there. If the user has a virtual helmet... And these products already work with helmets. And we will expand this line to Oculus Rift and others.
- ... And at some point you switched from yours to the most common engine - Unity? Why not on Unreal Engine or not on Source (which is home to Valve's acclaimed new VR game)?
- It is important. On our engine, this thing was developed by only two programmers - they did it for ten years. The engine was not bad, but Unity already had 150 programmers at the time of our transition to it (2013), and its capabilities were already far superior to what we were doing. Well, in our work the most important component is content. This is a copy of space from the outside and from the inside. And through what technical engine the content is displayed for the user is not important. We are currently working on both Unity and Unreal Engine.
- As I understand it, you have two directions - programs that are quite significant, which the user downloads, installs on his computer and watches everything in them, and the second is something like a game stream...
— Yes, this is an interactive stream, when the application is instantly launched on a powerful server cloud and the stream stream is transmitted. All you need is good internet. This solution is promising, we will also develop it. We did it - it works. But the best quality is when you downloaded the archive - and it works on your side. Next up is WebGL. And the stream only works well if you have a good Internet channel. In the future we will do everything, this is important. We saw the demand and have been selling it all to museums since the end of 2016.
The fact that Pushkinsky chose us as partners indicates that our decision was already recognized by the best museums in the world; we really missed this.
That is, now we are working during this pandemic, realizing that we have new contracts on pause. The most important. We gained very good experience, and now our task is to establish such a serious process of manufacturing high-quality museums using our technology from this garage startup. We have been creating the technology for many years, and it is quite complex, there are seven stages, some of which we developed ourselves. We've made sure this works on any configuration of computer, from $200 computers with a built-in graphics card. We are currently creating ported versions for smartphones. All museums will work on all devices. Of course, they are best viewed on computers, but this also works on smartphones and tablets.
The technological chain consists of seven stages. At the entrance, we take tens and sometimes hundreds of thousands of photographs, particularly from drones. Spaces are created both outside and inside, and then the process of simplifying the model takes place while maintaining the detail of the textures, which allows us to get a not very heavy model, where there is very good detail in the desired areas and a huge amount of information in the textures.
You see how you can get there, where the highway, for example, comes from, from which autobahn. Where is the parking? That is, you get to know the territory of the park - for those who have not yet been to the museum, for example, but want to visit it, they are interested. If we make a virtual visit, it begins with an arrival at the museum itself, there is a three-dimensional model of the city, with a stop.
We show all the buildings on a three-dimensional interactive model, all the objects from the outside that you can click on individually and open, you can wander around the park, look at the sculptures, facades of buildings, and get to know it inside.
— You also did the Chauvet Caves? As far as I understand, the museum itself, where visitors are allowed, is also a replica of the cave? So you made a replica of a replica?
— Yes, we made a digital replica from a replica. The reason is very simple - we create copies of real-life exhibition spaces, in which there is a content information load, the connection of all information. Such a space could only have been created on the basis of a correctly located replica.
When the Chauvet Cave was discovered (in 1994), people began to walk into it, people began to change the microclimate there, and the drawings began to deteriorate. And the French government decided to mothball the cave completely, depriving it of oxygen in order to preserve it. But before conservation, the consortium of companies that created the replica scanned the cave in all details, and from this scan they recreated it - they moved the replica, which is located several kilometers from the cave itself, closer to the highway, and made a park there. Several French companies created a conglomerate that built a replica for 55 million euros, and also created a stunning architectural educational space, in which the main focus is the cave itself, where all the main frescoes and halls were recreated...
-Did you film all this? What's your impression?
“We spent more than two weeks of continuous filming there. Moreover, filming could only be done when there were no visitors. Accordingly, it was from 5:30 a.m. to 8:30 a.m.—three hours. And an hour and a half after the visitors. Accordingly, in this mode, visiting several times, we filmed everything inside and, in addition, we filmed a separately built museum building, in which a dozen stuffed animals that lived at that time were very beautifully recreated. These drawings are 35 thousand years old, and in appearance they surpass any analogues around the world. This cave is unique, the only cave in the world and a UNESCO heritage site.
— Are these some kind of plastic walls?
— This is reinforced concrete with added concrete and gypsum. This design is very durable. Even wet stalagmites have been recreated, that is, the feeling that they are wet, water is dripping. It's done amazingly, well lit. The cave provided us with a description and audio guides. Besides the fact that we made a park, we made a replica of a cave, we made this museum, we did everything. Now visiting a museum is a 100% visit, the only thing is that you don’t need to buy a ticket, get a French visa, spend a lot of money, etc. And this allows you to open the museum to the whole world.
Museums are the building blocks that make up the library that we want to deliver to schools, colleges and institutes. We will give licenses to schools, institutes and colleges so that they can use this in their educational process - access to these museums.
“ You gave me access to a demonstration of this cave on my computer?”
- Yes, we are now finishing three projects - the Villa Ephrussi de Rothschild , the Chauvet Cave and the largest - such an architectural masterpiece - St. Nicholas Cathedral in Nice . Nicholas II built it with his own money in 1912 on the site of the death of the Tsarevich, who died suddenly in Nice. This is the largest Orthodox cathedral in Europe. And there are amazing interiors and exteriors, it’s such a cathedral-museum, we also scanned it completely.
And we are releasing these three major French projects, and here is the Pushkin Museum , which is free to visit; from the very beginning we had such an agreement that this museum is free to visit. Both the Pushkin Museum itself and we are distributing it free of charge so that everyone can see this exhibition, which opened in a closed museum [ 4 ]. Quarantine struck just a few days before the opening of the exhibition. We managed to scan it and completely recreate it in four days. No one except us and the staff of the Pushkin Museum themselves saw her.
“ I walked around and was amazed that you so lovingly recreated everything, right down to the cracks in the marble and some fire-fighting devices and pillars, but somehow it was unlucky that the objects themselves—the paintings—were flat. They are wonderful, but not everyone will be inspired by the coolness of the technology, its greatest strengths... Moreover, gamers (and we have all tried to play) are not quite accustomed to motion controls. And most importantly, you have to download a huge 1.7-gigabyte file.
- Well, this is a pilot project with Pushkinsky. The task was to show the atmosphere that you can be in this. The final task is to create a complete replica of Pushkinsky in 4D.
Dmitry Lobachev
Interviewed by Maxim Borisov