On 1 September 1939, the Second World War began. Although Lithuania was still independent, many refugees from occupied Poland started moving towards it, hoping to emigrate to other countries after spending at least some time in safety here. However, in June 1940, the Soviet Union occupied Lithuania. When the Winter War broke out in Finland, and the Soviets agreed to release the Jewish people, the only way to escape was through the Soviet Union. However, it was still not that simple. In order to leave occupied Lithuania, war refugees had to obtain transit visas issued by the Japanese consulate. More than a hundred Jewish people began arriving daily at the Japanese Consulate to obtain these visas and to reach America or other possible destinations via Japan. The transit visas issued by Sugihara saved more than 6,000 war refugees who were turned back to Lithuania at the beginning of the war.

The Japanese consulate at the time was located at a villa in the Zaliakalnis area of Kaunas. While the building might look relatively small from the street facade with only one floor, when it is observed from the backyard, it has two floors and an attic. The building is located on a plinth from the front facade, and to reach the main entrance, there is an L-shaped stair. The first part of the stairs has four steps, and the second part has five; there is a curved balustrade next to the stairs, which is one of the most eye-catching elements of the facade. At the back facade, there are two balconies which are curved on one side. There is also another entrance to the building from its side facade, with a small eave.

In 2017, a unique campaign raised money to renovate the museum building. Together with local craftsmen, work was carried out on window frames, walls, windowsills and floors.