This question is difficult to answer definitively because virtual memory is an integral part of modern systems are designed to support virtual memory and most software is written and optimized using systems with virtual memory.
However, in the early 2000s Microsoft Research developed a research OS called Signularity that, among other things, did not rely on virtual memory for process isolation. As part of this project they published a paper where they analyzed the overhead of hardware support for process isolation. The paper is entitled Deconstructing Process Isolation (non-paywall link here). In the paper the researchers write:
Most operating systems use a CPU’s memory management hardware to
provide process isolation, using two mechanisms. First, processes are
only allowed access to certain pages of physical memory. Second,
privilege levels prevent untrusted code from manipulating the system
resources that implement processes, for example, the memory management
unit (MMU) or interrupt controllers. These mechanisms’ non-trivial
performance costs are largely hidden, since there is no widely used
alternative approach to compare them to. Mapping from virtual to
physical addresses can incur overheads up to 10–30% due to exception
handling, inline TLB lookup, TLB reloads, and maintenance of kernel
data structures such as page tables [29]. In addition, virtual memory
and privilege levels increase the cost of inter-process communication.
Later in the paper they write:
Virtual memory systems (with the exception of software-only systems
such as SPUR [46]) rely on a hardware cache of address translations to
avoid accessing page tables at every processor cache miss. Managing
TLB entries has a cost, which Jacob and Mudge estimated at 5–10% on a
simulated MIPS-like processor [29]. The virtual memory system also
brings its data, and in some systems, code as well, into a processor’s
caches, which evicts user code and data. Jacob and Mudge estimate
that, with small caches, these induced misses can increase the
overhead to 10–20%. Furthermore, they found that virtual memory
induced interrupts can increase the overhead to 10–30%. Other studies
found similar or even higher overheads, though the actual costs are
very dependent on system details and benchmarks [3, 6, 10, 26, 36, 40,
41]. In addition, TLB access is on the critical path of many processor
designs [2, 30] and so might affect processor clock speed.
Overall I would take these results with a grain of salt since the research is promoting an alternative system. But clearly there is some overhead associated with implementing virtual memory, and this paper gives one attempt to quantify some of these overheads (within the context of evaluating a possible alternative). I recommend reading the paper for more detail.