Even when you could swear others are looking at you, there’s a good chance your brain is playing a trick, University of Sydney researchers say.

You know the feeling: you walk into a room and catch people looking at you. They’re staring, right?

Not always. The fear that people are staring is often your brain playing tricks on you.

Researchers at the University of Sydney have discovered that when your brain is unsure of what you’re seeing, it tells itself someone is looking at you and perhaps even passing judgement.

“Judging if others are looking at us may come naturally, but it’s actually not that simple—our brains have to do a lot of work behind the scenes,” lead researcher Colin Clifford, a professor of psychology at the University of Sydney, said in a press release.

Our brains determine if someone is looking at us by figuring out where their eyes are pointing and the direction of their head, but without all the necessary information, the brain fills in the blanks using information from prior experience.

Researchers tested this by creating images of faces and asking test subjects where they believed the people pictured were looking. They intentionally made it difficult to determine where the figures’ eyes were pointing. The test subjects’ brains made a lot of assumptions during the test.

“It turns out that we’re hard-wired to believe that others are staring at us, especially when we’re uncertain,” Clifford said. “So gaze perception doesn’t only involve visual cues—our brains generate assumptions from our experiences and match them with what we see at a particular moment.”

The study, published in the journal Current Biology, concludes that the adult nervous system incorporates prior experiences with information about a person’s gaze and applies it to a situation it’s unfamiliar with.

Your brain does this every day outside of the laboratory, especially when it’s hard to see someone’s eyes, such as at night or when the person is wearing sunglasses.

There are plenty of reasons to worry about whether or not someone is looking at you. After all, fear and vanity are both important elements of the human psyche.

Children with autism, on the one hand, are less able to tell if someone is looking at them, whereas someone with social anxiety constantly fears that others are staring, researchers said.

“Direct gaze can signal dominance or a threat, and if you perceive something as a threat, you would not want to miss it,” Clifford said. “So assuming that the other person is looking at you may simply be a safer strategy.”

When someone looks at you, it may also be a sign that they want to communicate, so your body goes on alert, anticipating the interaction.

Babies, the researchers said, prefer it when people gaze directly at them, so Clifford’s team wants to further explore whether this behavior is learned or innate.

For now, remember: you’re not being paranoid. You’re just built that way.