Our friendship is hot topic among the office gossips

To be clear, I haven't slept with this woman or even kissed her, but we are close friends.
We message and call outside of work (we usually speak when I'm taking the dog out at night), and I admit we confide in each other about our partners and life at home.
She's younger - 26 - and has been married for a year, and it's not going great. I feel a bit lonely sometimes, as my wife is so involved with our two kids. I admit I haven't told my wife how friendly I am with this other woman and she doesn't know we message each other.
The colleague who confronted me says it's obvious to the whole office that we fancy each other because we flirt all the time.
If I am flirting, it's not something I'm doing consciously.
I felt really wound up by this and told her she was wrong and not to get involved in things that don't concern her. Am I right or is there something in what she's saying?
Coleen says
WELL, you might not have had sex, but it sounds like an emotional affair because there's an intimacy to it - you've become close, you're sharing details about your partners and you're consciously hiding this friendship and the messages from your wife.
There's nothing transparent about it and you have to ask yourself what your wife would think if she saw these messages. And what would you think if you found similar messages on ...