Why is someone telling you to go faster?
I say it is all up to you. If there's a time you want to get, or someone faster you want to run with, then maybe you want to work on your speed. If you want to enjoy a leisurely run, rather than a race, you don't have to try to get faster.
And you don't really have to pick going either fast or slow. If you don't want to work on speed now, you may change your mind and try it later.
There is actually a popular book on running, "No Need for Speed."
There is a dimension of the running experience that involves pushing yourself to make improvements in strength, endurance, and speed, and competing with others. I have found it enjoyable.
But there is a lot to enjoy without trying to go fast.