The root of “Pinyin accent” is interference from the native language, but it’s not caused by the fact that Pinyin and, say, English use the same character set. Rather it’s caused by the nature of the reading.
When you read English (as a fluent reader) you are not sounding out each and every word. You are being “reminded” of language that already exists in your head and makes sense to you immediately in the order it’s presented. Often people only glance at the shape of the word, not all its letters. You’ve seen those Internet memes with paragraphs with letters replaced with numbers and stuff like that, and they’re still very readable to a fluent reader.
Pinyin is not generally used that way. In legacy teaching methods, Pinyin is used to teach how to pronounce words that are not yet known. The sound of the word isn’t there in the brain waiting to be quickly matched to the visual form. Rather, the Pinyin word is (often painfully) sounded out. Non-fluent speakers who don’t have the “Chinese voice” in their heads already don’t have a reliable set of Chinese sounds to attach to those Pinyin letters and combinations (like -an, etc.) When the brain comes up short on authentic sounds to use, it falls back on sounds it’s matched to those letters before (English, or really any language using the Roman alphabet that the student reads well or not).
A student taught with comprehensible input doesn’t have to sound out Pinyin, because (in class, during the period they’re learning to read Pinyin gradually, not through dense direct instruction) they never have to read Pinyin alone. If my students see Pinyin, they are either hearing that word simultaneously from me (I point to each one as I say it) or the Pinyin corresponds to words they have already heard literally hundreds of times before. Their built-in “Chinese voice” also “suggests” what the word probably is through context. These things don’t happen with legacy-taught students because they don’t hear enough understandable Chinese to build that voice.
This is also how we teach students to read characters without having Pinyin involved. I use Pinyin for oral input and to make lists and notes for students, but when they read, they only see characters, right from the first day, because the “Chinese voice” is sufficiently strong to let their ears inform their eyes as to what words those squiggles are. I know it’s strong enough for a very simple reason: I don’t have a class or student read until their response to the spoken language included in the reading is strong and immediate.
As time goes on obviously students encounter more words and read more complicated things, but the crucial period is the first, say, 50 hours of instruction. That’s when you make a reader. And it’s also when you build a user of Pinyin who uses it automatically to arrive at the correct sounds.