If you are making a DVD (rather than a bluray), Encore is going to transcode your mp4 into an m2v and an audio file (pcm or a3c depending on your project settings).
When your m2v file is not good quality, you need to fix that, not use an mp4. But see what you get. Look in the project panel and find the "DVD Transcode Settings" column. What does it say for your mp4?
I have created my DVD menu with an mp4 inside the timeline of my Encore project...
I don't know what you did. My assumption is that you have a video asset in your Encore project (an mp4). That video was used to create a timeline. You have linked a button on your menu to the timeline. Is that it?.