A common thought among people starting bird recording is to use something like digital notetakers or the similar function on their phone. Digital notetakers tend to be small, handy, and reasonably cheap. They are, however, designed for the specific application of recording speech at a fairly close range1], and design compromises mean the results on anything other than speech are rarely any good. Sometimes the advertising on these is misleading, claiming wav recording as opposed to mp3. While mp3 recordings are by compressed by definition, and wavs are generally not compressed, notetakers often record speech using ADPCM compression, and record only the voice frequency range. Some more recent (2016) designs can be set to record uncompressed WAV files, so if you have such a machine it is worth perusing the manual. All of this is correct in a voice recorder to give you the longest recording quality for the price. And absolutely horrendous for nature recording.
The degree of the distortion is hard to put across in words, but easy to hear - listen to these two recordings. One was made using a Sony MZ-NH700 HiMD recorder in PCM mode, using a Sennheiser MKE300 mic. The other was made, simultaneously and from the same location, using an Olympus VN-120PC digital voice recorder in HiQuality mode at high sensitivity. Olympus, by the way, do make recorders that are suitable for wildlife sound recording - the Olympus LS10 is a very good machine.
This pits a digital voice recorder against a not particularly ambitious setup. The autumn flock of long-tailed tits were within about 5m of the recorders. About 50s into the recording a light breeze picks up, and you can hear that the voice recorder is not particularly resistant to wind noise.
Sonograms from HiMD(left) and Olympus VN120PC(right). The restricted frequency range of the digital voice recorder is apparent, and the sonogram is indistinct and smeared compared to the HiMD recording.
Finally, another pair of recordings, this time of a blackcap recorded via a MKE300 into a VN-120 and then into the HiMD. I have converted the HiMD recording to a sample rate of 22.05kHz which is the same as the VN120 in the interests of fairness. There is still a significant difference in sound quality, showing that sample rate and the mic quality are not at fault. The recordings are of the same bird and the same bird/mic locations, separated by about 10 seconds.
to confirm your suspicions, the second (lowest) sound was the digital voice recorder!