That cannot be answered by a simple yes or no. It depends.
Let's take the 5.4 example.
If you follow the current branch it leads us to megi's orangepi-5.4 branch. If you check the Makefile (https://github.com/megous/linux/blob/orange-pi-5.4/Makefile)
you will notice that the actual version of this branch is 5.4.18. But when you compile the kernel it is actually 5.4.47 or something like that.
This means these patches come from Armbian. If you check the patch directory for your board family (https://github.com/armbian/build/tree/master/patch/kernel/sunxi-current)
you can see that the upstream patches are added here. If you need a very specific kernel version remove or rename the patches you do not need.
That was a quite easy example to get what you want.
On other sources, vanilla for example, it is a bit tougher. You would need to specify a commit at which point you want to use the sources.
Sometimes a board is fixed to a specific commit or kernel version because it is known that newer version of the same kernel branch are known to be broken and (as most of the times) nobody has time or resources to deal with it.
I made videos showing how to do this.
It is now also possible to build Armbian with an ARM64 SBC. Tho best to use Balbes images for that.
I think the main build script now can do that. In this video I had to use another branch. So that info is outdated. All others should still be valid.