Depending upon the processor in the Note 4, this doesn't come as a surprise to me. If it's an Exynos processor rather than the SD805, then it makes sense that the newer Exynos processor in the S6 would be noticeably faster. It wouldn't even surprise me if the newer Exynos processor is more powerful than the 805, which is a couple SOCs old. The Exynos processors are slightly behind the equivalent Snapdragons, but not by much, so this all depends on which model of Note 4 that is.
But hey, the S6 is newer anyways, so it SHOULD be faster!