First let's look at using various numbers of colors:
1 color - 1 way (all corners same color)
2 colors - 4 ways
(2 with 3 and 1, 1 checkerboard, and 1 stripes)
3 colors - 9 ways (6 with a stripe, 3 checkerboard)
4 colors - 6 ways
(put color #1 top left. 3 choices for bottom right, and 2 ways to put the others)
Then for each of those there are various ways to choose the colors.
If there are N colors available, we can choose:
1 color in Nc1 = N ways
2 colors in Nc2 = N (N-1) / 2 ways
3 colors in Nc3 = N (N-1) (N-2) / 6 ways
4 colors in Nc4 = N (N-1) (N-2) (N-3) / 24 ways
So for various numbers of colors we can color the 4 sections of a square in ...
1 color: (1c1 * 1) = 1 tile
2 colors: (2c1 * 1) + (2c2 * 4) = 6 tiles
3 colors: (3c1 * 1) + (3c2 * 4) + (3c3 * 9) = 24 tiles
4 colors: (4c1 * 1) + (4c2 * 4) + (4c3 * 9) + (4c4 * 6) = 70 tiles
5 colors: (5c1 * 1) + (5c2 * 4) + (5c3 * 9) + (5c4 * 6) = 165 tiles
This type of counting is addressed by Burnside's Lemma.
For a general formula for n colors see:
OEIS Sequence A006528
(n^4 + n^2 + 2*n)/4
and for a variation where reflections as well as rotations
are not counted:
OEIS Sequence A002817
n*(n+1)*(n^2+n+2)/8