The first thing to do is to establish the size of the square. If we assume that the L-triomino is made up of three unit squares, the diameter of the circle will be the hypotenuse of a triangle with base 5 units and height 3 units.

This diameter, and therefore also the side length of the square, is the square root of 34.

If we place an L-triomino in each corner, then tilt the square 45 degrees, how large a square can we fit in the remaining space?

The side of the whole square is sqrt(34), so the diagonal is sqrt(68). From this we need to take off twice the diagonal of a 1.5 x 1.5 square to find the side length of the inner square.

These combine to be sqrt(18). So sqrt(68)-sqrt(18) = 4.0036…

Just to be safe we need to confirm that the diagonal of the inner square doesn’t exceed the side length of the larger square, or else the corners will extend beyond the original square. This isn’t the case so we are safe to continue.

How many L-triominoes can we fit in this 4x4 square? If we place one in each corner we have space for a fifth in the middle. Added to the four we already placed means that overall we can fit 9 L-triominoes in the square of area 34 square units.

Of course this doesn’t prove that 10 or 11 isn’t possible (more than 11 is impossible as the area of the triominoes would be greater than that of the square), but such a proof is a little more involved.