This problem comes from the defunct Wall Street Journal Varsity Math Week collection.
“The coach then shows the team the diagram to the left and asks: What is the maximum area of a rectangle contained entirely within a triangle with sides of 9, 10 and 17?”
I changed the numbers a bit to make my calculations easier, but left the problem otherwise unchanged. When I checked the Varsity Math Week solution, I saw they used a simplifying formula that I could not remember. I also believed their solution left out a justification for the maximal area. Besides an intuitive solution for this, I also included a calculus version.
See the Triangular Boundary Problem for solutions.