Im not sure there is a debate or that this is subjective. The market research indicated there was not enough demand to justify the costs of compliance with US emissions standards. Yes there is a niche market, however a majority of US consumers had a negative opinion of diesels in the late 70s and onward.
Toyota is incredibly aware of their markets. Manufacturers float
the idea of a diesel via rumors or concept cars, then gauge interest off the response. Many times they do not feel there is enough enthusiasm to justify the risks of bringing it to market at a higher price tag and often with more requirements like urea systems.
When I interned at Toyota Motor Sales in Torrance it started with a 2 day long indoctrination in the company history and the TPS. They are VERY self aware and acknowledge this is from their initial US offerings being perceived as under-powered, small and inadequate for American drivers. They know their strength is "quality" which was defined to me by a Quality Engineer there as such: "Quality is what the customer expects." (Ive since seen a similar quote "quality is what the customer says it is.") Toyota, a brand hyper focused on quality as part of their identity, does not want to fail to deliver on any customer expectations. In the case of diesels, that is the American consumer's expectations of performance which rightly or wrongly is juxtaposed with American V8s. People have criticized them as being risk adverse because of this, but the results of decades of success speak for themselves.