# Oral argument — Stupp Corporation v. United States > Briefs, arguments, decisions, and more. URL: https://www.frixlaw.com/law-library/documents/argument%3A97596 ## Record - **Collection:** Oral argument - **Document type:** Oral argument - **Published:** March 5, 2025 ## Text We have six cases on our calendar this morning. Two of them are submitted on the briefs and will therefore not be argued. Our first case is Stupp Corporation et al. v. United States in Hyundai, Steele, 2023-16-63. Mr. Winton. Good morning. May it please the Court, I'm Jeffrey Winton of Winton and Chapman. Here today on behalf of defendant appellant Sayers Steele Corporation. This is our second time before the Court on this case. In the first go-around, the Court held that when Commerce applies a mathematical tool like Cohen's d, it has to apply it in a manner consistent with the tool's assumptions. And the Court said if it doesn't, then that might transform what might be a conservative cutoff into a meaningless comparator. And so the Court remanded this case back to Commerce to explain why it was appropriate, either why the assumptions underlying Cohen's d were met in this case, or why Commerce could ignore those assumptions. In its remand determination, Commerce conducted a limited review, only addressing the articles that the Court had cited. Commerce asked us to submit those articles on the record. They did not give us an opportunity to expand the record. They did give the other parties an opportunity to rebut the articles we had submitted. And Commerce then reviewed the articles and concluded that none of those articles really addressed what Commerce was doing in its differential pricing analysis. The articles were addressing the use of Cohen's d in normal statistical practice in analyzing samples, and Commerce said we're not analyzing samples, so we can disregard what's in those articles. The conclusion is that Commerce's use of Cohen's d is not consistent with normal statistical practice. Any doubt you might have would be removed by a report that Professor Hedge submitted subsequently in another case, and I know it's not on the record here. But if we ever were to come back again, he submitted in all the cases now. And so if this Court is going to say only Commerce could decide based only on the articles that were cited, you will then have a subsequent case where you'll have to deal with Professor Hedge's report. Professor Hedge's is a big deal in effect size. His Hedge's g is as famous as Cohen's d. It's very similar. It uses a different denominator. And, in fact, Professor Hedge's was so upset by what Commerce did that he subsequently wrote an article, which he submitted in the Journal of Educational and Psychological Measurement, explaining why the use of Cohen's d when the data was not normal or did not have equal variances. Can I ask you, it seems to me that Commerce has a lot of discretion here. They have to act in a reasonable manner. And I recognize in the remand we said some things about what we were looking for in terms of how they were exercising their discretion. But I think the question is, how much discretion do they have left after the remand? So, for example, you write at page 44 of your brief that before Commerce can rely on the rule of thumb, it must demonstrate that the phenomenon from which the rule of thumb was derived has some rational relationship to the situation for which Commerce is applying it. That makes common sense to me. But isn't that Commerce's call to make? What authority would you cite that says Commerce has to, you know, if it uses a rule of thumb, use it in a way consistent with how the rule of thumb was devised? Well, it's not that it has to use a rule of thumb. So there are two issues here. One is Commerce's position was, we're using this normal statistical test, and therefore it's reasonable because we're just following statistical practice. And that's not true. The second is Commerce says, and I think this— You mean that's not true because the prerequisites for application are not met. Right, exactly. Let me just make sure I understand. From that, what you're saying is maybe there's not a blanket rule that Commerce has to use a statistical test in the way that the statistician who developed it used it or developed it. But if they say that's what they're doing, then they have to do what they say they're doing. Well, there are two issues. And I think the trade court said, okay, I don't think they're doing what Professor Cohen said to do, but it looks reasonable to me, right? And that's the issue is not—they're not doing what Professor Cohen said. So the question is, is this reasonable? Is it a reasonable methodology? And Commerce has discretion, but it still has to be reasonable. They can't use something that's not reasonable. What we would argue is when the conditions that are specified by Professor Cohen are not satisfied, the Cohen's d-test basically gives you a random result, right? Judge Kelly at the trade court seemed impressed that Commerce submitted statistics, well, this only affects 20% of the cases, and so we shouldn't be too worried about it. But the issue is, is it getting the right 20%? If I told you that 20% of the motorists on I-95 were speeding, and so the police decided let's pull one out of five motorists over at random, you'd say that's not right because you pull one out of five motorists over at random, you're not getting—one out of five of those will be someone who's speeding. Four out of five won't be people who are speeding. You're getting the wrong—you're giving tickets to the wrong people. And that's the concern here. When you're not using Cohen's d properly, you get basically random results, which the court recognized in its previous decision when it said you get a meaningless comparison. I'm not sure which it was. And so you're left with, you know, is this something that's reasonable? Now Commerce would tell you it's reasonable because it's based on real-world observations. It's based on the heights of teenage girls. But again, should I understand your argument is they don't necessarily have to defend what they're doing based on its comparability to real-world, you know, data, but if they choose to give that as the basis for why what they're doing is reasonable, then that ought to be true. Is that your position? Yes, yes. So there's two positions. One is if they were using a statistical test in the proper way, if the conditions for using Cohen's d were met in this case, I would say I don't have an argument. They used Cohen's d properly, but they didn't. The second is, okay, is it still reasonable for them to do this? You know, is it reasonable to say a measure based on the heights of teenage girls and the IQs of different students tells you something about whether there's a pattern in our prices? And of course, Commerce has defined pattern, and we agree with this, as something that wouldn't arise by chance. So the question is, okay, can you use the heights of teenage girls or the IQs? The problem is both the heights of girls, heights of everybody, and IQs are normally distributed data. Actually, it satisfies the requirements for using Cohen's d. So saying we're relying on real-world data doesn't move you anywhere from, but you have to show the requirements for using Cohen's d are satisfied. Counsel, I think there's another case relating to Cohen's d pending before this court. I think it might have been argued in January. Marmon, yes. How relevant is that case to this case? Or because they deal with different facts, is it not that relevant? I would say I think it's relevant. I was listening to the argument the other day. I would think. There's a lot of relevance. I don't 100% agree with the appellants in that case, but I think largely I agree with the appellants in that case. How do you differ from them? Well, because— With respect to the Cohen's d issue. Yeah, because— There were two issues in that case. Yeah, there was a verification issue. Just on the Cohen's d. On the Cohen's d, I think the concept of effect size, if applied properly, does what commerce wants to do, right? The issue here is you see a difference in our prices. Is that difference in prices something that just happened because, you know, at random, you know, prices vary? Or is it something that's a pattern, something that didn't happen by chance? And effect size is a measure, can be used as a measure of that, but you have to use effect size properly. I'm not sure that, you know, the Marmon appellants would agree with me on that, but I think largely they do. They focus on things like overlap and such, which I think is relevant but not critical here. Okay. We remanded, as you pointed out when you started, for a further explanation. What's your best example of how, in your view, commerce failed to give us the explanation we were looking for? It's a long decision, but when you go through it, it's explaining why, and I should have this right in front of me and I apologize, but they go through and they say, you know, these are addressing a different situation than what we're addressing. And I agree. Commerce is using this in a way that statisticians don't use Cohen's d. That is clear. They focus very extensively on the distinction that they draw, commerce, between use of a sample versus use of the full population. Now your answer to that, I take it, is that Cohen's d and the prerequisites for the application of Cohen's d apply whether you're talking about a sample or a full population. Yes, I think that's right. If you can expand on that. Sure. That's such an important factor. Let me, and I apologize because this requires me to get into little weeds of statistics. That's all right. None of us are statisticians. A normal distribution is defined entirely by two items, the mean, the average point, and the standard deviation. And Cohen said the standard deviation should be the same. So you have two distributions that look exactly the same. The only difference between them is the average, the mean. If you think of heights of 13-year-old girls, the average height of a 13-year-old girl is 5'2". The average height of a 17-year-old girl is 5'4". There are two curves and they look exactly the same, but one is shifted to the right by two inches. That's what Cohen's D is. You have a normal distribution and that you could have the entire population because the features of a normal distribution don't depend on whether it's a sample or a population. It depends on it's a normal distribution. That's what it means. And you can say, you know, Professor Cohen actually says, look, if it's two inches, we could just say two inches. Everybody understands what two inches are. But in other contexts, we don't know what the difference means. And so we're going to express that two inches not as inches, but in terms of standard deviations. And so if the standard deviation of our 13-year-old girls and our 17-year-old girls is 2.5 inches and the difference is two inches, we can calculate a D of 0.8. But that only makes it, otherwise the two curves are exactly the same. One is just shifted over by two inches or by 0.8 standard deviations. And you can describe the difference between those distributions simply by saying that. One is 0.8 standard deviations shifted from the other. If you don't know that the data is normal, what if it's like bimodal where it's got two peaks? Or what if it's just a straight distribution? There's an equal number of cells at every price from a minimum to maximum. So it looks like this. Well, then you can't, or what if it's a different standard deviation? One looks like this and one looks like this or something different. Then you can't describe that difference simply by saying, oh, all we care about is the difference in the average. You can't say, okay, yes, there's a two-inch difference in the average. But what does that mean? It's not the same as if the two curves are the same. What if the 17-year-old girls are all either 5'8 or 5' tall and nobody's in between? But the 13-year-old girls is a normal, nice curve. Those are not, you can't just say the average tells us what's different. You have to look into it in more detail. There are other things going on there. And that's basic statistics. There's a whole branch of statistics to deal with situations where you don't have a normal distribution. But it has nothing to do with samples and populations. It has to do with whether the data is normally distributed or not. Counsel, you wanted to save five minutes. You're down to two. Oh, thanks. You can save it or continue. I always feel that I should leave you wanting more. So let me save two minutes and allow the other side to speak. Thank you. Mr. Kipora, you're going to take 12 minutes. Good morning, Your Honor. So may it please the Court. Robert Kipora on behalf of the United States. Following this Court's partial remand in its prior decision, Commerce addressed the concerns raised by the Court in its earlier decision. Specifically, Commerce addressed the points regarding comparison groups that were not normally distributed, having small number of data points, or having disparate variances. Now, we recognize the Court's concerns on these issues. But as Commerce demonstrated in its remand determination, the Cohen's d equation is a robust mathematical tool used for the purpose of telling the difference between two groups. And I want to point out something here at the top, which is that the Cohen's d itself is one part of the differential pricing analysis. The Cohen's d coefficient does not tell you whether or not there's a pattern. It's just the first step in that process. Before Commerce will determine that there's a pattern, they look to the ratio test, which this Court already addressed in its prior decisions. So, again, as already has been addressed here, Commerce walked through the different academic literature that was highlighted in the Court's previous decision, and it's already been stated, that the response to that is the concerns that are raised in those various papers deal with a statistical difference, not a practical difference. And the point here is that the academic literature on this is silent as to the question of can you use Cohen's d on a population that is not normally distributed or has these other issues. Let me ask you a question about the point you made just a moment ago about saying, well, there are other tests to the ratio test and so forth that are used here in addition to Cohen's d. If we were to say that Cohen's d is simply inapplicable under the circumstances such as in this case, do you think that the judgment in this case from Commerce could be sustained without any regard for Cohen's d? I'm not sure when you say that Cohen's d is inapplicable. In other words, we say, well, simply under circumstances such as this, you cannot rely on Cohen's d. Could you then say, ah, but we have two other tests which demonstrate to us, to our satisfaction, that there is dumping in this case. Now, would that be a position that you think Commerce could defensively take in this case? Possibly, Your Honor. I think it would depend on the contours of the court's decision if it were to go that way. I think that certainly we always turn back to the statute here, which is directing Commerce to do this. Commerce has an instruction from Congress that they are to look for these patterns and shift to an alternative comparison methodology if they find a pattern. So if the court were to find that, I don't think Commerce would move away from a practice of looking for patterns or potentially applying a different methodology. And as far as the rest of the differential pricing analysis, yes, it's possible to continue using those. You know, the meaningful difference test, the ratio test, these things have already been affirmed. If the court takes issue with the Cohen's d itself, it's possible for Commerce to determine another way to show this difference. Another way based on this record? In other words, based on what Commerce has done so far with respect to the other two tests? Or based on going back and starting over? The answer to the first question is yes, they can do it on this record because what we're talking about here is a methodology. Now, again, depending on what the court says about the Cohen's d and... Well, let's suppose hypothetically the court says Cohen's d in circumstances such as this is simply not reliable enough to be given any credence. And therefore, you're down to the two remaining tests. And my question really is, can you say, well, those tests independently establish everything we need to find in order to find dumping in this case? Or would you have to go back and say, well, we'll approach this in another way or we'll use a different test? I think you would have to come up with an explanation for how they were showing the difference. Because that's all Cohen's d is doing. It's just saying, okay, we've got a test group, which is the group we're analyzing for whether or not there's targeted dumping in the control group, which is the rest of the sales. And they run this multiple times as they look through, whether it's regions, purchasers, time periods. There has to be some methodology for determining whether or not the test and control groups are different. They're using Cohen's d to do that. And I think there's one other thing that this leads me to here that's been raised in this case. It's this idea that somehow Cohen's d is producing false results or false positives or something to that effect. And again, a lot of those comments come from places in the academic literature where they're, again, talking about these statistical results. If you have a sample that's not reflective of its population, then, yes, comparing samples is meaningless. You're not going to get a result that tells you anything about the populations. What Commerce has said is we don't need to worry about that because we have the full population. And I'd also like to use as an example here the court's own hypothetical from its previous decision where it raised a concern about essentially extreme results coming from the Cohen's d test, where you have a test group and a control group where the sales data are clustered very closely together and the values between the two groups are close. Commerce ran through this hypothetical in its remand determination, and they pushed it to the absolute extremes. We have a test group, all sales are at $100, and the test group and the control group is all at $101. So you have zero variance. In that instance, the Cohen's d coefficient goes to infinite, and that sounds like an extreme result, but it's not a false result because all that that's telling us is it's a mathematical determination that these two groups are different. There's not a single price in those two groups that is different. So that's all the test is doing, and then we go through... Not a single price that's the same. Excuse me, yes, not a single price that's the same. I'm sorry, I misspoke. Now, they're close together, but there's no overlap between them, and so then we move on to these other tests. And that's where Commerce explained that in that example, you could not shift to an alternative methodology because the meaningful difference test would show that you wouldn't cross the de minimis threshold. They're too close together, the test and control groups. So again, all the Cohen's d test is doing is saying whether or not these two things are different, and then we proceed to the rest of the analysis to determine whether or not that matters. Am I right that the rest of the analysis, the second step we call the ratio test? That's correct. Isn't that inherently tied to Cohen's d because it's a measure of how many times you pass or how much data passes Cohen's d? It is tied to Cohen's d, Your Honor, but there's a key point that I want to make here that I think there may have been some confusion about earlier. The ratio test is based on value of U.S. sales, and the reason why that's important, and the trial court highlighted this, is that let's say you have $100 million in U.S. sales, and you run the Cohen's d test on $2 million worth of sales. It doesn't matter what happens in the Cohen's d test because in order to even consider an alternative methodology, you have to have more than 33% of total sales crossing that threshold. So $33 million in your example? At least, yes. Okay, that is helpful, but going back to where Judge Bryson was exploring with you, what would we do if Cohen's d were not available here? Could we really do the ratio test? If we threw out Cohen's d, how do we know what percentage of sales pass the Cohen's d test if we're not using Cohen's d? That is, wouldn't the second step in the DPA also fall? It wouldn't fall in the sense that it is no longer valid, because I think what you could do in answer to Judge Bryson's question is, okay, you could develop a different methodology for determining whether or not the groups are different. Commerce is using this as a consistent mathematical methodology. They'll apply it every time they run the differential pricing because, again, we can't look at groups of sales and just say, well, these things look like they're pretty close or there's a lot of overlap. The things we would be comparing at step two of a modified DPA couldn't be the number of sales that pass Cohen's d if Cohen's d is not being used at step one. Right. It could be sales that pass a different test. Whatever that test were to be. That's correct. I just want to understand what Commerce thinks is the scope of what it can do within the reason that would constitute reasonableness. If we were to conclude that the way you're using Cohen's d, which is a statistical test at the end of the day, is not statistically sound, does Commerce think that its discretion allows it to use a statistical test in a non-statistically sound manner? Or would you agree that would be unreasonable? Well, I think if the court were to hold that, then yes. Again, I think the thing here, though, is that Commerce has explained that the Cohen's d is being used for this specific purpose. And, you know, despite all the discussion about the academic literature, there is nothing there on this record that says that it's inappropriate to use Cohen in this manner. There's nothing that says that explicitly that it's okay. But what we're saying is the academic literature is silent. And so Commerce's explanation as to how and why it's using this particular equation crosses that reasonableness threshold. If we are persuaded that on this record what you're doing is not statistically sound, can we nonetheless say, well, you have authority to do something that's not statistically sound? Well, yes, Your Honor. I would say that... Why are you asking us to do that if we were to reach that conclusion? If you were to reach that conclusion, yes, Your Honor. Because the equation, like I said, it's being used for this singular purpose. And every time we've gone through every different hypothetical that's presented, it works out, right? There hasn't been a single thing that's been presented. Even the hypothetical that appellants presented in their brief doesn't show that the Cohen's d is not robust enough to deal with these issues and determine whether or not there's a difference between the two groups. So, you know, again, I understand it might sound strange to say that, but really what, like I said, what Commerce is using this particular equation for is the singular purpose. Counsel, I think you need to yield to Mr. Garrett. Yes, Your Honor. Thank you. Good morning. May it please the Court, I am Jeff Garish of Chagrin Associates on behalf of Wells Pontubular. The core issue here is whether Commerce's methodology is reasonable, whether it provides a reasonable means of effectuating the statutory text and purpose of determining whether there are significant price differences among customers, regions, or time periods in order to unmask targeted dumping. The methodology that Commerce uses here is an adaptation or variation of the Cohen's d test. They're not using the Cohen's d test as Professor Cohen and others have used it for purposes of conducting behavioral research experiments. Commerce at some point concluded that they would attach the Cohen's d name to their calculations, but I think it's created a lot of confusion and, of course, a lot of litigation. What Commerce has actually done is to adapt the test for its own purposes as part of an integrated methodology for determining whether targeted dumping is occurring. The test they're applying is not unlike the other tests they've used for targeted dumping in that it uses differences in average prices and standard deviations. Commerce has also adopted this test because it provides effect size thresholds as operational definitions derived from real world observations that are easy to understand and that are not based on statistical analysis. So what has happened here is that Commerce has decided in its expert judgment to use the Cohen's d formula and the most conservative effect size threshold corresponding to a grossly perceptible difference to identify whether price differences are significant for its own purposes in determining whether targeted dumping is occurring. In applying the test in this way, Commerce is not beholden to conform to what the statistical literature says about the correct application for conducting behavioral science experiments. As you suggested, it seems to me Commerce decided to call it a version of Cohen's d, defends its use of it based on Cohen and the work that Cohen did, and therefore it seems only fair that we would assess how what Commerce is doing compares to what the statistical literature would say you should do with Cohen's d. Isn't that a fair analysis of where we are? Well, I do think it has created confusion that the fact that they've used the Cohen's d name, and I think they've done that for certain reasons. But, you know, again, it's a test that's like the tests they've used in the past. It uses standard deviations in price, average price differences, and, you know, it's akin to the rules of thumb they have used in other contexts. And, of course, this Court has upheld those rules of thumb in the ratio test, the 33% and 66% rules of thumb, and then the 25% rule of thumb that Commerce uses is for the meaningful difference analysis. And this Court has upheld all those rules of thumb. And these are all reasonable cutoffs for Commerce to use in that they've developed using their expertise to do what the statute directs them to do, which is to address. Do you have a final summary thought? Well, again, I don't think Commerce is beholden to conform to what the statistical literature says on this, and the statistical literature cannot be the barometer for what is reasonable under the statute. What Commerce has done under the statute is reasonable, and I think this Court should uphold it. Thank you, Counselor. Mr. Winton has some rebuttal time. We'll give you two minutes. Good to see you again. In our briefs to the Court below, and we referenced this in our reply brief at page 18, we looked at the effect of exchange rates on the DPA analysis, and exchange rates are not under our control. In fact, academic literature says that exchange rates are random. We did the calculations, and it turns out that just the variations in exchange rates would lead to a finding that we passed Cohen's d, that we had more than 66% over the ratio test, and that difference alone. What the evidence shows and the academic literature shows, when the requirements set forth by Professor Cohen are not satisfied, the Cohen's d test gives you a number that's random. It could be high. It could be low. It's not a meaningful number. You're counting up random numbers and saying, well, 60, you know, in this case, you know, I rolled the dice, and I get, you know, this many, and one-third of the value when I rolled the dice was below, so it's less than one-third passed, so we use average-to-average, or more than two-thirds passed, so I use average-to-transaction. But counting up, you're counting up random results, and that's what makes this unreasonable. Commerce hasn't shown that the results of the DPA, excuse me, of Cohen's d, when the conditions set forth by Professor Cohen are not satisfied, he hasn't shown that those are meaningful results, and they haven't shown that 0.8 is a meaningful comparison number when the criteria required by Professor Cohen are not satisfied. And it's important to note, Professor Cohen didn't say, I have one effect size measure. He actually had eight. He said, for use in different circumstances, the d statistic is used in conjunction with a statistical test called a t-test, which requires normal distribution, equal variance, sufficient number of data points. Other situations, he has other measures of effect size. There's nothing that says d is the end-all and be-all of effect size, and the only number that you can use. And in this case, the evidence shows that the results are random. They're affected simply random variations can lead to positive results and can affect the dumping calculation, and that makes it unreasonable. Thank you very much. Thank you to both counsel. The case is submitted, and I can assure you we will not be rolling dice. --- Source: Frix Law Library, https://www.frixlaw.com/law-library/documents/argument%3A97596. Public record. Not legal advice.