> Are you prepared to do some very very fancy statistics?
I'd extend this with "... while understanding what you're doing?"
I've seen it so many times already, someone does some A/B-test and then presents a very fancy looking slide-deck with all kinds of crazy-looking math. But if you start to ask questions, it's all very obvious that they didn't really understood what they were doing and that very often it doesn't really matter to them in the first place; it's all about reaching a decision using some pseudo-scienty method that nobody dares to question because 'data' and 'science', without having to take responsibility.
I think "Be brutally honest about you many assumptions and caveats" at least implies that.
I mean, in an informal setting there's room for an honest person to say "well I did some math and I don't really get it but I think it says...," but I think this article is addressed to software engineers and scientists. Someone representing themself as an engineer or scientists has a professional ethical responsibility to some sort of... I dunno, epistemic honesty, the knowledge of what their expertise covers, and communicating their limitations to laymen.
The person with the A/B test in your example is either a liar because they are misrepresenting what their tool says, or they are a liar because they are misrepresenting their ability to tell you what it says, but either way they are a liar.
I'd extend this with "... while understanding what you're doing?"
I've seen it so many times already, someone does some A/B-test and then presents a very fancy looking slide-deck with all kinds of crazy-looking math. But if you start to ask questions, it's all very obvious that they didn't really understood what they were doing and that very often it doesn't really matter to them in the first place; it's all about reaching a decision using some pseudo-scienty method that nobody dares to question because 'data' and 'science', without having to take responsibility.