This government has finally issued its 1st government rule on electronic reverse auctions which was followed in late August 2,024, just.
Last month my proposed rule generally barring reverse auctions.
Or complex construction services. This webinar will discuss both rules, both the final rule and the proposed rule and their possible impact in public procurement across the United States around the world when it's extend the special welcome to our colleagues from the National Association of.
State procurement officials. They're having their annual conference.
Indianapolis, and we'll be having a session as part of our State and local procurement seminar. We'll be having a session where we invite.
The the folks from Nasbo to join us to talk about.
The implementation of reverse auctions at the State level in the United States.
As well.
The recording the materials from this program are available, public procurement will have questions and answers. QA. During the course of the session. Please feel free to to post your questions in the QA. Just remind us. Speaker statements are in their personal capacities.
And the background materials are on the the program page for this program at public Procurementcom we have a special program page set up with resources that will be referring to over the course of today's session.
Today is an important day in our history. So a remembrance for September 11.th
If for those of you we have a lot of guests around the world, and if you're visiting the United States, we strongly encourage you to visit the September 11.th
This is an aside. One of our guests was actually.
It it the work that he did David did helps strengthen the Pentagon Building, so he helped save lives on September 11, th so.
Still part of our history.
So today's webinar has over 300 from 51 countries. Just. And we're really so gratified to have so many people participate from around the world, and from 38 States. So for from over 2 thirds of our States.
Here in the United States. We're really glad to have you all join us.
So going through our panels, Albano and John, Luigi and I were. We're in South Africa last week talking about their new procurement lot. It's always a delight to have them.
He's a senior manager of concept. The Italian governments.
Centralized purchasing agency.
He wrote his doctoral dissertation on auction theory, help, design and wide public auction mechanism on behalf of the Italian Government. He'll be teaching a course on auction theory at Louise, a leading university in Rome. So we react and geology. You'll see it over. The course of the session has an extremely wealth of practical and importantly theoretical understanding.
Reverse auctions which you'll be adding.
David Dragkins, the chair of the Procurement Roundtables, which is a group of leaders from the Federal Procurement community. He was previously the senior Procurement Executive for the General Services Administration, and that role. He was a member of the Far Council which writes, fellow procurement rules.
And participate in early assessments of a possible far rule on reverse options. He'll be talking about that history and the the process, the mechanical process for getting the rules prepared. I'm the moderator, my professor at Gw, as I said, I've been speaking and writing on reverse auctions. In fact, one of the 1st things I did as a professor after I became a professor, 2,002 was hold a colloquium in 2,002, 2,003, and that timeframe.
About electronic reverse options. David was part of that. So it's it's exciting for me to see. After 20 years this rule finally published. During that period I worked for about 10 years on the UN Model Procurement Law, the unsold model procurement Law, which includes provisions on reverse auctions.
So we're gonna talk about the rule and reverse auctions. After we discuss some of.
Understanding how reverse auction works. And then, when we close out, when we talking about the proposed rule and reverse auctions and construction services.
In essence the proposed rule is pending.
Carves out.
Reverse options for construction services, and, as in other countries, it says, the reverse options.
Shouldn't be used for construction services.
So for centuries governments have asked bidders to submit bids against government and make awards the lowest price qualify better. This is the standard way that it's been run, really, since Roman times or earlier, that how you would run run a normal procurement is you're seeking the lowest price.
Reverse options, electronic reverse options. So the next step in that process, it's an iterative, iterative bidding process, so that instead of having just one bid.
You have multiple bids over time where the bidders are competing against each other.
To understand this, we use this graph. And I wanted to go through just a few terms here. So you understand, or how reverse option works.
Cause we found in our in our teaching that.
This guy the this illustrations in a particularly good way of understanding some of the basic terms.
That you'll hear David and John Luigi refer to as we go through in terms of understanding. So this is the history of a reverse auction, and an important 1st thing to understand is this level of detail would not necessarily be available to bidders during the course of auction. This might be where the Government sees, but not necessarily what the bidders would see. And we'll be talking about that issue.
How much access.
Better should have 2 pricing information as we're going through today's session.
The historic price is what is the price, and this is very common, and, for instance, the Oecd.
Wrote a famous report about a coal auction in Mexico and Mexico is a colleague of ice of ours. Help put us put this together. Max Kaiser had experienced historic price above $100 a ton, and they could not, through any other competitive means.
Force the coal.
Transportation companies that are providing the.
To get that price below $100 a ton. So they ran a reverse auction like this one. And they and here you can see, the historic price is marked here in purple, and then you see the initial bids here are bids above that historic price, because that's what the contractors are used to. That's what the vendors are used to.
Being staying above the structure. In essence they're reaffirming their their price lock above the historic price.
One quick question is, when it comes, what about that guy who is when there's a sudden drop below the historic price? What about that guy that who seems to be like he didn't get the memo that we've now we've dropped below the historic price.
How to explain a couple places in this history where bidders are bidding. It seems like they're bidding, you know, like they're drunk or something that they're they're off, course.
Sure.
The answer may be that they didn't see the information during the course of the of the presumption. It will be talking about a model for an example used.
In the Us. Government already.
It was out a governing rule, but a model which is called a lead lag model, where, when you bid, all you know is whether or not you are the lead or the lowest. Bid.
These? Have.
If you're if you're not, you have to keep bidding in order to catch up. And suddenly the the program, the lead.
So that lead approach gives much less pricing information.
To the to the bidders, and we'll talk about the the benefits of a more careful approach to price disclosure during the course of auction here. Now you see the pattern here they drop below the historic price.
Reserve price is the price.
Above which the government will not make a purchase, and the government discloses that ahead of time and says, if you don't go below this price. We're not going to make your contract.
So they have an incentive then to drop. But you see, there's still dangling above that.
Darren, go ahead!
At reserve price, which itself is an interesting question, why would market participants.
Bid at a level.
Would do that.
But they do it. There's no way they're gonna win.
They! They!
In the sample is, if you, if you allow me, there is a very, I would say, well established explanation and dynamic option. Why, this may happen.
So if you're now a reserve price then, but at the same time you also announce a what you you put there a schedule. So a bider doesn't want to submit a serious bid until a moment where it comes very close to the, to the deadline. So when you approach it, you see that all of a sudden there is a drop in prices.
Because obviously you don't want to close your cars too early, because you don't want to see your opponent.
To observe, they, in fact, very aggressive. So the temptation is basically to disclose your cards as close as possible to the schedule.
Dead closing. But in doing so the danger is basically transforming a dynamic auction and then standard one round auction because he's just a bit. One serious bit very close to the schedule close in. You may end up just having the time of submitting one bit. So, in fact, what normally.
Reverse auction rules for C is the possibility that a sudden drop in price you extend.
You stretch the close in.
The closing moment further, in order to allow for more bids, so at the same time.
You. This has to be announced. Obviously, otherwise, it becomes a temptation for bidder to send just one bid a second before the closing hour. But at the same time you don't want to abuse this, because otherwise the bidders anticipate you're gonna you're gonna have more and more, you know.
Closing moment. They're gonna.
Withdraw their fears, and stay in the grass, behave like.
But.
You know, standard auction would would say that they behave like a sniper right? So there is a tension between for a little bit of flexibility. But too much flexibility is bad, because you may provide the perverse incentives. So this is a very nice picture, because it contains sort of already 80% of the problems of the high.
To put the right incentives in terms of information and other auction rules that make an auction work or just fail.
Okay, thank you for this, Chris. By the way.
No, I think it's very important, and and we'll be returning this this diagram later on. But just for the audience to understand what John Leech is talking about. So though it's it's it's the the labeling see the word historic on the right hand side that's covering up the key. So I just want to explain it. The the scheduled closing was the black vertical line that was the original scheduled closing.
And the the agency decided to extend the time.
4 into a to an actual closing. You see a frenzy of bidding just before the schedule time, and by the rules of the reverse auction they could extend the extended time, and then you get a huge frenzy of bidding down there at the at the bottom, and the one of the things we're discussing is.
At the end that last winning bid.
Is that very low bid? Is it below the marginal cost of production? Because if it is below the marginal cost of production, there's significant performance risk. The the contractor will just default. Because why would it perform the contract?
If it's at below the marginal cost of production. So that's 1 of the issues we'll be discussing about. That's called the Winners.
So over the last 20 years, various procurement regimes around the world have embraced electronic, the EU directives, which are the overarching directives that the the Member States, like Italy, have to follow. They already had reverse auction provisions. The Defense Logistics Agency had its own reverse auction. We'll be talking about that. Colorado has its own very interesting reverse auction. Well, one of.
Students. Justin Duffy will be writing his paper in our class about various States implementations of reverse.
And Brazil was really a leader in the 19 nineties. In reverse auction I want to emphasize because.
Brazil.
Is now at this point, according to studies by multilateral banks.
In Brazil reverse auctions are the predominant means.
Never done.
Of.
Procurement. That's predominant method of procurement in Brazil.
We'll be talking about whether that's a good idea or bad idea. But the reality is that they're used pretty aggressively around the world.
So a Gao report in 2,013 found that reverse auctions.
So sir.
Can generate substantial savings. They can. They found 12% savings, 14% savings, really significant savings where you were using reverse options. Now for both of you, for for Dave, and for John Luigi, that one of the one of the concerns here is that this is too seductive, that reverse auctions are touted as well. An easy way to save 1215.
Is, do you want to speak to that that question of.
Sort of the opiate of reverse auctions.
And and and therein lies the problem with reverse auctions. It is an approach to procurement.
That seems attractive.
And is then immediately misused. I will get to at some point this morning, but, you know reverse auctions were initially an industry initiative.
Used to buy commodities coalition oil spot markets and to drive the price down. It was not used by industry initially to buy anything more complicated, like computers or phones, or whatever.
And.
Unfortunately, at least in the Us. Federal Government. When somebody uses a strategy, a methodology for running a procurement.
Hmm.
And it's successful. All of a sudden. Everybody thinks you gotta use that methodology for everything without an analysis as to one what savings is really available, and 2.
How do you make sure that you control things like quality and and performance? Histories.
Of of the sellers. And I know we're gonna talk about that later. And it's those things aren't insurmountable.
But it poses a challenge.
To the system. If you're one of the people like I was at Gsa. Responsible for making sure the system and that are.
Our end. Item, users in the government got the goods or services they want to buy. And, by the way, some people have tried to use.
For example, reverse auctions to buy services which.
In my mind is still.
Nuts.
Think that they are.
So. So we have. Just so everybody has this background as well that we have. We. We have registrants today, both from the State of Georgia and the nation of Georgia and the the nation of Georgia, actually applied reverse auctions to everything in their system. And then what happened? Following up on on Dave's observation.
It was an interesting response. And transparency International did a study. This is all available on the web.
The, the.
A individual agencies like the Defense Ministry pulled out of the system because it was reverse auction system, who is forcing everything through this channel, reverse auctions, and, as Dave points out, reverse, auctions are inappropriate for certain, for certain types of purchases.
Sure.
John, would you would be fair to say that.
I was just something about saving. So the the temptation of, you know, kind of her behavior like there is a leading organization, and all of a sudden all of it sort of.
Makes everything more difficult. You're unable to separate, you know, with the good practice from the bad practice. So as soon as you start having problems. Then it's the revert that will guess reverse. So everyone steps back and you stop using it. So the point is, you know, my Latin ancestors would have said, you use them with a little bit of wisdom.
So make it sure. Make sure. Did you understand why it does work the conditions or prerequisite.
For heaven, a good man! They was pointing out something that it's actually not a global evidence. What is what are the most successful.
Experiences of reverse. It may be extremely sophisticated, and I let me mention.
There. There is a family of Us. Academics who have contributed to the use by governments, and with the fcc auctions going to electricity auction market. Now.
Wind generation, and Scotland, and going to the consider the most interesting examples, which are threeg auctions in the U. And.
Co. 2 permits auction in Europe. And this is actually the one particular Docia contributed personally to. And what is the common thread? Exactly what they were saying. We're talking about CO. 2 oil. Let's not forget that the the Department and Interior has been running for decades.
For allocated rights for extraction rights in the Gulf of Mexico. So the Us. Is a massive.
Evidence? Why oil.
Co. 2. Electricity. That's exactly it are extremely homogeneous standardized items for which there is a market, and then you can leverage that and trigger, and sort of, you know.
Competition by using appropriate rules. So this temptation of sort of spillover kind of virus.
Going from dead to other words.
Right.
And.
So makes a life of everybody so difficult, because you don't understand why it doesn't work elsewhere. So you have to understand why it does work in particular. Before taking this to another word.
And
So that's very important.
And let me modify my comments earlier. Just a little bit.
There's a difference between using reverse auction as a pricing tool, as part of an overall procurement.
And using it as the only tool.
For making a purchase.
Yeah.
And and and I absolutely believe that for buying commodities.
The Co. 2. The oil coal reverse auctions are an excellent way.
Of of making those purchases and reducing the price the Government ultimately pays for those things. And so really, it's something that has to be a addressed as part of the acquisition strategy.
When at least in the Federal Government.
You. You begin looking at how you're going to buy something.
So I just wanted to clarify for the audience that. So Dave's talking about policy concerns, about economic concerns, economic theory, they can inform how you frame a reverse auction, and and and the audience members are going to be cripping their heads and say, Wait a minute. We thought this webinar was about a rule. But here you're talking about all these theoretical issues that should come into a reverse auction.
That's the point of the webinar that there are very serious economic and policy concerns that should inform the rule, because the rule is what's going to guide the contracting officials who are floating a reverse auction. The rule has to reflect these economic and policy concerns, and as we'll see in a few minutes, unfortunately, doesn't.
And.
To Michael, who's asked the question, What is a reverse auction.
I encourage you to read the final rule. The final far rule which has in it a description of what a reverse auction is from the perspective of the Federal Government.
Which is not the only perspective, because clearly many of our colleagues, state and local government, and from foreign countries.
Have a slightly different. Take.
On run of reverse auction is, and and one other point I would make, and I failed to make. To begin with, is in the Us. Government up until 1,994.
It was prohibited to have an auction of any.
It was.
Specifically prohibited by regulation.
In 1,994. As a result of the Federal acquisition streamlining act of 1,994. We eliminated the prohibition against auctions.
Dave. They've raised it when, in his reference to the definition of auctions, raised a a subtle but important legal point. There's a definition of reverse auctions in the final rule became effective. On August 29, th on the same day a proposed rule regarding construction services we'll be talking about was published, and it has a slightly different.
Definition of reverse auctions, and the question is, which will prevail.
Hopefully the latter one that in turn leads to a broader question, which is.
As we'll be seeing in this discussion. There are major problems.
With the final rule that covers all reverse auctions.
Could, the.
Pending rule on construction services be used as a vehicle for fixing some of those problems cause there's definitely an overlap.
Between the 2 rules. I just want to quickly talk about this 2,018 report.
And what we we talked about this in our prep. The the what lies behind this is that there was so a 3.rd
In the 3rd of the auctions you only had one bidder, or you had multiple bidders, only giving one bid. What lies behind this is, and is that the these reverse auctions which are really being done at the margins of the of the procurement system? And we're not covered by an overarching rule. So the John Luigi understands that these auctions by and large, were ones that were specially constructed, so you would have 2 stages.
Yeah.
The 1st stage would be, let's say, a fight between apple and dell going down, down, and then and then Dell would be the low one and say, Hey, I won the reverse option. Right? I get to sell the laptops. Answer in stage 2.
The agency through the reverse auction provider, the agency would reserve the right to choose the apple instead, because they liked it better. So after a while Dell would just.
And I'm using I'm using. Apple's is.
Conjectural, hypothetical. They're not.
Really campus.
Sure.
But don't just wouldn't show up for the future auctions. That's why you had this weird breakdown, because the price.
Wasn't really tied to the ultimate award, and I'm not sure that's sustainable under the certainly under the definition in their proposed rule. Do you have a comment on that approach.
Well, this is the idea of looking at competition today is very attractive, very myopic, you know the the. So what you're pointing at is a sort of I would say, an abnormal phenomenon, but there is even more, even if you went with something that looks like attractive. You have to keep it in the future.
Will be more an auction. So the auction design should be, can you spy by this idea? That might be a compromise between competition today and safeguarding competition tomorrow. So you want to keep the supply base or.
Car numbers will call them the same participants.
Kind of Akat, and populated over time.
So the reason dynamic dimension, which is not the dynamics within a single auction, but is the over time. You need to keep this in mind, and especially when you.
Use auction in markets where they are very low margins, etc. Today, which sounds very nice, looks very nice. And then you're gonna kill.
Participants tomorrow. And that's.
It's something that publicment practitioners are not used to consider what happens tomorrow. The safeguard and a supply base is fundamentally markets. You know, I'm from my experience at borrowing from Italian experience, especially in healthcare procurement markets. There's a temptation of riven, you know, profits out of the pockets of the big guys which are multinational, global, nasty guys.
But at the same guys, these are the guys, the only guys on the market. So if you, for instance, you give the whole.
Contract the sizable to one guy and all others. Go away. Tomorrow will be observing only one guy bidden, which is exactly what you pointed and emphasized in that slide. So again, the compromise for option rules for competition today. But at the same time off your rules to make sure that I'm not killing potential competition to.
Right.
And I'm really upset. I'm sorry I'm real obsessed with this, because procurement is a dynamic game. It happens over time. So we have to look.
Ahead of us. Not only what happens today.
A. A and at the Us. Federal level.
This Administration. The Biden Administration has made a con policy decision.
To drive greater participation by businesses in the Federal marketplace.
And.
Using a technique, a procurement technique which would reduce the participation.
By businesses is inconsistent with.
For the gover.
That policy objective which is again to bring in a more businesses to do business with the Us. Federal Government.
To to, just to follow up on both these points that were made. One is as general pointing out, this is a key aspect. Pro markets is a repetitive markets. They're repeat players over and over again. And that's an important part of understanding how the rules have to function.
The second thing is, our opportunity costs.
If Dell leaves the marketplace and doesn't come back, that's an opportunity cost to the Government. That's a that's an erosion of its supplier base.
Hmm.
Sure.
And government is very bad at assessing opportunity cost. And that's a that's a problem they have to be recognized in in the rulemaking process.
And and and one other small point to bring up.
Our contracting officers.
Who typically get a request for procurement.
At.
Aren't.
Aren't national policy.
Today.
Diviners they don't go out to their concern is, how do I satisfy the purchase that's on my desk today and and get and get my customer what he or she wants.
A. And they are not.
Likely, and this is not the meaning to them. It's the nature of the business.
It's their their focus isn't on that bigger government strategy. As a general rule.
Just quickly go through this history here so that we can, as we go. As we go into the rule, we understand the the history behind it. So we talked about the spread of electronic in 2,004. Actually, Ralph Nash wrote an article in 2,004 calling for this rule. So 20 years we've been waiting 20 years from that rule, from the excuse me from from his article 2,009, Dla. Publishes its rule on public on reverse AR since we've.
Law, published in 2,011, has reverse auction. Provision. Ga. Published 2 reports, one in 2,013 and one in 2,018, the the 2,013 report noted that more guidance was needed on reverse auctions. Congress. This is important. Congress in 2,014 call for reverse auction, guidance.
That call for guns, for rulemaking in 2,014 wasn't fulfilled until 2,024, and, in fact, the Gao report in 2,018. The report in 2,018 drew a link between that 2014 statutory requirement for rule and the rulemaking. We're in right now the Ovp issue guidance in 2015 is linked on the program pages an important compliment.
To the final rule. You really need to take a look at that Ppp guidance as well, and we'll be referring to it 2,018 Ga. Report, 2,021 Congress called for rule on construction services, walling them off.
There was that nothing happened on that until last month, August 29.th The proposal was finally published. So 3 years, 3 years wait for that proposed rule.
The general. The general proposed rule was published in 21, and but the final rule also waited for 3 years. We didn't get a final rule, the one that we're discussing today that covers in general until 2024. The same day the final rule was published. The proposed rule regarding construction services was was published on August 29.th
So, John, would you have reverse options been adopted more broadly in public procurement around the world?
I mean they there's been. It's a kind of as always a fashion waves, I mean, I'm personally advised.
The European Commission's in observing a wave like this one.
I would say, around 1015 years ago in Eastern Europe, and this was driven mainly. This is going to be a very important point. I know you're gonna you're gonna pick this later.
Chris, it's about the the at, you know, tech. When technology becomes very efficient, there is a temptation.
Since, you know, platforms have become sophisticated. Why don't we use electronic? Because we have a platform. So you see, it's not the thinking or what data is is was pointing out the idea of market features. But it's technology. It's so easy. It's so cheap to have a platform running for 2 h we can run very fancy reverse option.
When I use it.
In that scenario. Temptation. Technology is instrumental to implement, but should be the fact that we have technology that we're going to use the reverse option everywhere, and unfortunately this has been the case in many countries, I would say, mostly in Eastern Europe, and if fiber from I/O experience.
I have to say that when the other way around. So I was gonna be a little bit of self criticism actually.
We start, I mean we. And now we're in the Italian government, centralized procurement organizations.
Almost 20 years ago they saw they use it in some reverse, but at that time.
The sophisticate. You know, the technology was not good enough. So, in fact.
What they said to my colleagues at that time set aside the possibility because they were not helped by technology. So where you're basically making the decision.
Based on the real rationale, the good one. But they ended up not using it, and the last 8 years, which corresponds to my experience here.
And we've we've tried basically to get more and more sophisticated contracts. Actually, we had not to use electronic auction which is completely compatible with what Dave was saying, as you move, to what concept in which what is really important at any of the bidders, so on and so forth. And you want to really get very sophisticated solution.
So.
Reverse options. I know the way to go. So I see other the other way around. Okay, very short summary. But I would be happy to provide more.
To those who are interested.
This.
Thanks for much yelling, and this will be an ongoing discussion, I can assure you.
The so just quickly. And what the final rule says. The final rule anticipates the coordination with the opp memorandum from the Austin Federal procurement, policy, memorandum from 2,015.
Dave, if if you are a contracting official, you have a rule that's that's in 4 Part 17 now, and you have app memorandum. How does thepp memorandum come into your thinking when you put in the other reverse option.
Well, 1st of all, you assume it does at all.
Yeah.
The the people in the field.
Generally speaking.
Do not consult.
On a regular basis, if at all.
Osp memorandums. Sometimes they don't even review the memorandums issued within the department that they work.
They look to the far.
They look to the far supplements from their agencies.
If they're far supplement. The agency has some sort of.
Guidance book like.
Like Dod does, and so do other agencies. They might look there.
But, generally speaking, in a Gs. 11 2.
That's our contracting officers. Journeyman levels at Gs, 12.
They are.
They every day is a race to to get something awarded so they can start work on the next thing that's not like they have lulls in the average contracting officers, so they don't do.
The kind of research that a professor would do that a lawyer advising one of his clients would do.
Yeah, some of them may not even know that even it issued the memorandum, although, there are plenty of ways to find it on the web.
So.
Actually the ofpp memorandum is interesting for the senior procurement executives at the various agencies.
But I suspect, if you ask the average contracting officer.
They wouldn't know what you're talking about.
Again to click through more of the the final rule. Yeah, the during the proposed rules, the proposal suggested, the data be wiped out at the end of reverse auction community was very unhappy with that, and due to several comments, saying, Look, we need to have access to the auction data at the end of a reverse auction and needs to be retained in case there's an investigation, the rule has changed so that the contracting officer will frame out.
How the the the data will be preserved for potential enforcement purposes.
We're gonna be talking about disclosure of lowest prices. There's very limited guidance on when to use reverse options in the final role. Very limited guidance.
On how to structure, or how to conduct a reverse auction.
Importantly, although the and reverse auction service providers. Just to note this can be either a private company or can be another agency. The the private companies pushed in the comments to be allowed to push out bidders that aren't complying with the private service providers rules.
And they and the firewall says, no, that's a decision for the contracting officer to make, not for the service provider to make the reverse options are integrated in other methods, so they can be used, as Dave suggested. As a, for example, the tail end of an Id, or what's known internationally as a framework contracting process.
The the the rule talks about. We'll talk about some gaps in this.
Right.
The relationship between the reverse and service. Providers and bidders protect you, for example, bidden proposal information, and the and the final rules, as if there's only one bidder, it's up to the contracting official whether or not to accept that single bid, or just to strike the reverse option and start over either using a different method potentially.
Somebody is sorry.
And and I know how important to you data is, Chris, and it is to a lot of different people. But keep in mind.
That if your contract writing system.
If you're the buying agency, doesn't capture that data.
And automatically reported.
That's just another step.
For a contracting officer to take. Generally speaking, they won't do it until after the procurement's completed, and often those kinds of things that wait till after the procurement is completed. Wait.
Wherever.
If ever, if ever.
The. So the so they're the DNA rule from 2,009 calls for an explain is presumptuably a presumptive use of reverse options much like there's a presumptive look to environmentally sustainable products now, under the recent final rule on on that side. So the deal rule in 2,009 calls for that. Dave.
The the contracting official has to explain why she hasn't used her option. Why hasn't the Federal Government do you think they've moved more broadly to and more quickly to embrace, reverse. Why, we waited 20 years.
Well.
Sure, so first.st
1st of all, you had, like me, who in the my policy role, thought we should never write a rule on reverse auctions, because the existing rules I thought at the time were sufficient to govern. How reverse auction is used. It didn't reverse auctions don't fundamentally change.
Any of the procurement rules that we had before. It doesn't change how you structure the pricing. It doesn't change any of that.
But now.
It. Congress has decided that this is a tool that should be used, and there are certain parts of the.
Marketplace that would like this tool to be used, and
And so I think you'll see more use of it as long as we find a way to avoid any horrible stories.
About terrible things that happened cause someone took a reverse auction.
Tool and used it to buy surgical equipment.
Well, not the equipment.
A Surgeons.
We actually had we. I have to tell you that we had a student one year who's buying surgical services for the reverse auctions, and the other literally in the class. The other students all moved away from her.
I. And and, by the way, it's going to happen because some young innovative person is going to think I can use this tool in this particular, and a very successful way, and gave it a try.
And discovered it didn't work. The problem is, as you know, our system is unforgiving.
It should accept experiments and subsequent failures as a way to learn in advance. It doesn't.
So in terms of the guidance on when, to use a reverse option under the final reverse, auctions can be used when 3 criteria met. There's a competitive marketplace. Multiple vendors can satisfy the agency's requirements and the nature of the acquisition lends itself to it or bidding. For example, where they're clearly defined specifications.
The the we did actually ran through this in our in one of our classes, already.
This could conceivably cover automobiles right.
And that's no one thinks auctions, for we'd all be driving. You guys. No one thinks reverse auctions for automobiles is a good idea. And yet these very loose criteria could open the door to reverse auctions in the automobile environment. W. Do you? Generally? What's your view on these criteria?
No, I would be skeptical, because that would violate, you know. Ask him one.
So there is a competitive market out there. But the this is a market with very healthy solutions.
So.
Alas! You want to standardize the more they want to buy. And unless you want to think of a government.
Purchases advantage as if we were in Soviet Union in the 1970 S. You fixed the bulky 3 torn car with 4 windows, and that's it. And then you go for price.
I would like to think about the possibility of considering, let alone all post purchase services.
And so I would never consider to be honest. I would never, ever advise any country, obviously including my own country, to use.
Reverse the option for.
Vehicles, or, yeah, beat them. You know all the mobility, like whatever.
So these exactly. The market exists, but.
Very ethereal genius solution. It's actually.
The more I mean, we're increasing the number solution. It's a market becoming more and more inich market.
Where the tens of different solutions I don't want to think of is a procurement that actually pushes towards a standardization and solution like in Co. 2, which is one thing, it's molecule, right? Molecule Co. 2 is the same. Believe it or not, in Italy, in us and Canada and South Africa. It's always the same. What quantities vary, but it's still the same molecule.
But but our system, at least in the United States, for a very long time used to take the position that the government was endem user was only entitled to the absolute, minimumly responsive.
Item or service.
And so the idea of buying.
A standard gray automobile with, you know, 4 wheels and an engine. I mean, we didn't buy. We wouldn't buy them if it had radios in it back in the day. Those cars we.
Seriously. I I mean, I'm not talking about fancy radios. I'm talking about FM and radios or air conditioning, and and and so.
Our our approach to acquisition, and delivering capability to our.
Our customers. Our internal customers has changed dramatically, and it was principally driven by the changes we made in 1,994 as part of where we decided that our end item users deserved more than a pencil.
They needed computers.
Yeah.
But the court, the course is still pending. If it does this, do these criteria, which is what's in the final rule? Do these criteria. Tell the average contracting officer. No, don't use reverse options for a car.
I don't see that in the rule. I hope.
Hmm.
That the.
The contracting officer who's developing an acquisition strategy to buy a car which, by the way, let's you should take that off the list.
Only Gsa. In the Us. Federal Government buys cars.
Except for specialty cars overseas.
But pick something else.
And.
And and and I think the answer is, as you develop your acquisition strategy, you need to answer the important questions of.
What is it you're trying to acquire?
What's available in the marketplace.
Sure, exactly.
And.
And how would you conduct the auction? Will you be happy at the end of the auction? If the only question that's been resolved is lowest price.
Yes.
And if it, if not, then you have to go back and adjust your thinking about whether this is a completely purchases through reverse auction, whether you do a.
Yeah.
For example, a 2 step under far Part 14, where you do a qualifications review, and then.
You do price it. So it's a.
Yeah, yeah, very sparring, Dave. If I, Chris, I'm sorry. But this is gonna it's it's it's a like a door open in many other doors.
The when you think of CO. 2, or electricity, believe it or not, even if you have a vivid imagination, it would be very hard to put on top of that contract.
Other, you know, socially related.
Closes very hard. I mean you can do it, but you have to have a very vivid imagination right.
So this is to say, then, in fact, you can still concentrate yourself on that particular homogeneous product and price. But as soon as you move away, you enter the word of products and services in which there are externalities that have like effect in some stakeholders. You want to enter in a sustainability world there. So the more you think of public hearing, acquisition strategies, a way of.
You know, pursuing socially desirable goal.
The less you're going to look at as a way of reverse option as a way of wording.
So it's in the.
Also in the development of the way the world is going, that there are some markets which you will never.
Person with normal capacity. Obviously you never think of reverse auction in those markets. So again, that reinforces theorem, let me upgrade your.
Comment as a theorem that you're never going to go outside of a quite well defined set of commodities to do so.
Do you do? You guys have? Really, you've anticipated. So the army has reverse rules. And so and it the reverse action rules spelling out how you, when you use reverse auction, anticipate exactly what you guys have said. Except for one thing, it's the 4th bullet here, procurements in which there's a well defined, supplier, base.
Why beyond the Legion? Why would the army think that that's a consideration as well, not just on the nature of the product, but looking at the actual supplier base. Why is that important.
Well, if the I mean, I guess we're pointing towards this idea, having a market which is stable over time.
It to me, I mean have to be honest with you, I mean, if I think a very stable. So with very low end exit. This is what economists would consider. And if you supply this, we have all these dynamic option actually.
This should lend itself to possible drawbacks of the competitive process. You have the same guys beating all over again in a dynamic form, and actually might be a good thing for collusive behavior. So, to be honest, the way describe it seems to be a little bit counterintuitive. So I should talk to the army guys. It seems to me a little bit counterintuitive.
To have, especially if the number of guys is limited. They're always the same you used in a dynamic format in which they learn a lot that is very easy to implement strategies. So my answer would be, I don't know. I'm not sure why they're considering this or what they might be hidden factors that I cannot.
You know.
Understand. But to me, Priya, you know, let me go back to Latin again. That 1st site these seems to be a little bit counterintuitive. To be honest.
It. It could be that if you have a fragile, supplier base where you're not really sure if there's a new supply, for instance, AI providers.
Hmm.
If you don't know which AI providers will leave the marketplace once you try to monetize it through a reverse auction. That's not a well defined, solid, well established, supplier base. You maybe want to be reluctant to use reverse auctions.
Yeah.
So no
So so on the guidance on how to design an auction. There's actually almost no guidance at all in the final rule, even though the National Defense Authorization Act, called in 2018, in 2014, called for a some aspects of the rules that would define some aspect such as the fact that inherently governmental functions wouldn't be done by a private contractor. This is a service provider, and the past performance and financial responsibility, information.
Information.
From created by a 3rd party like a reverse option service provider, be able to offer us.
Sure, is guidance on structure important? And is it enough that there's some guidance in the Ovp memorandum.
David.
Guidance is is always important. The problem is developing. It.
And and and.
Coordinating it.
Across, at least in the Federal Government level. All the various agencies.
Congress.
Rarely provides guidance in terms of what they think the rule should say, and.
And I think that's a good idea.
Because Congress.
There are very few people on the hill who actually can spell.
Acquisition, much less understand what it means. Maybe a dozen.
And I'm being generous. So.
So. Yes, there should be some guidance. The question is, how do you.
How do you make that guidance? And you, more importantly.
The guidance you want, but if it's not used, it's not helpful.
It's got to be used and useful.
And
So do, and it's Pgi.
Has a a great deal of helpful information about how to do various things, and that might be an appropriate place.
To provide some additional.
Guidance on how to use reverse auctions. Don't tell John to nagulate, I said, that he's got more than enough work to do without me, adding to his workload.
But yeah, if if you're gonna do guidance, which I think you should do, it's got to be understood that one is guidance. It's not mandatory. You're not going to have the Ig come in and say, Well, you didn't follow this guidance, so therefore you have obviously not done a good job. And that happens frequently. By the way.
So.
But the answer to your overall question is, yes, there should be some guidance, and yes, it should be made available, so that the actual contracting officers who do this work can pick it up and use it, since most of them aren't gonna use reverse auctioning.
In their day-to-day experiences. It it's not good. The average 11 officer.
In the Federal Government. They're about 36,000 of them aren't. Gonna do reverse auctions ever.
In their, in, their career.
Period.
So.
So we got an interesting comment from Sean, who is our host.
Yeah, exactly. I was going to ask you to sort of.
Yeah.
So, did.
Of.
And if and Shawn, if you could send us a link to the South African experience, we'll put it onto the program page. Very interesting comments on in terms of how dynamic options here.
Chris, do you mind if I want to? If I borrow from Sean? Because it's sort of trigger? I know there might be other details which are not in his comment, but even an auction open for many days like a week. And that's it. It's not enough. I mean, the auction design needs more than that. So I'm pretty sure there are more additional, finer details.
Because otherwise we go back to exactly the 1st picture that Chris was showing us, that if you have a deadline, then all of a sudden, you see a lot of frenzy going on. You you want to expand that deadline. So having a fixed deadline is not a a more circumstances, is not the way to go, because you're going to trigger the temptation.
That what we call the strategy and C until the last second, and then, coupled with the technological uncertainty that when you send you a signal your bid. You're not sure that the platform is going to receive the bin in real time you may be and cut off from the competition because of the lag.
Bar.
Right. And so this is really.
Important. And, secondly, I would like to add, you know, David, saying usable usable, and used.
But at the same time the guideline needs to be sufficiently high, because auction design is a flexible thing. So you want to adopt the auction design depending on the market commodity, you know. So.
Basically, David said Rook, I need guidance, but the guy is not going to be the tail guidance. So if you're thinking of a guidance which you can follow an algorithm. Do this and that. You're not going to find anywhere in the world a guidance like this. Right? So you need more than that, you need the sort of the logical approach on when how to use it, but then you will have.
Come up with additional features of the doctoral design based on your experience, know-how, because you.
Gathered. You have colleagues who've done this in the past. So it's a pretty challenging work. And as they were saying.
It's not gonna be an open door to everybody.
They're going to be a subset of.
Procurement officers in a position to do that. It's like a surgeon. The more to do it the better they become if they never do it don't ever do it right because it's gonna make a disaster.
Out, of.
Just so, the audience understands. So John Luigi is a Phd. Very highly trained, very accomplished economist, lodged inside the Italian version of Gsa. And that gives you an idea of the sophistication that will have to be brought to when, in fact, to use a reverse option and how to structure. There's some more guidance from the best of the Defense Logistics Agency.
In its role on how to structure auctions. Again, most of this guidance just doesn't appear in the funnel in the government, wide, final rule at all, which is frankly a little terrifying.
The winner's course is a common issue in reverse auctions is as you're driving down on the prices at the end, because things are happening so rapidly.
Bidders in order to win our pricing below their marginal cost of production, and that's called unrealistically low pricing in the Us. System, or abnormally low bids around the world. Is this a problem, Luke? Is this a problem that this final rule doesn't address it.
Should should American.
Procurement personnel be concerned about this.
It's I'm sorry I don't want exaggerates. It's gigantic problem. It's been tested and observed, and you know I would say thousands of lab experiments, reverse auctions, and there is a very simple explanation, which is very, I would say, standard business for behavioral economies like.
There is this drive. There's excitement because you get hooked up by the competition, and you and you kept actually to be honest. If I, if I were to mention one single one case.
There was the very famous threeg auction Uk spring, 2,000.
The British telecom ended up paying enormous amount of money because it was a struggle, because the between the 2 top managers who were competing against each other in terms of showing to the company which one was prevailing in terms of the bidders. They ended up paying 20% more. Or that plan. It's very, I would say it's the most I would say.
Gigathi version of windows. It's a firm's curse. So we have to be aware that there are behavioral benefits being triggered during dynamic. So there are many ways of preventing or limited.
Dead risk. This is why I think.
They just reinforces my point that in order to minimize the risk, you have to make sure that procurement designers anticipate that, and so take all the due actions in order to limit the possibility which is always going to be there. The risk is there driving the price down. I get excited. There is this idea. Oh, my God! I'm a good contract. There is just the objective of getting the contract.
It. A second later. The Europe spent too much money, or you're not getting enough money from the government, and you cannot renew your word, because it's gonna be in trouble. So you have to make sure that you. It is the risk of that. There are many ways that I mean, we don't have the time to go through this. But let me mention that this is really a substantial problem in implementing reverse option.
We're gonna have to go a little bit over time with the audience's indulgence. So we have a few more slides to get through just quickly in the unsatisfying process. When we're doing the reverse actual, we realize that this is such a problem. We actually drafted a separate rule on a number of tenders on.
Unrealistically low pricing because it is such a you couldn't have a reverse auction rule without having a sophisticated rule to assess this type, among other things, to assess the performance of abnormally low bids in the Us. System. Under the Us. Rule structure, the Us. Regime under Geo. Case law, it's actually pretty hard unless the agency anticipates the problem put something in the solicitation.
Very possible that this risk will be overlooked.
Okay on the the, the somewhat awkward structure, the rules is of the of the final rule, and the pending rule is that a reverse auction service provider has to allow offers to see success and lowest prices offered in the auction without revealing the the offer is identity. So John Luigi, going back to the model, to the example that we're using at the beginning of the of the session.
Sure.
Look at that pricing above historic pricing. Look at that.
Could that if this were transparent to the bidders, the bidders could see this during the course of the auction.
The dead.
Is it possible that the bidders are signaling to each other in order to preserve and.
Monopoly.
And they like it. And again, let me mention, you know, all the literature on Fcc. Are so when I when I when I mentioned 10 to hundreds of paper. I literally meant this. There are massive production or papers, empirical papers, showing that actually, during the 1st season, I would say mid 90 s.
The the government, I mean economies at that time at the Fcc. Absurd behavior is using the most sophisticated signaling techniques like in numbers related to location. So every single time you enlarge what are called the strategy space, so the richness of the language they can use.
And the number is a language.
Right.
The more they tend to signal. And so they can implement strategies. You bid too low. I'm gonna react so on and so forth. So let me just mention one very singular option is that you observe something going on. You may learn something. You may learn that there is a competitive market out there, and you still are in a position to react.
Alright. So this is the good thing. This is the left hand of the story. This is the Highland of the story, which is, if you know too much, if you learn too much, you can manipulate that information to calm down competitions exactly the opposite effect. So a proper auction design is nothing more than striking the right balance between information which is good for competition.
Healthy competition, as many legal scholars say, around the world, versus too much information which may be manipulated to reduce competition. And how? What is the optimal trade off? Well.
It depends.
It depends on the market depends on, you know, commodity, so on and so forth. But this is a never last intention.
Which needs to be resolved by the guys in charge of the auction design.
What was interesting about the rulemaking process is the coalition for government, which is one of our largest associations. Vendors.
Actually raise this risk, they said, having full price disclosure in the auction raises a risk of collusion, and the rule makers didn't address that risk.
Yeah.
In the final rule. So it's it's a it's a. It's a serious, lingering issue.
Touch.
In terms of when the qualification is made. If the the American norm is to do qualification at the end of the procurement process, because that raises the risk of the contract, among other things, raises the risk of the contracting officer, will simply, at the end of the auction. Say, you know what I've gone through a exhausting auction. I'm just gonna accept the lowest bid. I'm not. I'm just going to find that bitter qualified there. The contracting office essentially has.
A conflict of interest because she wants to be able to move on some systems.
Actually put a create, an, a system where you have a qualification process at the beginning, it's more expensive because you're doing qualification across all the bidders, and not just with just not just on the winner.
Sean Luigi, which which is a which, do you think which is the sound or approach.
Well, the the standard approach is to make sure that a set of potential bidders are qualified. You know, you can count those guys and then leave the uncertainty only.
In the price area. But you have to know that the guys visioners right? And these just fundamental, otherwise, that.
This sort of strategy.
From the government side. Figuring competition does not make sense, because you may use exactly the temptation a non-stars to jump in and drive their price down and then start, you know, kind of argument, and negotiate with the procurement afterwards. So you want to make sure if you have N. Bers there. These are the bidders, or in principle.
Will be entitled to.
Run the business afterwards.
And, in fact, Utah does exactly that the Utah does a pre-qualification assessment be as a 1st phase, and only runs a reverse option.
In a second phase.
The. In terms of the service providers and source concerns the the rule does go into this. There are some aspects of it that are just not addressed. So, for example, the Multilanal Development Bank, several multilateral development banks have issued guidance that suggested very careful.
About the source selection that an independent service provider, private service provider, they know when the ox is gonna take place in that sense that itself is sensitive. Information. Have to be careful about that.
Yeah.
The there are alternative approaches, and one of our students, Ryan Taft.
God bless him! I hope he's on the call today. Wrote a paper in our class about Dave, and I teach a course together on procurement form. He wrote a paper on.
Using victory auctions, and he submitted it as part of the rulemaking process.
Rick. Reaction is this picture you see on the right hand side is where you have various bidders. In this case this reaction is high bit as opposed to low bid. But you have the most successful bidder, the most aggressive bidder. In this case. $23. He wins.
But he only has to pay the $20 price from the second best bidder, and this and for those of you in the Washington real estate market.
You, this actually happens.
The highest bidder.
Get the house at that price.
He pays the second best, better price, because I and I could speak as a survivor of that marketplace. You go a little crazy during auction you overbid, and the victory auction for very complicated and sound economic reasons give awards on the second best price.
No mention of that. Although Ryan submitted his paper, no mention of that alternative approach, no mention or alternative approach that's used in Oregon, where you actually end in the European Union, where you can do a quality assessment first, st and and by feeding the quality assessment into the algorithm can actually assess quality as well during the price based auction. Essentially the the quality is reflected in the prices so.
Sure.
Mercedes would normally be up here forward here. But the algorithm shows Mercedes here because Mercedes has a better quality score, so you could. All you could do, both quality and price auctions together in a very thoughtful way. No mention of that in the final rule also, no mention of this is something you see in Brazil a government.
Using a private reverse, auction, just piggybacking into a private reverse option for quantities.
No mention of that itself either.
Dave, do you think that the the rule should speak more broadly about possible other approaches.
The rule. No, I I think it.
I don't like.
The use of the rule. I think there needs to be some.
Guidance. That's.
That's produced and shared.
But.
The rule.
The more you put in the rule.
The more you limit yourself.
Yeah.
Because.
We aren't.
As a general. As a general rule, we aren't very good.
At at writing discretion into the rules.
We? We're good at putting limitations in, you know, right and left, and and don't.
Don't go over the limitations.
But how do you write?
Use common sense, I mean Mark Twain.
Pointed out,
Quite aptly that common sense isn't so common.
And.
And that that, and and Dick.
Dick Jenman, a friend of mine from a Dod days, used to say that regularly. It's it's not common sense, isn't common, but people need to have some.
Understanding and ideas. If you're going to use the technique.
What ways to do it. And again.
The Ig issue is is the real problem.
If you write it into the rule.
And they don't do it.
It's problematic.
If you don't.
Write it in the rule, then at least, the contracting officer, and hopefully his or her.
Supervisory chain will say, Hey! They use their judgment, which we they already have license to do.
And and they they made their best efforts, and it either worked or didn't work. And hopefully, we've learned something from that. And we add that to our body of knowledge.
Does that answer your question?
It does. Thank you, David. We're gonna return to just a moment. Very briefly, after, after local bright.
Sh! The final rule, unlike the ofpp memorandum, is silent on sharing the pricing information across the government.
She. So you have a reverse option. You get a much lower price for Dell Computer. And there's nothing the final rule that calls for sharing that information, so that other agencies buying dell computers would know about it.
Knowing.
David.
That that is such a tragedy.
Just upgrade.
I mean it's and and it's, and it's not just in the pricing arena. But we ought to.
Learn from the information we get.
And we ought to share it across the Government to benefit both.
The end item users and the taxpayer.
Who's ultimately paying for this.
And, by the way, if you talk to a company.
To the to the supply chain manager of companies.
They know they do, that.
They they share the information for their for on pricing with their various divisions and organizations, because they, you know, you'd they'd fire you if you were the supply chain manager, and you didn't use information you just obtain about what.
A laptop should cost, or what a bolt should cost.
Yeah, if I may, if I may, on this.
This is tragedy is a shame, a shame, and a tragedy. Why? Because, going back to your point, Chris, about the the potential risk of collusion.
If there is, for some reasons those guys end up signalling to each other. And so the agreement takes the 4, which is very customary in markets of a market. I go to one agency, I get, I get the conspirations A, and you go to Agency B, and you get with a very similar price.
You know, potentially high price.
The contract there. If the 2 guys, the 2 agents don't share that information, they will never know. That might be the risk of a market share and agreement going on there. So you maintain that you get that winner, and it's a multi competition, whereas, in fact, there is a market share and agreement going on there. So you need to have.
A way, an incentive or an obligation. I don't know.
What it would be the sort of better ways or more feasible ways in us.
To share, to have them collect, gather information, share information, so that you know, 1st of all, how to set reserve prices in the futures.
And test whether there is anything bad going on.
The marketplace. So this is really crucial.
Hmm.
So one of the things. And this is really for generally this is that this is a quirk in the Us. Rulemaking process. If you have a significant rule that's going to have more than 200 million dollars impact.
On the marketplace, on the economy. You have to have a cost benefit analysis done that's under executive under executive order. This kind of this is kind of a nerdy thing like role. Why, people often say this whole cost benefit thing is.
Is sort of a neo liberal thing that we should do. And there's there's controversy about it.
Yeah.
I think it cost benefit analysis be really useful, because it would force the rulemakers to say, these is where we're using reverse auctions. And this is where we would like to use.
Exactly, exactly.
These are the potential benefits. So the mechanical question is.
In an 800 billion dollars procurement? Is there likely to be more? Say, one is 8 billion dollars? Is there likely to be more than 1% impact from reverse auctions. And I think because 8 billion dollars is much more.
And so.
It's worth doing it.
This is a no-brainer from my perspective, that a cost benefit analysis would be very useful, because it would map out and would give us a benchmark in the future to say, you know what the rulemakers expected to use reverse here. They're not being used here. They're on. It's we need a roadmap, and they're over, and the the cost benefit analysis will allow for that.
Okay, quickly in the lower right effect would, would David and I were talking about, you know, using guidance, using opp memorandum, using the final rule. Right? Was the decision issue by the Supreme Court at the end of June, and we had a We had a webinar on it, and Lower bright says that the Ca, that if someone challenges the final rule or challenges is implementation in a particular procurement that underlow bright, the courts are, gonna look.
Back at the statutory direction, and the important thing to recognize is that there is statutory direction here on the statutory direction, was actually came in 2014, in 2014.
When Congress said that reverse, excuse me. In 2021, Congress, in launching the pending rule on construction services, the 2021 legislation said, the reverse option, provide value for the vast.
Majority of Federal acquisitions. Congress having made a finding.
That reverse auctions provide value for the vast majority of Federal positions. If, for example, a new entrant wants to get into marketplace and is effectively locked out because of the methods being used, and believes that a reverse option would allow that engine to come in. It's very possible that they might challenge the.
Current method being used, saying, Hey, look! If you use the reverse auction, you would.
Opening up the market, which is call for another Competition Act.
And second recognizing call Congress is finding that reverse options provide value for the vast majority.
So it's possible overwrite that we'd see that type of litigation. Now, those be interested on the proposed rule just regarding construction services.
Comments in that close. On October 28.th
2024.
So I just wanna note that we've had a number of different questions that we've that we've tried to address the QA. We appreciate that. Please feel free to follow up with me. If you have additional questions and see you can send a lot of Gw, I can post those onto the program page as well. We we again welcome to our colleagues from Nasbo for our colleagues from the National Institute for Government Procurement.
Those of you interested in international procurement issues. There's going to be a Conference and Nigg Conference, the Icap, Ica. PP. Conference in Dublin and early November, and the International Public Procurement Conference is going to be coming up.
Very soon David and I are going to go. I promise to bring back a picture that's in the United Arab Emirates. I promise to bring back a picture of David standing at the edge of the desert. You should be very.
Conference.
I don't like heat.
Get out of the kitchen.
Any final comments, gentlemen.
If I sorry if I add quis, I mean you. I don't know where you're gonna put the email. But you know any any participant, interested and sort of further exploring. The economic dimensions of reversion are currently, you know, are.
Really, I really look forward to.
Answering email or sharing information.
As they like.
I think.
I think you've proven series of webinars generally. You've proven to us all that we have to have.
An economist at the table.
Thank you very much.
Thank you very much, both of you. Pleasure, as always.
Have a great day.
Thank you for allowing me to join. You guys have a great day.
You too.
Think I did. Bye.