Time-consistency problem under condition of a random game duration in resource extraction Текст научной статьи по специальности «Математика»

Time-consistency Problem Under Condition of a Random Game Duration in Resource Extraction

Ekaterina V. Shevkoplyas

St.Petersburg State University,

Faculty of Applied Mathematics and Control Processes, University pr., 35, Petrodvorets, 198 904 St.Petersburg, Russia E-mail: [email protected]

Abstract We consider time-consistency problem for cooperative differential n-person games with random duration. It is proved, that in many cases the solution (or optimality principle) for such games is time-inconsistent. For regularization of solution the special imputation distributed procedure (IDP) is introduced and the time-consistency of the new regularized optimality principle is proved. At last we consider one game-theoretical problem of nonrenewable resource extraction under condition of a random game duration. The problem of time-consistency for the Shapley Value in this example is investigated.

Keywords: time-consistency, random duration, Shapley Value, differential game, non-renewable resource.

1. Basic Model: a Game-Theoretic Model of Nonrenewable Resource Extraction with Random Game Duration

Consider one simple model of common-property nonrenewable resource extraction published in (E.J. Dockner, S. Jorgensen et al., 2000).

Let x(t) and ci(t) denote respectively the stock of the nonrenewable resource such as an oil field and player i’s rate of extraction at time t. We assume that ci(t) > 0 and that, if x(t) = 0, then the only feasible rate of extraction is ci(t) = 0. Let the transition equation has the form

Kt) = -^2 ci(t); (1)

i=1

lim x(t) > 0; (2)

t — ><X>

x(to )= xo. (3)

The game starts at t0 from x0. We suppose that the game ends at the random time instant T with exponential distribution f (t) = p * e—p(t—t0\ t > t0.

Each player i has a utility function h(ci), defined for all ci > 0. The utility function depends on elasticity of marginal utility n > 0, so we have two forms of h(c):

fA1n(ci)+ B, if n = 1; h(ci) = ^ i-V (4)

[A^ + B, if v?i. [ }

Here, A is positive and B is a constant which may be positive, negative or zero.

We define integral expected payoff

p OO pt

Ki (xo, ci,... , cn) = I I h(cj (r))pe-p(t-t° ^ drdt, i = 1,...,n.

-'to -'t°

and consider total payoff in cooperative form of the game:

n n

maxV' Ki (xo ,ci ,...,Cn) = V' Ki (xo, c[,..., cn) = iCii ^ ^

1 ^ i=i i=i

p OO p t

= I I h(cf )pe-p(t-t° ^ drdt.

«/ t° «/ t°

2. The Shapley Value

We suppose that before begining of the game players chose the Shapley Value as optimality principle. It is common knowledge, that the formula for a Shapley Value in the n-person game has the form

Sh, = 2 («-«W-nyp,_mm «.i.................................(5)

n!

ScN

ieS

The common way to define the characteristic function in r(xo) is as following:

f0, S = 0;

V(S, xo) = J ma™S£^ Ki(xo,u), S c N (6)

I ma^n=i Ki (xo, u), S = N.

V u

But this approach doesn’t seems to be the best in context of environmental or other problems, because unlikely that if a subset of players form a coalition to tackle an enviromental problem, then the remaining players would form an anti-coalition to harm their efforts. For enviromental problem we can use another method of characteristic function construction with assumption that left-out players stick to their feedback Nash strategies. This approach was proposed in (Petrosjan L. A., Zaccour G.,2003).

Then we have the following definition of the characteristic function:

[0, S = 0;

V(S,x*($)) = \Wi(x*($),$), i = 1,...,n; {i} e I; (7)

[wk(x* ($),$), K C I,

where Wi(x*($),'#), WK(x* ($),'#) are the results of the corresponding Hamilton-Jacobi-Bellman equations. Remark, that the constructed function V(S, x* ($)) (13) is not superadditive in general.

If we use the characteristic function V(S, x*($)) (13) constructed by concept of Petrosjan and Zacoour there is a need to examine the superadditivity of this function.

The Hamilton-Jacobi-Bellman equation appropriates to a problem with random game duration had been derived i paper (Shevkoplyas E.V., 2005). So, we get

the Hamilton-Jacobi-Bellman equation in general case of arbitrary distribution function F(t) (and we suppose an existence of density probability function f (t) = F'(t)):

f ($) ^ dW (x,$)

-W(x,'i9) = -------—--------h max

U

(8)

1 - F($) v

Suppose that the final time instant T has the exponential distribution. Let us remark that for a problem with random duration (T—to) e [0, to) the first term on the right-hand side (8) is equal to zero (dWdl’x^ = 0) f°r a case °f exponential distribution, but it doesn’t satisfy for arbitrary distribution. Then the Hamilton-Jacobi-Bellman equation (8) get the form:

pW(x, t) = max -f H(x(t), u(tj) -|-----^ ^ g(x, u) 1. (9)

« ^ dx J

This equation looks like Hamilton-Jacobi-Bellman equation for the infinite time horizon problem with discount factor p (E.J. Dockner, S. Jorgensen et al., 2000).

2.1. Algorithm

Thus, in the case of the random game duration we propose the following algorithm of the Shapley Value calculation.

(1) Maximize the total expected payoff of the grand coalition I.

1 n r> OO p t

Wj(x,i9) = max-------- hi{x{T))drf{t)dt, (10)

Ci,iG/ 1 — F ($)

=i

x($) = x.

Denote n=i hi() by H(). Then the Bellman function W/(x, $) satisfies the HJB equation (8). Results of optimization are optimal trajectory x1 (t) and optimal strategies c1 = (c{,..., cn).

(2) Calculate a feedback Nash equilibrium.

Without cooperation each player i seeks to maximize his expected payoff (??). Thus the player i solves a dynamic programming problem:

Wi(x,ti)= max----------1—— [ [ hi(x(T))dTf(t)dt, (11)

ci 1 — F ($) J^ 7^

x($) = x.

Denote hi () by H(). In this notation Wi (x, $) satisfies the HJB equations (8) for all i e I.

Denote by cf(■) = {cf(■), i = 1,...,n} any feedback Nash equilibrium of this noncooperative game r(xo). Let the corresponding trajectory be xf (t). We calculate Wi (x* ($), $) under condition that before time instant $ players use their optimal strategies c1.

(3) Compute outcomes for all remaining possible coalitions.

1 rO rt

WK(x,ti)= max------------hi(x(T))dTf(t)dt, (12)

c„iGK 1 — F Wf-K-h .h

cj = cf for j e I \ K,

x($) = x.

Here we insert for the left-out players i e I \ K their Nash values (see step 2). In the notation ^iGK hi() = H() the Bellman function WK(x,$) satisfies the corresponding HJB equation (8).

(4) Define the characteristic function V(S, x*($)), VS C I as

'0, S = 0;

V(S,x*($)) = { Wi(x*($),$), i = 1,..., n; {i}e I; (13)

^ WK(x*($),$), K C I.

(5) Test of the superadditivity for characteristic function (13).

We need to check the following inequality satisfiability:

V (xo ,SiUS2)>V (xo ,Si) + V (xo ,S2).

If the superdditivity of the characteristic function is fulfilled, go to the next step. Otherwise we can not use the Shapley Value as optimality principle for the players, but we can consider another optimality principles like Banzaph Value and others.

(6) Calculation of the Shapley Value by formula:

Shi = £ (»-,)K«-l)![y(S) _ v(svm , =

ScN '

ieS

2.2. Solution

I. Consider the case of logarithmic utility function:

h(ci (t)) = Aln(c) + B.

Further we use 5 steps of our algorithm to calculate the characteristic function values.

Step 1. Let us consider the grand coalition I = {1,..., n}. Then the Bellman function is as follows:

pOO pt ____

W/(x) = max / / (A * ln(ci)+ A * ln(cj)+ nB)p * e-p(t-tf)dTdt. (14)

{c,,ie/Wtf

j=i

Let us define 5^n=i hi(ci()) = H(c()). Then we can use the Hamilton-Jacobi-Bellman equation (9) :

pW7(a;) = max (H(c) H------^ ^ g(x, c)^ . (15)

c \ dx )

Combining (15) and (14), we obtain pW/ (x) = max (a * ln(ci) + A *

EM,l + n6 + ^,_5-E«;l .

\ j=i j=i /

(16)

Suppose the Bellman function W/ has the form

W/ = A/ln(x) + B/. (17)

Then we get

dWtix) Af

= V (18)

Differentiating the right-hand side of (16) with respect to ci, we obtain optimal strategies

A

(19)

c' = *

dWj(x) ' dx

Using (18 and (19), we get

nA

x = —— x. (20)

Ai V 7

Substituting (31), (19) and (20) in (16), we have

pA/ln(x) + pB/ = —nA + nB + Anln(A) — Anln(A/) + Anln(x). (21)

The result is:

.4, = (22)

p

g Bn An Anln(n) ^ Anln(p)

P P P P '

From (14) and (22) it follows that

, An, , . Bn An Anln(n) Anln(p)

W/(x) =--------Zn(x) +-------------------------— +---------—. (23)

P P P P P

Then we get the optimal strategies

cf = —x, i = 1,... ,n. (24)

in

Finally, we have optimal trajectory and optimal controls

x/(t) = xo * e—p(t—to); (25)

cf(i) =

in

and

V (I,x/(tf)) = W/ (x/ (tf)) = (26)

An , Bn An Anln(n) Anln(P)

= ----/n(xJ) H---------------------— H---------— =

P P P P P

An, . . , . „ . Bn An Anln(n) Anln(P)

= —ln(xo) - An{{) - t0) H-------------------------— H---------—.

P P P P P

As you can see the optimal trajectory x/(t) satisfies Lyapunov stability condition. Let tf = to. Then

, An, , . Bn An Anln(n) Anln(P)

l/(/,x0) = ^/(x0) = ------ln(xo) +---------------------^ +(27)

P P P P P

Step 2. Now let us find a Feedback Nash equilibrium. The Bellman function for player i is as follows:

Ot

Wi(x) = max / / (Aln(ci(T)) + B)p * e-^-^drdt. (28)

c,

The initial state is

x(tf) = x/(tf). (29)

Now the HJB equation (15) has the form

pWi(x) = max ^n(cj) +B + —• (30)

We find Wi in the form

Wi = Af ln(x) + BN. (31)

As before we get

A

AN = -■ (32)

P

„ B An Aln(p)

B;v =-----------+---------—•

P P P

cf = px, i = 1,..., n; (33)

xN(t) = x/(tf) * e-np(t-^); (34)

cf (t) = px/(tf) * e-np(t-^;

V({t}, x1^)) = Wi(xI('&)) = -Zra(xJ(tf)) + — - — + (35)

P P P P

Let tf = to. Then

y({*},x0) = Wi(x0) = -Zn(xo) + - - — + (36)

P P P P

The main results obtained by steps 1,2 firstly had been published

in E.J. Dockner, S. Jorgensen et al., 2000 for a case of A = 1, B = 0.

Step 3. Let us consider a coalition K C I, |K| = k, |1 \ K| = n — k. For this case we have the Bellman function:

Ot

WK(x) = maW / (A ln(ci(T)) + fcB)p * e-p(t-^)dTdt. (37)

The initial state is

x(tf) = x/(tf). (38)

Let us recall, that the left-out players i e I \ K will use feedback Nash strategies

(45).

In the same way, we get

xK (t) = x/(tf) * e-(n-k + i)p(t-^) ; (39)

cf(t) = yX1^) * e-(n-k+1)p(.t-'a). k

V (K,x/(tf)) = WK (x/(tf)) = (40)

Ak, r/nw kB Ak Ak(n — k) Ak Akln(p)

= ----InixHti)) H-----------------------------------ln(k) H----------—.

P P P P P P

Let tf = to. Then

. Ak kB Ak Ak(n — k) Ak Akln(p)

V(K,x0) = WK(x0) = —Zn(x0)) +---------------------------------------------Zn(fc) +-------—.

P P P P P P

(41)

Thus we have constructed the characteristic function V(K,xo),K C I (see (27),(41)).

Proposition 1. Suppose the characteristic function V(K, xo),K C I is given by (27),(41). Then V(K, xo) is superadditive.

To prove this proposition, we need following lemma.

Lemma 1. Let si > 1,s2 > 1. Then

siln(si) + s2ln(s2) + 2sis2 > (si + S2)ln(si + S2). (42)

This lemma can be proved by standard methods. It is easily shown that the left-hand side is fast increasing than the right-hand side.

Now proof of Proposition1 is by direct calculations.

Finally, we get the Shapley Value in our example:

V(I,x) A ,■ \ B A Aln(n) Aln(p)

ShAxJt)) = v ’ 7 = -(n x +---------------------^ 43

n P P P P P

V (I,xo) A n B A Aln(n) Aln(p)

57ii(x0) = v ’ u/ = -ln(x0) +------------------------— +--------—. (44)

n P P P P P

II. Consider a problem with utility function

ci-n

/i(ci)=A-5------+B, 77^=1.

1 — n

We find W in form W = Axi-n + B. We get the optimal strategies

i P

q = —X, l = 1,..., n.

nn

Then we get optimal trajectory and optimal controls

r _ p(*~*o)

x (t) = x0 e ^ ;

I xo P o)

q t =--------e -7 .

nn

Then we get the values

V(I,x('0)) = Wi(x) = ——x('0)1-^ + —;

VP / 1 — n P

Vi^x1 (■<})) = WI(xI(i9)) = (^—x1^)1-’1 + — =

Vp/ 1 — n p

nri\v A 1_„ p(i-v)(#-to) nB

— i----------*o e " + —•

P / 1 — n P

Then we get the Nash feedback strategies and trajectory

cf = --------^* = 1 (45)

(1 — n + nn)

xw(t) = xJ(tf) e“a-"+-7>(t_,?);

W/,x PX1^) _ ( d)

d (t) = —-----------e (i-~+^)v >.

(1 — n + nn)

Obviously, cf > 0 if n > (1 — 1/n). So we’ll consider the game under condition

n > (1 — 1/n). Otherwise feedback Nash strategies as rates of resource extraction

make no sense.

So, we get the value

V({i}, x(<9)) = Wi(x) = ((1 ~ n + nT]) V x(tf)1^ +

V P / 1 — n P

y({*},xJ(tf)) = W^x1 (tf)) = f(1 n + n^y _j-x1^)1-^ + -

V P / 1 — n P

(l-n+W?)y A ^l_^^_£%JZl(^_to) + B

p ) 1 -ri 0 p'

In the same way, we get ’’optimal” for coalition K controls

K p(1 — k + kn)x

ci =7771---------T---V’ i = l,...,n;

kn(1 — n + nn)

and the value of coalition payoff

= vw*)) = (fo'(1

V p(1 — k + kn) / 1 — n P

Thus we have constructed the characteristic function V(K, xo),K C I. We can prove the following proposition.

Proposition 2. The characteristic function V(K,xo),K C I is superadditive.

To prove this proposition, we need following lemma.

Lemma 2. Let si > 1,s2 > 1. Then

__________(S1+S2)?)_____________ >_________S1_______+__________*1_______. (46)

(1 — (si + S2) + (si + S2)n)n (1 — si + si n)n (1 — S2 + S2n)n

This lemma can be proved by standard methods. It is easily shown that the left-hand side is fast increasing than the right-hand side.

We get the following components of the Shapley Value:

a (x(t) \1-n ( p Yn b

= —(irj UJ +7; <47)

= )l~,f£') " + -• №

1 — n V n ) \n J p

3. Time-consistency Problem

It would seem that the superadditivity of the characteristic function V (x0, S) should provide saving the cooperation of players during the game, but it is not so. Really, moving along the optimal trajectory x*(t) players enter into subgames with current initial states, in which the same player has different possibilities. Therefore, in some moment it may happen, that the solution of the current game will be not optimal in sense of originally selected optimality principle and the desire to operate jointly can expose the threat in some moment tf. It means that the optimality principle may loose time-consistency.

Let Sh = {Sh*} is the Shapley Value in the whole game.

Definition 1. Consider vector function 7 (t)} = {7* (t)}, such that

p OO p t

Sh* = / Yi (T)drdF(t). (49)

J 10 J 10

Vector function Y(t) = {7* (t)} > 0 is called the imputation distribution procedure (IDP).

IDP determines a rule, according to which the components of the Shapley Value are distributed on interval [t0, t], where random value t is the final time instant of the game.

Payoff obtained by player i on an interval [t0, ) is denoted by a* (tf):

a* (tf) = f ( Yi (t)drdF(t). (50)

t0 t0

Definition 2. The Shapley Value is time-consistent (TCSV), if exists IDP {y*(t)}>0, t <G [to, to), such that the vector Sh^ = {Shf} (expected payoff in subgame r(x*(tf)), calculated by the formula

1 rO rt

Sht = 1 _ J^ J^li{T)dTdF{t), i = (51)

belongs to the same optimality principle in the subgame , i.e. Sh^ is the Shapley Value in subgame r(x*(tf)).

It can be easily seen that TCSV is equivalent to the following formula for the imputation Sh:

Shi = a*(tf) + (1 — F(tf)) ( Jo 7*(t)dT + Sh/j . (52)

Differentiating (52) with respect to t we get the formula for IDP:

'KW = - (Shi*)'. (53)

Let us consider the case of exponential distribution f (tf) = F‘(tf) = peAt. Then we get the formula

Yi(tf) = pShi* - (Shi*)'• (54)

This formula coincides with formula for IDP for the problem with infinite time horizon and discount rate p had been derived in Petrosjan L. A., Zaccour G.,2003. So, we had another validation of the idea that the problem with infinite time horizon and discounting of the payoffs supplies the same result as the problem with random game duration and exponential distribution.

It is important that Yi(tf), i = should be nonnegative. As it is

impossible to guarantee this in general, not all optimality principles are time- consistent.

Remark. We can change the restriction for IDP Yi (tf), i = 1, • • •, n to be nonnegative for the problem of minimization of the total costs (but not maximization of the total payoffs), because in this case IDP means a rule of distribution of the players costs during the game.

3.1. New characteristic function

Define the following function

F(*o,S)= ^ f V(x*(r),S) drdF(t),

Jto Jto V (x*(t ),N)

S C N. (55)

It is clear that

V^(x0, N) = V(x0, N), V(x0,0) = 0.

Also we have

F(xo ,Si U S2) > V(xo ,Si) + F(xo ,S2) •

It follows from the superadditivity of the characteristic function V(x0, S). Thus, following lemma is true.

Lemma 3. Function V(xo, S) is the characteristic function in the game r(xo).□ Similarly, it is not difficult to show, that the function

W - S) = 1 ^ f V{x* (r), S)^1^*^ drdF(t).

1 - F(tf) J* J* V(x*(t),N)

is the characteristic function in the subgame r(x*(tf)).

3.2. Regularization

Introduce new optimality principles, which are based on classical optimality principles and are always time-consistent. Remark that the regularization will be correct only for the case of nonnegative instantaneous payoffs hi(■) > 0, i = 1,•• •, n. Consider new IDP

It is clear, that Yi(tf) > 0, Vtf, i = 1, • • •, n.

REMARK 1. It is easy to show, that sum of all new IDP components (56) at the intermediate time instant tf, tf <G [to, to) is equal to sum of all instantaneous payoffs at the time instant tf, i.e.

n n

Yl Yi(tf) = Y1 hi (x*(tf)x (57)

i=1 i=1

because YI”=i Shi* = V(x*(tf),N). Thus, according to new IDP at the each time instant we divide the sum of the instantaneous payoffs of the players obtained at the same time instant.

Define vector Sh = {Shi} by formula

Shi = [ [ Shi^71 ^ drdF(t). (58)

to to

V (x* (t ),N)

It is not difficult to show, that Y^n=1 Shi = V(xo, N) = V(xo, N). Hence, Sh = {Sh1, • • •, Shn} is the allocation of the total expected payoff.

Proposition 3. The allocation of the total expected, payoff Sh = {S hi} (58) is the imputation, i.e. S hi > V(xo, {i}).

To prove this proposition note, that ShT is the imputation in the subgame r(x*(t)), then ShiT > V(x*(t), {i}). Hence

S% = f Shi^f^f^ drdF(t) >

Jto Jto V(x* (t),N)

> r f V(x*(r), {i}) drdF(t) = V(xo, {*})•

to to

V (x* (t ),N)

Theorem 1. The sregularized Shapley Value S h is time-consistent optimality principle.

Introduce new imputation in the subgame by formula

Sh* = ‘ r

1 - F (tf) J * J* V (x* (t ),N)

Because of

ql ("° [* qh rEtl hi{x*(r)) f°° f* TEtl j

Sh' = LLSk' -n^FhWdTdm + i iSh‘ V(x-(r),Af)

we get

qi ("° [* qh rEi=i hi(x*(T)) , . n ("° qh rELi hi(x*(T)) j

, = LL W(r).№) Sh, V(x.(r),W) *+

+ J„ Xs** <iT<iF(f)' =

/** /* t

/** />t / /**

/ / Yi(T)dTdF(t) + (1 - F(tf))( / Yi(T)dT + Shi*

to to to

Thus, the imputation Sh has the form (52). Therefore, the imputation Sh* in the subgame starting at the moment tf belongs to the same optimality principle.

Theorem 2. The Shapley Value calculated for the characteristic function V(xo, S) is time-consistent.

We have the following formula for the Shapley Value:

S)>, = £ («-«w-nyp,_mm «.i.............................(59)

n!

ScN

ieS

Therefore, the components of the Shapley Value in r(xo) for the characteristic function V (xo, S) are computed by the formula

S%(x0) = V' ———— [V(x* (r), S) - V'(x*(r),5\{i})]. (60)

£N n!

i€S

Call Sh(xo) —regularized Shapley Value (RSV). From (55) and (60) we get the following form of RSV:

Sh,M = r f (n~^‘~1)! Y, №».«)-

Jt 0 Jto ! ScN

-y(x*(r),S\{i})]Eyft:(a;*(T))drdF(t) =

V (x*(t )) r fP1 , *, ^Er=iM**M) ?

= Jk <T)) V(x-(r))

Similarly, it is not difficult to show, that the components of the Shapley Value Sh(x*(t)) in the subgame r(x*(t)) are equal to

cl, , *i w 1 r f ci, t *t hi(x*(t)) j

Shi(x (t = ------—— Shi(x r -------- ---dr dF(t).

1 - F(tf) y* J* V(x* (t))

Define IDP 7 (r), r G [t0, t] as ^(r) = 5hj(x* (r)) ^ iy1(^((*))(T)) • We get

r* r t / r*

nt f c*

Y(t)drdF(t) + (1 - F(tf))( / Yi(T)dT + Shi(x*(tf))

o to

Thus, Shapley value for the characteristic function V(xo,S) is time- consistent.

Then for regularization of the Shapley Value we may use 2 ways such that to use new IDP y(t) or to calculate the regularized Shapley Value by new characteristic function V (xo, S).

3.3. Time-consistency problem in game-theoretical problem of nonrenewable resource extraction

Let us investigate the Shapley Value (44) for logarithmic utility function on time-consistency. We have to calculate the IDP by formula (54). We get:

It is obviously we cannot guarantee the nonnegativity of the IDP in (61). Then, the Shapley Value (44) is time-inconsistent. But we can not regularized it because the instantaneous function hi = ln(ci) is not nonnegative function.

In the case of power utility function we have the Shapley Value (47). We use the formula (54) for IDP calculation. Then we get

It is clear that IDP (62) is nonnegative for n G (0; 1) and B > 0. Then the Shapley

Value is time-consistent optimality principle for n G (0; 1) and B > 0.

References

Dockner, E. J., S. Jorgensen, N. van Long, G. Sorger. (2000). Differential Games in Economics and Management Science. Cambridge University Press.

Petrosjan, L. A., Zaccour G. (2003). Time-consistent Shapley Value Allocation of Pollution Cost Reduction. Journal of Economic Dynamics and Control, 27, 381-398.

Petrosjan, L.A., Shevkoplyas E. V. (2001). Cooperative Differential Games with Random Duration. St.Petersburg, ”Vestnik SPbGU”, series 4, release 1, pp. 21-28.(in Russian)

Petrosjan, L .A., Shevkoplyas E. V. (2003). Cooperative Solutions for Games with Random Duration. Game Theory and Applications, Volume IX. Nova Science Publishers, pp. 125-139.

Shevkoplyas, E. V. (2005). On the Construction of the Characteristic Function in Cooperative Differential Games with Random Duration. International Seminar ’’Control Theory and Theory of Generalized Solutions of Hamilton-Jacobi Equations” (CGS’2005),Ekaterinburg, Russia. Ext.abstracts. Vol.1,pp. 262-270 (in Russian).

Yi (tf) = Aln(xo) - Ap(tf - to) + Aln(p) - Aln(n) + B^

(61)

77(1 - ??) p'

1 + B

(62)