Bayesian reference analysis for exponential power regression models

Theorem 1

Proof

See the Appendix. □

While reference prior ${\pi }^{r_2}$ is a new prior that has not appeared before in the literature, there are similarities between the reference priors given in Theorem 1 and the independence Jeffreys priors given in Equations (5) and (6). Reference prior ${\pi }^{r_1}$ coincides with the independence Jeffreys prior ${\pi }^{I_2}$ given in Equation (6). Moreover, it is important to point out that reference prior ${\pi }^{r_2}$ is somewhat similar to the independence Jeffreys prior ${\pi }^{I_1}$ given in Equation (5), differing only by a factor of p^−1/2. However, as we show below this difference between ${\pi }^{I_1}$ and ${\pi }^{r_2}$ is enough to make ${\pi }^{I_1}$ yield a useless improper posterior distribution while the reference prior ${\pi }^{r_2}$ yields a useful proper posterior distribution.

Thus, in order to determine whether a prior of the form (4) leads to a proper posterior distribution, one needs to investigate the tail behavior of both the marginal prior and the integrated likelihood for p. The tail behavior of the marginal reference priors for p given in Theorem 1 is given in the following lemma.

Lemma 1

The marginal priors for p given in Theorem 1 are continuous functions in [ 1,∞) and are such that ${\pi }^{r_1}(p)=O\left(p^{-3/2}\right)$ and ${\pi }^{r_2}(p)=O\left(p^{-3/2}\right)$ as p→∞.

Proof.

Direct inspection shows that ${\pi }^{r_1}(p)$ and ${\pi }^{r_2}(p)$ are continuous functions in [ 1,∞). Their tail behavior when p→∞ follows from the fact that Ψ^′(1+p⁻¹)→1.6449 and Γ(p⁻¹)=O(p) as p→∞. □

Theorem 1 and Lemma 1 suggest the definition of an approximate reference prior inspired by priors ${\pi }^{r_1}$ and ${\pi }^{r_2}$ that has the same value for the hyperparameter a=1 and share their tail behavior with respect to p. We define such an approximate reference prior in Definition 1

Definition 1.

We define an approximate reference prior ${\pi }^{r_3}$ to be of the form (4) with a=1 and marginal prior for p equal to ${\pi }^{r_3}(p)\propto p^{-3/2}$.

Computation of prior ${\pi }^{r_3}$ is faster and more straightforward than that of priors ${\pi }^{r_1}$ and ${\pi }^{r_2}$. In addition, Section 4.1 shows that the frequentist properties of procedures based on ${\pi }^{r_3}$ are similar to those based on ${\pi }^{r_1}$ and ${\pi }^{r_2}$. As a consequence, the approximate reference prior ${\pi }^{r_3}$ may become more widely used than the reference priors ${\pi }^{r_1}$ and ${\pi }^{r_2}$. Therefore, henceforth we drop the term “approximate” and simply refer to ${\pi }^{r_3}$ as a reference prior.

The following lemma, that was proved by Salazar et al. (2012), provides the tail behavior for the integrated likelihood for p.

Lemma 2 (Salazar et al.2012)

Provided that n>k+1−a, the integrated likelihood for p under the class of priors (4) is a continuous function in [ 1,∞) and is such that L ^I (p;y)=O(1) as p→∞.

The following proposition establishes that the two reference priors that we have obtained yield proper posterior distributions.

Proposition 1

Provided that n>k+1−a, the two reference priors ${\pi }^{r_1}$ and ${\pi }^{r_2}$ given in Theorem 1 yield proper posterior distributions.

Proof.

This proposition follows directly from condition (11), and Lemmas 1 and 2. □

To implement posterior analysis for the parameters of the EP regression model based on the reference priors developed here, we use an approach proposed by Salazar et al. (2012) that combines Laplace approximations and Newton-Cotes integration.

4.1 Frequentist properties

In this section we perform a simulation study to access the frequentist properties of Bayesian procedures based on the reference priors ${\pi }^{r_1}$, ${\pi }^{r_2}$, and ${\pi }^{r_3}$. In addition, we compare the performance of these reference priors to that of a competing noninformative prior π ^U that takes the form (4) with a=1 and π ^U (p)∝1 for 1<p<10 and π ^U (p)=0 otherwise. The joint prior π ^U (θ) leads to a proper posterior distribution, however as we see below the uniform prior π ^U (p) is a naïve way to express lack of information about p. The Bayesian procedures we consider are the posterior modes and posterior medians for point estimation, and the 95% highest posterior density (HPD) credible intervals for interval estimation. Finally, we consider three frequentist measures of quality. For evaluating the quality of point estimation, we consider the square root of the frequentist relative mean squared error. For evaluating the performance of interval estimation, we consider two frequentist measures: the frequentist coverage and the mean length of the credible intervals.

We have considered several combinations of sample sizes and parameters. Specifically, we have considered three sample sizes: n=30, n=50 and n=100. Moreover, we have considered a grid of values for p on the interval from 1 to 3. Further, for each simulated dataset we have used k=2, x _i =(1,x_1i), x_1i∼N(2,1), β=(1.5,−3), and σ=1. Finally, for each combination of parameter values and sample sizes, we have simulated 1,500 datasets to estimate the frequentist properties of the several procedures.

The square root of the relative mean squared error (RMSE), $\sqrt{\mathit{\text{MSE}}(\hat{\theta })}/\theta$, for estimators of p and σ is shown as a function of p in Figure 1. As intuitively expected, for all priors and for both posterior mode and median, as the sample size increases the RMSE decreases. The most substantial differences are between the performances of the posterior mode and posterior median, and between the performances of the reference priors when compared with the π ^U prior. First, we compare the performance of the posterior median and the posterior mode. For each prior, for the estimation of p, the posterior median provides smaller RMSE than the posterior mode for most values of p considered except for p close to one. And this advantage of the posterior mode becomes less pronounced as the sample size increases. For each prior, for the estimation of σ, the posterior median provides smaller RMSE than the posterior mode. Therefore, for the reference analysis of the EP regression model we recommend the use of the posterior median.

Second, we compare the RMSE performance of the different priors. For each type of point estimator considered here, in terms of RMSE the reference priors ${\pi }^{r_1}$, ${\pi }^{r_2}$, and ${\pi }^{r_3}$ provide qualitatively similar results, with ${\pi }^{r_1}$ and ${\pi }^{r_3}$ being slightly better for smaller values of p and ${\pi }^{r_2}$ being slightly better for larger values of p. In addition, the difference in performance of the three reference priors becomes smaller as the sample size increases. In contrast, the performance of the reference priors differs dramatically from that of the π ^U prior. For each class of estimators of p and for all values of p considered, when compared to the π ^U prior the reference priors lead to smaller RMSE. For the estimation of σ, the results are mixed; for small sample sizes while the reference priors lead to smaller RMSE when p is small and π ^U leads to better results when p is larger. But for larger sample sizes the reference priors-based posterior medians have smaller RMSE for all considered values of p.

The frequentist coverage (FC) of 95% HPD credible intervals for p and σ is shown, as a function of p, in Figure 2. As the sample size increases, the FC of the credible intervals based on the four priors becomes more similar. For both parameters, the ${\pi }^{r_1}$-, ${\pi }^{r_2}$-, and ${\pi }^{r_3}$-based credible intervals have frequentist coverage closer to the nominal level. This superiority of the Bayesian reference analysis is particularly pronounced for sample sizes equal to 30 or 50 and when p<2.

The mean length of the 95% HDP credible intervals for p and σ is shown, as a function of p, in Figure 3. For the credible intervals based on the three reference priors, the mean lengths of the credible intervals are similar with slightly better results for ${\pi }^{r_1}$. For interval estimation for σ, the mean lengths of the credible intervals based on the three reference priors are smaller than the mean lengths of the credible intervals based on the π ^U when p<2 and are larger when p>2. For interval estimation of p, in the range of values that we consider the ${\pi }^{r_1}$-, ${\pi }^{r_2}$-, and ${\pi }^{r_3}$-based credible intervals are on average shorter that those based on π ^U . Therefore, for the interval estimation of p, in the range of values we consider, the credible intervals based on ${\pi }^{r_1}$, ${\pi }^{r_2}$ and ${\pi }^{r_3}$ provide uniformly superior results.

In summary, the reference priors ${\pi }^{r_1}$, ${\pi }^{r_2}$, and ${\pi }^{r_3}$ lead to procedures that have similar frequentist properties. In addition, when compared to the competing noninformative prior π ^U , the reference priors ${\pi }^{r_1}$, ${\pi }^{r_2}$, and ${\pi }^{r_3}$ lead to overall superior results. Finally, the reference prior ${\pi }^{r_3}$ has a simpler functional form and is more straightforward to be implemented. Therefore, in cases when there is no prior information for the analysis of EP linear regression models, we recommend the use of the reference prior ${\pi }^{r_3}$.

4.2 Applications

This section illustrates the use of the Bayesian reference analysis we propose for exponential power regression models with applications to two real world datasets. The first dataset illustrates leptokurtic errors and the second dataset illustrates platykurtic errors. Because the results based on the reference priors ${\pi }^{r_1}$ and ${\pi }^{r_3}$ are extremely similar, we show only the results for priors ${\pi }^{r_1}$, ${\pi }^{r_2}$, and π ^U .

In both applications, we use the same truncation point at p=10 used for π ^U (p) in Section 4.1 and assume π ^U (p)∝1 for 1<p<10 and π ^U (p)=0 otherwise. We have chosen the truncation point at p=10 because datasets generated with p=10 or with p close to 10 have similar statistical behavior. Hence, to distinguish whether a process follows an EP distribution with p=10 or, say, p=10.1 we would need an extremely large data set. Moreover, the choice of truncation should be made before the analyst looks at the data. For example, for the first application below, after looking at the scatterplot one may think about truncating the prior for values of p that correspond to leptokurtic distributions, that is, 1<p<2. However, doing that would mean to use the data twice in the Bayes Theorem formula: once through the prior, and another time through the likelihood. Usually, such double use of the data leads to underestimation of the uncertainty. Therefore, we prefer to decide the truncation of the prior before looking at the data.

4.2.1 School spending

We analyze the relationship between per capita spending in public schools and per capita income by state in the United States. This dataset has been previously analyzed by Greene (1997), Cribari-Neto et al. (2000), and Fonseca et al. (2008). Specifically, Greene (1997) and Cribari-Neto et al. (2000) proposed analyses based on heuristic approaches to the so-called problem of heterocedasticity-of-unknown-form. In contrast, Fonseca et al. (2008) have analyzed this dataset in the context of linear regression models with Student-t errors. Fonseca et al. (2008) found that when errors with distributions with heavy tails are assumed, a linear model is superior to a quadratic model. Here, we take a similar approach as that of Fonseca et al. (2008) in that we assume a linear model with errors that may have a heavy tail distribution. However, we assume that the errors follow an exponential power distribution.

Table 1 presents the posterior summaries based on the priors π ^U , ${\pi }^{r_1}$, ${\pi }^{r_2}$. For all three priors both the posterior mode and the posterior median for p are smaller than one. In addition, both ${\pi }^{r_1}$- and ${\pi }^{r_2}$-based 95% credible intervals for p are contained in the interval (1,2) indicating evidence that the errors are leptokurtic. In contrast, the π ^U -based 95% credible interval for p is not fully contained in the interval (1,2). However, from the results in Section 4.1 we know that for small true values of p, the use of the π ^U prior leads to on average wider credible intervals for p that have lower coverage than nominal. Thus, this application provides an example when the superiority of the ${\pi }^{r_1}$ and ${\pi }^{r_2}$ priors matters to the conclusion that in this data set the errors distribution is leptokurtic.

	π U			${\mathit{\pi }}^{{\mathit{r}}_1}$			${\mathit{\pi }}^{{\mathit{r}}_2}$
	Mode	Median	95% C.I.	Mode	Median	95% C.I.	Mode	Median	95% C.I.
p	1.18	1.33	(1.02, 2.03)	1.06	1.26	(1.00, 1.91)	1.08	1.27	(1.00, 1.92)
σ	52.73	54.75	(38.59, 74.44)	51.21	53.23	(38.08, 73.43)	51.72	53.23	(38.08, 73.43)
β 1	-89.37	-92.51	(-131.79, -37.85)	-91.51	-88.38	(-131.81, -36.86)	-91.51	-88.38	(-131.81, -36.86)
β 2	616.07	603.95	(525.16, 667.59)	609.99	600.90	(525.15, 667.57)	609.99	600.90	(525.15, 667.57)

Figure 4(a) shows the scatterplot for the school spending data set along with the fitted EP regression model based on ${\pi }^{r_1}$ (solid line), ${\pi }^{r_2}$ (dashed line), and π ^U (dotted line). Figure 4(a) also shows the fitted Gaussian linear model (dot-dashed line). While the Gaussian model fit is clearly and strongly influenced by the outlier, the use of exponential power errors (with the four priors considered here) automatically makes the analysis robust against outliers. In particular, the model fits using the ${\pi }^{r_1}$ and ${\pi }^{r_2}$ priors (considering the posterior median) coincide and are equal to $\hat{y}=-88.38+600.9x$. Another way to make the analysis robust against outliers is to use Student-t errors. Assuming Student-t errors, a model fitted by Fonseca et al. (2008) was y=−75.3+583.2x. We can see that both Student-t and exponential power errors fits are robust against outliers. However, the Student-t distribution cannot accommodate platykurtic errors and, therefore, the exponential power distribution provides more flexibility.

Figure 4(b) presents the marginal posterior densities for p based on ${\pi }^{r_1}$ (solid line), ${\pi }^{r_2}$ (dashed line), ${\pi }^{r_3}$ (long-dashed line) and π ^U (dotted line). In addition, the vertical lines indicate the limits of the 95% HPD credible intervals. The three reference priors lead to similar posterior densities for p, while the π ^U prior leads to a substantially different posterior density for p. Figure 4(b) illustrates why the π ^U leads to unnecessarily wider credible intervals. That combined with π ^U -based credible intervals having coverage lower than nominal leads us to prefer the data analysis based on the reference priors.

4.2.2 Sold home videos vs. profits at the box office

We analyze a dataset about the relationship between the number of sold home videos in thousands (videos: y) and the profits at the box office in million of dollars (gross: x). This dataset has been previously analyzed by Levine et al. (2006) and Salazar et al. (2012) and comprises observations on 30 movies. A scatterplot of the variables of interest is shown in Figure 5(a). Using a linear model with EP errors and the independence Jeffreys prior ${\pi }^{I_2}$ given in Equation (6), Salazar et al. (2012) found evidence of a platykurtic distribution for the errors. Here we compare three analyses of this home videos dataset with an EP linear regression model obtained by applying the reference priors ${\pi }^{r_1}$ and ${\pi }^{r_2}$, and the noninformative prior π ^U .

Figure 5(a) shows the model fit for each of the priors we consider. The fits based on the reference priors visually coincide, whereas the fit based on π ^U is slightly different. This is confirmed by Table 2, that shows that the slopes for the three fits are similar and around 4.33, whereas the intercept for the π ^U -based fit is about 4.5% larger than the intercept for the ${\pi }^{r_1}$- and ${\pi }^{r_2}$-based fits. Even more striking are the differences between the reference analyses and the π ^U -based analysis for σ and p. For σ, both posterior medians based on ${\pi }^{r_1}$ and ${\pi }^{r_2}$ are very similar and equal to 67.37 and 68.38 respectively, while the posterior median based on π ^U is 77.50. Moreover, the 95% credible intervals for σ based on ${\pi }^{r_1}$ and ${\pi }^{r_2}$ are very similar and equal to (38.08,93.64) and (38.10,93.64) respectively, while the interval based on π ^U is substantially different and equal to (47.17,98.69).

	π U			${\mathit{\pi }}^{{\mathit{r}}_1}$			${\mathit{\pi }}^{{\mathit{r}}_2}$
	Mode	Median	95% C.I.	Mode	Median	95% C.I.	Mode	Median	95% C.I.
p	2.64	4.36	(1.36, 9.64)	1.82	2.64	(1.00, 7.01)	1.83	2.64	(1.00, 7.18)
σ	80.51	77.50	(47.17, 98.69)	67.37	67.37	(38.08, 93.64)	68.38	68.38	(38.10, 93.64)
β 1	83.11	83.11	(54.92, 107.65)	79.39	79.40	(53.03, 105.76)	79.53	79.53	(53.16, 104.98)
β 2	4.31	4.35	(3.42, 5.24)	4.32	4.32	(3.31, 5.33)	4.33	4.33	(3.32, 5.34)

The reference analyses for p are also strikingly distinct from the π ^U -based analysis for p. First, the posterior medians for p based on ${\pi }^{r_1}$ and ${\pi }^{r_2}$ coincide and are equal to 2.64 while the π ^U -based posterior median differs tremendously and is equal to 4.36. Second, the 95% credible intervals for p based on ${\pi }^{r_1}$ and ${\pi }^{r_2}$ are similar and equal to (1.00,7.01) and (1.00,7.18) respectively, while the π ^U -based interval for p differs tremendously from the reference CIs and is equal to (1.36,9.64). Hence, the π ^U -based CI is more than 30% wider than the reference CIs. This undesirable feature of π ^U -based CIs coincides with the results from the simulation study presented in Section 4.1.

Finally, Figure 5(b) presents the marginal posterior densities for p based on ${\pi }^{r_1}$, ${\pi }^{r_2}$, and π ^U . This figure sheds light on the reason for the striking difference between the ${\pi }^{r_1}$- and ${\pi }^{r_2}$-based CIs and the π ^U -based CI. The problem with the π ^U -based analysis is that the right tail of the marginal posterior density for p decays too slowly. As a result, for the home video dataset the π ^U -based CI depends dramatically on the right side truncation of the prior, which in this manuscript has been fixed at 10. Figure 5(b) makes it really clear that a larger truncation point would have a huge impact in the resulting π ^U -based CI for p. This dataset clearly illustrates the superiority of the Bayesian reference analyses.